skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Comparative Performance Evaluation of High-performance Data Transfer Tools

Abstract

Data transfer in wide-area networks has been long studied in different contexts, from data sharing among data centers to online access to scientific data. Many software tools and platforms have been developed to facilitate easy, reliable, fast, and secure data transfer over wide area networks, such as GridFTP, FDT, bbcp, mdtmFTP, and XDD. However, few studies have shown the full capabilities of existing data transfer tools from the perspective of whether such tools have fully adopted state-of-the-art techniques through meticulous comparative evaluations. In this paper, we evaluate the performance of the four highperformance data transfer tools (GridFTP, FDT, mdtmFTP, and XDD) in various environments. Our evaluation suggests that each tool has strengths and weaknesses. FDT and GridFTP perform consistently in diverse environments. XDD and mdtmFTP show improved performance in limited environments and datasets during our evaluation. Unlike other studies on data transfer tools, we also evaluate the predictability of the tools' performance, an important factor for scheduling different stages of science workflows. Performance predictability also helps in (auto)tuning the configurable parameters of the data transfer tool. We apply statistical learning techniques such as linear/polynomial regression, and k-nearest neighbors (kNN), to assess the performance predictability of each tool using its controlmore » parameters. Our results show that we can achieve good prediction performance for GridFTP and mdtmFTP using linear regression and kNN, respectively.« less

Authors:
 [1];  [2];  [2];  [2]; ORCiD logo [3];  [1]
  1. University of Nebraska, Lincoln
  2. Argonne National Laboratory (ANL)
  3. ORNL
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1566967
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: IEEE International Conference on Advanced Networks and Telecommunications Systems - Bangalore, , India - 12/16/2018 5:00:00 AM-12/19/2018 5:00:00 AM
Country of Publication:
United States
Language:
English

Citation Formats

Nadig, Deepak, Jung, Eun-sung, Kettimuthu, Rajkumar, Foster, Ian, Rao, Nageswara S., and Ramamurthy, Byrav. Comparative Performance Evaluation of High-performance Data Transfer Tools. United States: N. p., 2018. Web. doi:10.1109/ANTS.2018.8710071.
Nadig, Deepak, Jung, Eun-sung, Kettimuthu, Rajkumar, Foster, Ian, Rao, Nageswara S., & Ramamurthy, Byrav. Comparative Performance Evaluation of High-performance Data Transfer Tools. United States. doi:10.1109/ANTS.2018.8710071.
Nadig, Deepak, Jung, Eun-sung, Kettimuthu, Rajkumar, Foster, Ian, Rao, Nageswara S., and Ramamurthy, Byrav. Sat . "Comparative Performance Evaluation of High-performance Data Transfer Tools". United States. doi:10.1109/ANTS.2018.8710071. https://www.osti.gov/servlets/purl/1566967.
@article{osti_1566967,
title = {Comparative Performance Evaluation of High-performance Data Transfer Tools},
author = {Nadig, Deepak and Jung, Eun-sung and Kettimuthu, Rajkumar and Foster, Ian and Rao, Nageswara S. and Ramamurthy, Byrav},
abstractNote = {Data transfer in wide-area networks has been long studied in different contexts, from data sharing among data centers to online access to scientific data. Many software tools and platforms have been developed to facilitate easy, reliable, fast, and secure data transfer over wide area networks, such as GridFTP, FDT, bbcp, mdtmFTP, and XDD. However, few studies have shown the full capabilities of existing data transfer tools from the perspective of whether such tools have fully adopted state-of-the-art techniques through meticulous comparative evaluations. In this paper, we evaluate the performance of the four highperformance data transfer tools (GridFTP, FDT, mdtmFTP, and XDD) in various environments. Our evaluation suggests that each tool has strengths and weaknesses. FDT and GridFTP perform consistently in diverse environments. XDD and mdtmFTP show improved performance in limited environments and datasets during our evaluation. Unlike other studies on data transfer tools, we also evaluate the predictability of the tools' performance, an important factor for scheduling different stages of science workflows. Performance predictability also helps in (auto)tuning the configurable parameters of the data transfer tool. We apply statistical learning techniques such as linear/polynomial regression, and k-nearest neighbors (kNN), to assess the performance predictability of each tool using its control parameters. Our results show that we can achieve good prediction performance for GridFTP and mdtmFTP using linear regression and kNN, respectively.},
doi = {10.1109/ANTS.2018.8710071},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {12}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: