skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Building a Wide-Area File Transfer Performance Predictor: An Empirical Study

Abstract

Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.

Authors:
ORCiD logo [1];  [1];  [1]; ORCiD logo [2];  [1]
  1. Argonne National Laboratory (ANL)
  2. ORNL
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1550725
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Journal Volume: 11407; Conference: First International Conference on Machine Learning for Networking (MLN 2018) - Paris, , France - 11/27/2018 5:00:00 AM-11/29/2018 5:00:00 AM
Country of Publication:
United States
Language:
English

Citation Formats

Liu, Zhengchun, Kettimuthu, Rajkumar, Balaprakash, Prasanna, Rao, Nageswara S., and Foster, Ian. Building a Wide-Area File Transfer Performance Predictor: An Empirical Study. United States: N. p., 2019. Web. doi:10.1007/978-3-030-19945-6_5.
Liu, Zhengchun, Kettimuthu, Rajkumar, Balaprakash, Prasanna, Rao, Nageswara S., & Foster, Ian. Building a Wide-Area File Transfer Performance Predictor: An Empirical Study. United States. doi:10.1007/978-3-030-19945-6_5.
Liu, Zhengchun, Kettimuthu, Rajkumar, Balaprakash, Prasanna, Rao, Nageswara S., and Foster, Ian. Wed . "Building a Wide-Area File Transfer Performance Predictor: An Empirical Study". United States. doi:10.1007/978-3-030-19945-6_5. https://www.osti.gov/servlets/purl/1550725.
@article{osti_1550725,
title = {Building a Wide-Area File Transfer Performance Predictor: An Empirical Study},
author = {Liu, Zhengchun and Kettimuthu, Rajkumar and Balaprakash, Prasanna and Rao, Nageswara S. and Foster, Ian},
abstractNote = {Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.},
doi = {10.1007/978-3-030-19945-6_5},
journal = {},
issn = {0302--9743},
number = ,
volume = 11407,
place = {United States},
year = {2019},
month = {5}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: