skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Learning Concave-Convex Profiles of Data Transport Over Dedicated Connections

Abstract

Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or credit limits. We present a machine learning method that explicitly infers these concave and convex regions and transitions between them using sigmoid functions. We also provide distribution-free confidence estimates for the generalization error of these concave-convex profile estimates. Throughput profiles for data transfers over 10 Gbps connections with 0-366 ms RTT provide important performance insights, including the near optimality of transfers performed with the XDD tool between XFS filesystems, and the performance limits of wide-area Lustre extensions using LNet routers. A direct application of generic machine learning packages does not adequately highlight these critical performance regions or provide as precise confidencemore » estimates.« less

Authors:
; ; ; ;
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
USDOE Office of Science - Office of Advanced Scientific Computing Research; U.S. Department of Defense (DOD)
OSTI Identifier:
1574712
DOE Contract Number:  
AC02-06CH11357
Resource Type:
Conference
Resource Relation:
Conference: 1st International Conference on Machine Learning for Networking, 11/27/18 - 11/29/18, Paris, FR
Country of Publication:
United States
Language:
English
Subject:
concavity-convexity; data transport; generalization bounds; throughput profile

Citation Formats

Rao, Nageswara S. V., Sen, Satyabrata, Liu, Zhengchun, Kettimuthu, Rajkumar, and Foster, Ian. Learning Concave-Convex Profiles of Data Transport Over Dedicated Connections. United States: N. p., 2019. Web. doi:10.1007/978-3-030-19945-6_1.
Rao, Nageswara S. V., Sen, Satyabrata, Liu, Zhengchun, Kettimuthu, Rajkumar, & Foster, Ian. Learning Concave-Convex Profiles of Data Transport Over Dedicated Connections. United States. doi:10.1007/978-3-030-19945-6_1.
Rao, Nageswara S. V., Sen, Satyabrata, Liu, Zhengchun, Kettimuthu, Rajkumar, and Foster, Ian. Tue . "Learning Concave-Convex Profiles of Data Transport Over Dedicated Connections". United States. doi:10.1007/978-3-030-19945-6_1.
@article{osti_1574712,
title = {Learning Concave-Convex Profiles of Data Transport Over Dedicated Connections},
author = {Rao, Nageswara S. V. and Sen, Satyabrata and Liu, Zhengchun and Kettimuthu, Rajkumar and Foster, Ian},
abstractNote = {Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or credit limits. We present a machine learning method that explicitly infers these concave and convex regions and transitions between them using sigmoid functions. We also provide distribution-free confidence estimates for the generalization error of these concave-convex profile estimates. Throughput profiles for data transfers over 10 Gbps connections with 0-366 ms RTT provide important performance insights, including the near optimality of transfers performed with the XDD tool between XFS filesystems, and the performance limits of wide-area Lustre extensions using LNet routers. A direct application of generic machine learning packages does not adequately highlight these critical performance regions or provide as precise confidence estimates.},
doi = {10.1007/978-3-030-19945-6_1},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2019},
month = {1}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: