Learning concave-convex profiles of data transport over dedicated connections

Rao, Nageswara; Sen, Satyabrata; Liu, Zhengchun; Kettimuthu, R.; Foster, Ian

doi:10.1007/978-3-030-19945-6_1

Learning concave-convex profiles of data transport over dedicated connections

Conference · Wed May 01 04:00:00 EDT 2019

DOI:https://doi.org/10.1007/978-3-030-19945-6_1· OSTI ID:1513419

^[1]; ^[1]; ^[2]; Kettimuthu, R. ^[2]; Foster, Ian ^[2]

ORNL
Argonne National Laboratory (ANL)

Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or credit limits. We present a machine learning method that explicitly infers these concave and convex regions and transitions between them using sigmoid functions. We also provide distribution-free confidence estimates for the generalization error of these concave-convex profile estimates. Throughput profiles for data transfers over 10 Gbps connections with 0–366 ms RTT provide important performance insights, including the near optimality of transfers performed with the XDD tool between XFS filesystems, and the performance limits of wide-area Lustre extensions using LNet routers. A direct application of generic machine learning packages does not adequately highlight these critical performance regions or provide as precise confidence estimates.

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE National Nuclear Security Administration (NNSA)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1513419

Country of Publication:: United States

Language:: English

References (12)

On Analytics of File Transfer Rates over Dedicated Wide-Area Connections Sen, Satyabrata; Rao, Nageswara S. V.; Liu, Qiang 2017 IEEE 13th International Conference on e-Science (e-Science) https://doi.org/10.1109/eScience.2017.93	conference	October 2017
Overlay Networks of In Situ Instruments for Probabilistic Guarantees on Message Delays in Wide-Area Networks Rao, N. S. V. IEEE Journal on Selected Areas in Communications, Vol. 22, Issue 1 https://doi.org/10.1109/JSAC.2003.818797	journal	January 2004
UDT: UDP-based data transfer for high-speed wide area networks Gu, Yunhong; Grossman, Robert L. Computer Networks, Vol. 51, Issue 7 https://doi.org/10.1016/j.comnet.2006.11.009	journal	May 2007
Experiments and Analyses of Data Transfers over Wide-Area Dedicated Connections Rao, Nageswara S. V.; Liu, Qiang; Sen, Satyabrata 2017 26th International Conference on Computer Communication and Networks (ICCCN) https://doi.org/10.1109/ICCCN.2017.8038432	conference	July 2017
Simple sample bound for feedforward sigmoid networks with bounded weights Rao, Nageswara S. V. Neurocomputing, Vol. 29, Issue 1-3 https://doi.org/10.1016/S0925-2312(99)00113-7	journal	November 1999
Cross-geography scientific data transferring trends and behavior Liu, Zhengchun; Kettimuthu, Rajkumar; Foster, Ian Proceedings of the 27th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '18 https://doi.org/10.1145/3208040.3208053	conference	January 2018
A technique for moving large data sets over high-performance long distance networks Settlemyer, Bradley W.; Dobson, Jonathan D.; Hodson, Stephen W. 2011 IEEE 27th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2011.5937236	conference	May 2011
Experimental Analysis of File Transfer Rates over Wide-Area Dedicated Connections Rao, Nageswara S. V.; Liu, Qiang; Sen, Satyabrata 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS) https://doi.org/10.1109/HPCC-SmartCity-DSS.2016.0038	conference	December 2016
TCP Throughput Profiles Using Measurements over Dedicated Connections Rao, Nageswara S. V.; Liu, Qiang; Sen, Satyabrata Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '17 https://doi.org/10.1145/3078597.3078615	conference	January 2017
Software as a service for data scientists Allen, Bryce; Pickett, Karl; Tuecke, Steven Communications of the ACM, Vol. 55, Issue 2 https://doi.org/10.1145/2076450.2076468	journal	February 2012
Wide-area lustre file system using LNet routers Rao, Nageswara S. V.; Imam, Neena; Hanley, Jessey 2018 Annual IEEE International Systems Conference (SysCon) https://doi.org/10.1109/SYSCON.2018.8369541	conference	April 2018
Nonparametric estimation and classification using radial basis function nets and empirical risk minimization Krzyzak, A.; Linder, T.; Lugosi, C. IEEE Transactions on Neural Networks, Vol. 7, Issue 2 https://doi.org/10.1109/72.485681	journal	March 1996

Similar Records

Learning Concave-Convex Profiles of Data Transport Over Dedicated Connections

Conference · Mon Dec 31 23:00:00 EST 2018 · OSTI ID:1574712

Measurements and analytics of wide-area file transfers over dedicated connections

Conference · Mon Dec 31 23:00:00 EST 2018 · OSTI ID:1509527

On Data Transfers Over Wide-Area Dedicated Connections

Book · Sun Dec 31 23:00:00 EST 2017 · OSTI ID:1399399

Learning concave-convex profiles of data transport over dedicated connections

Citation Formats

References (12)

Similar Records

Related Subjects