Learning concave-convex profiles of data transport over dedicated connections

Conference · Wed May 01 00:00:00 EDT 2019

DOI:https://doi.org/10.1007/978-3-030-19945-6_1· OSTI ID:1513419

Rao, Nageswara  ^[1]; Sen, Satyabrata  ^[1]; Liu, Zhengchun  ^[2]; Kettimuthu, R. ^[2]; Foster, Ian ^[2]

---

1. ORNL
2. Argonne National Laboratory (ANL)

Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or credit limits. We present a machine learning method that explicitly infers these concave and convex regions and transitions between them using sigmoid functions. We also provide distribution-free confidence estimates for the generalization error of these concave-convex profile estimates. Throughput profiles for data transfers over 10 Gbps connections with 0–366 ms RTT provide important performance insights, including the near optimality of transfers performed with the XDD tool between XFS filesystems, and the performance limits of wide-area Lustre extensions using LNet routers. A direct application of generic machine learning packages does not adequately highlight these critical performance regions or provide as precise confidence estimates.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE National Nuclear Security Administration (NNSA)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1513419
• Resource Relation: Journal Volume: 11407; Conference: International Conference on Machine Learning for Networking (MLN) - Paris, , France - 11/27/2018 10:00:00 AM-11/29/2018 10:00:00 AM
• Country of Publication: United States
• Language: English