Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections

Conference · Tue Aug 01 00:00:00 EDT 2017

DOI:https://doi.org/10.1109/ICCCN.2017.8038432· OSTI ID:1399533

Rao, Nageswara S. ^[1]; Liu, Qiang ^[1]; Sen, Satyabrata ^[1]; Hanley, Jesse A. ^[1]; Foster, Ian ^[2]; Kettimuthu, R. ^[3]; Wu, Qishi ^[4]; Yun, Daqing ^[5]; Towsley, Don ^[6]; Vardoyan, Gayane ^[6]

---

1. ORNL
2. University of Chicago
3. Argonne National Laboratory (ANL)
4. University of Memphis
5. Harrisburg University
6. University of Massachusetts, Amherst

Dedicated wide-area network connections are increasingly employed in high-performance computing and big data scenarios. One might expect the performance and dynamics of data transfers over such connections to be easy to analyze due to the lack of competing traffic. However, non-linear transport dynamics and end-system complexities (e.g., multi-core hosts and distributed filesystems) can in fact make analysis surprisingly challenging. We present extensive measurements of memory-to-memory and disk-to-disk file transfers over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory-to-memory transfers, profiles of both TCP and UDT throughput as a function of RTT show concave and convex regions; large buffer sizes and more parallel flows lead to wider concave regions, which are highly desirable. TCP and UDT both also display complex throughput dynamics, as indicated by their Poincare maps and Lyapunov exponents. For disk-to-disk transfers, we determine that high throughput can be achieved via a combination of parallel I/O threads, parallel network threads, and direct I/O mode. Our measurements also show that Lustre filesystems can be mounted over long-haul connections using LNet routers, although challenges remain in jointly optimizing file I/O and transport method parameters to achieve peak throughput.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1399533
• Country of Publication: United States
• Language: English