Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections
Abstract
Dedicated wide-area network connections are increasingly employed in high-performance computing and big data scenarios. One might expect the performance and dynamics of data transfers over such connections to be easy to analyze due to the lack of competing traffic. However, non-linear transport dynamics and end-system complexities (e.g., multi-core hosts and distributed filesystems) can in fact make analysis surprisingly challenging. We present extensive measurements of memory-to-memory and disk-to-disk file transfers over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory-to-memory transfers, profiles of both TCP and UDT throughput as a function of RTT show concave and convex regions; large buffer sizes and more parallel flows lead to wider concave regions, which are highly desirable. TCP and UDT both also display complex throughput dynamics, as indicated by their Poincare maps and Lyapunov exponents. For disk-to-disk transfers, we determine that high throughput can be achieved via a combination of parallel I/O threads, parallel network threads, and direct I/O mode. Our measurements also show that Lustre filesystems can be mounted over long-haul connections using LNet routers, although challenges remain in jointly optimizing file I/O and transport method parameters to achieve peak throughput.
- Authors:
-
- ORNL
- University of Chicago
- Argonne National Laboratory (ANL)
- University of Memphis
- Harrisburg University
- University of Massachusetts, Amherst
- Publication Date:
- Research Org.:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- OSTI Identifier:
- 1399533
- DOE Contract Number:
- AC05-00OR22725
- Resource Type:
- Conference
- Resource Relation:
- Conference: IEEE 26th International Conference on Computer Communication and Networks (ICCCN) - Vancouver, , Canada - 7/31/2017 4:00:00 AM-8/3/2017 4:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Citation Formats
Rao, Nageswara S., Liu, Qiang, Sen, Satyabrata, Hanley, Jesse A., Foster, Ian, Kettimuthu, R., Wu, Qishi, Yun, Daqing, Towsley, Don, and Vardoyan, Gayane. Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections. United States: N. p., 2017.
Web. doi:10.1109/ICCCN.2017.8038432.
Rao, Nageswara S., Liu, Qiang, Sen, Satyabrata, Hanley, Jesse A., Foster, Ian, Kettimuthu, R., Wu, Qishi, Yun, Daqing, Towsley, Don, & Vardoyan, Gayane. Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections. United States. https://doi.org/10.1109/ICCCN.2017.8038432
Rao, Nageswara S., Liu, Qiang, Sen, Satyabrata, Hanley, Jesse A., Foster, Ian, Kettimuthu, R., Wu, Qishi, Yun, Daqing, Towsley, Don, and Vardoyan, Gayane. 2017.
"Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections". United States. https://doi.org/10.1109/ICCCN.2017.8038432. https://www.osti.gov/servlets/purl/1399533.
@article{osti_1399533,
title = {Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections},
author = {Rao, Nageswara S. and Liu, Qiang and Sen, Satyabrata and Hanley, Jesse A. and Foster, Ian and Kettimuthu, R. and Wu, Qishi and Yun, Daqing and Towsley, Don and Vardoyan, Gayane},
abstractNote = {Dedicated wide-area network connections are increasingly employed in high-performance computing and big data scenarios. One might expect the performance and dynamics of data transfers over such connections to be easy to analyze due to the lack of competing traffic. However, non-linear transport dynamics and end-system complexities (e.g., multi-core hosts and distributed filesystems) can in fact make analysis surprisingly challenging. We present extensive measurements of memory-to-memory and disk-to-disk file transfers over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory-to-memory transfers, profiles of both TCP and UDT throughput as a function of RTT show concave and convex regions; large buffer sizes and more parallel flows lead to wider concave regions, which are highly desirable. TCP and UDT both also display complex throughput dynamics, as indicated by their Poincare maps and Lyapunov exponents. For disk-to-disk transfers, we determine that high throughput can be achieved via a combination of parallel I/O threads, parallel network threads, and direct I/O mode. Our measurements also show that Lustre filesystems can be mounted over long-haul connections using LNet routers, although challenges remain in jointly optimizing file I/O and transport method parameters to achieve peak throughput.},
doi = {10.1109/ICCCN.2017.8038432},
url = {https://www.osti.gov/biblio/1399533},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Aug 01 00:00:00 EDT 2017},
month = {Tue Aug 01 00:00:00 EDT 2017}
}