skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections

Abstract

Dedicated wide-area network connections are increasingly employed in high-performance computing and big data scenarios. One might expect the performance and dynamics of data transfers over such connections to be easy to analyze due to the lack of competing traffic. However, non-linear transport dynamics and end-system complexities (e.g., multi-core hosts and distributed filesystems) can in fact make analysis surprisingly challenging. We present extensive measurements of memory-to-memory and disk-to-disk file transfers over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory-to-memory transfers, profiles of both TCP and UDT throughput as a function of RTT show concave and convex regions; large buffer sizes and more parallel flows lead to wider concave regions, which are highly desirable. TCP and UDT both also display complex throughput dynamics, as indicated by their Poincare maps and Lyapunov exponents. For disk-to-disk transfers, we determine that high throughput can be achieved via a combination of parallel I/O threads, parallel network threads, and direct I/O mode. Our measurements also show that Lustre filesystems can be mounted over long-haul connections using LNet routers, although challenges remain in jointly optimizing file I/O and transport method parameters to achieve peak throughput.

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1];  [2];  [3];  [4];  [5];  [6];  [6]
  1. ORNL
  2. University of Chicago
  3. Argonne National Laboratory (ANL)
  4. University of Memphis
  5. Harrisburg University
  6. University of Massachusetts, Amherst
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1399533
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: IEEE 26th International Conference on Computer Communication and Networks (ICCCN) - Vancouver, , Canada - 7/31/2017 4:00:00 AM-8/3/2017 4:00:00 AM
Country of Publication:
United States
Language:
English

Citation Formats

Rao, Nageswara S., Liu, Qiang, Sen, Satyabrata, Hanley, Jesse A., Foster, Ian, Kettimuthu, R., Wu, Qishi, Yun, Daqing, Towsley, Don, and Vardoyan, Gayane. Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections. United States: N. p., 2017. Web. doi:10.1109/ICCCN.2017.8038432.
Rao, Nageswara S., Liu, Qiang, Sen, Satyabrata, Hanley, Jesse A., Foster, Ian, Kettimuthu, R., Wu, Qishi, Yun, Daqing, Towsley, Don, & Vardoyan, Gayane. Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections. United States. https://doi.org/10.1109/ICCCN.2017.8038432
Rao, Nageswara S., Liu, Qiang, Sen, Satyabrata, Hanley, Jesse A., Foster, Ian, Kettimuthu, R., Wu, Qishi, Yun, Daqing, Towsley, Don, and Vardoyan, Gayane. 2017. "Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections". United States. https://doi.org/10.1109/ICCCN.2017.8038432. https://www.osti.gov/servlets/purl/1399533.
@article{osti_1399533,
title = {Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections},
author = {Rao, Nageswara S. and Liu, Qiang and Sen, Satyabrata and Hanley, Jesse A. and Foster, Ian and Kettimuthu, R. and Wu, Qishi and Yun, Daqing and Towsley, Don and Vardoyan, Gayane},
abstractNote = {Dedicated wide-area network connections are increasingly employed in high-performance computing and big data scenarios. One might expect the performance and dynamics of data transfers over such connections to be easy to analyze due to the lack of competing traffic. However, non-linear transport dynamics and end-system complexities (e.g., multi-core hosts and distributed filesystems) can in fact make analysis surprisingly challenging. We present extensive measurements of memory-to-memory and disk-to-disk file transfers over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory-to-memory transfers, profiles of both TCP and UDT throughput as a function of RTT show concave and convex regions; large buffer sizes and more parallel flows lead to wider concave regions, which are highly desirable. TCP and UDT both also display complex throughput dynamics, as indicated by their Poincare maps and Lyapunov exponents. For disk-to-disk transfers, we determine that high throughput can be achieved via a combination of parallel I/O threads, parallel network threads, and direct I/O mode. Our measurements also show that Lustre filesystems can be mounted over long-haul connections using LNet routers, although challenges remain in jointly optimizing file I/O and transport method parameters to achieve peak throughput.},
doi = {10.1109/ICCCN.2017.8038432},
url = {https://www.osti.gov/biblio/1399533}, journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue Aug 01 00:00:00 EDT 2017},
month = {Tue Aug 01 00:00:00 EDT 2017}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: