Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure

Kim, Youngjae; Atchley, Scott; Vallee, Geoffroy R.; Lee, Sangkeun; Shipman, Galen M.

doi:10.1109/TPDS.2016.2550439

Title: Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure

Authors:

^[1]; Atchley, Scott ^[2]; Vallee, Geoffroy R. ^[2]; Lee, Sangkeun ^[2];

^[3]

Sogang Univ., Seoul (Korea, Republic of). Dept. of Computer Science and Engineering
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Publication Date:: Tue Apr 05 00:00:00 EDT 2016

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Sogang Univ., Seoul (Korea, Republic of); Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE Office of Science (SC); Ministry of Science, ICT and Future Planning (MSIP) of Korea; National Research Foundation of Korea (NRF); USDOE Office of Science (SC). Advanced Scientific Computing Research (ASCR) (SC-21)

Contributing Org.:: Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

OSTI Identifier:: 1361284

Alternate Identifier(s):: OSTI ID: 1407899

Report Number(s):: LA-UR-17-27410
Journal ID: ISSN 1045-9219; KJ0502000; KJ0404000; ERKJZN1; ERKJM05; TRN: US1702181

Grant/Contract Number:: AC05-00OR22725; R0190-15-2012; 2015R1C1A1A0152105; AC52-06NA25396

Resource Type:: Accepted Manuscript

Journal Name:: IEEE Transactions on Parallel and Distributed Systems

Additional Journal Information:: Journal Volume: 28; Journal Issue: 1; Journal ID: ISSN 1045-9219

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Data transfer; Servers; Throughput; Data models; Instruction sets; Big data; Computational modeling; File and storage systems; Parallel file systems; networks; I/O scheduling

Citation Formats


                    Kim, Youngjae, Atchley, Scott, Vallee, Geoffroy R., Lee, Sangkeun, and Shipman, Galen M. Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure.  United States: N. p., 2016. 
Web.  doi:10.1109/TPDS.2016.2550439.

Copy to clipboard


                    Kim, Youngjae, Atchley, Scott, Vallee, Geoffroy R., Lee, Sangkeun, & Shipman, Galen M. Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure.  United States.  https://doi.org/10.1109/TPDS.2016.2550439

Copy to clipboard


                    Kim, Youngjae, Atchley, Scott, Vallee, Geoffroy R., Lee, Sangkeun, and Shipman, Galen M. Tue .  
"Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure".  United States.  https://doi.org/10.1109/TPDS.2016.2550439.  https://www.osti.gov/servlets/purl/1361284.

Copy to clipboard


                    
@article{osti_1361284,

  title        = {Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure},

  author       = {Kim, Youngjae and Atchley, Scott and Vallee, Geoffroy R. and Lee, Sangkeun and Shipman, Galen M.},

  abstractNote = {},

  doi          = {10.1109/TPDS.2016.2550439},

  journal      = {IEEE Transactions on Parallel and Distributed Systems},

  number       = 1,

  volume       = 28,

  place        = {United States},

  year         = {Tue Apr 05 00:00:00 EDT 2016},

  month        = {Tue Apr 05 00:00:00 EDT 2016}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/TPDS.2016.2550439

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 13 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referencing / citing this record:

Optimizing communication performance in scale-out storage system
journal, July 2018

Song, Uiseok; Jeong, Bodon; Park, Sungyong
Cluster Computing, Vol. 22, Issue 2
DOI: 10.1007/s10586-018-2831-6

Async-LCAM: a lock contention aware messenger for Ceph distributed storage system
journal, July 2018

Jeong, Bodon; Khan, Awais; Park, Sungyong
Cluster Computing, Vol. 22, Issue 2
DOI: 10.1007/s10586-018-2832-5

New Bargaining Game Model for Collaborative Vehicular Network Services
journal, March 2019

Kim, Sungwook
Mobile Information Systems, Vol. 2019
DOI: 10.1155/2019/6269475

Similar Records in DOE PAGES and OSTI.GOV collections:

NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

Journal Article Kim, Taeuk ; Khan, Awais ; Kim, Youngjae ; ... - Scientific Programming

The evergrowing trend of big data has led scientists to share and transfer the simulation and analytical data across the geodistributed research and computing facilities. However, the existing data transfer frameworks used for data sharing lack the capability to adopt the attributes of the underlying parallel file systems (PFS). LADS (Layout-Aware Data Scheduling) is an end-to-end data transfer tool optimized for terabit network using a layout-aware data scheduling via PFS. However, it does not consider the NUMA (Nonuniform Memory Access) architecture. In this paper, we propose a NUMA-aware thread and resource scheduling for optimized data transfer in terabit network. First,more »« less
Cited by 2
https://doi.org/10.1155/2018/4120561

Full Text Available
End-to-end data movement using MPI-IO over routed terabits infrastructures. In: NDM '13 Proceedings of the Third International Workshop on Network-Aware Data Management, Article No. 9

Conference Vallée, Geoffroy ; Atchley, Scott ; Kim, Youngjae ; ...

Scientific discovery is nowadays driven by large-scale simulations running on massively parallel high-performance computing (HPC) systems. These applications each generate a large amount of data, which then needs to be post-processed for example for data mining or visualization. Unfortunately, the computing platform used for post processing might be different from the one on which the data is initially generated, introducing the challenge of moving large amount of data between computing platforms. This is especially challenging when these two platforms are geographically separated since the data needs to be moved between computing facilities. This is even more critical when scientists tightlymore »« less
https://doi.org/10.1145/2534695.2534705
End-System Aware Peak Link Utilization Transport for Big Data Transfer over Dedicated Connections

Conference Rao, Nageswara ; Lyu, Xukang ; Wu, Chase Qishi

High-performance networks provisioning 10/40-Gbps dedicated channels have been rapidly developed andincreasingly deployed to support big data transfers on the order of terabytes or petabytes in large-scale network-intensiveapplications. However, end users have not seen corresponding increase in application throughput mainly because traditional end-to-end transport methods are not optimized for such connections. New congestion or flow control mechanisms are desirable to meet the challenges brought to transport protocol design by dedicated connections. The advent and proliferation of multi-core processors make it now possible to improve application throughput by providing multiple processing and network resources to a single data transfer. Based on themore »« less
Full Text Available
Profiling optimization for big data transfer over dedicated channels

Conference Yun, D. ; Wu, Qishi ; Rao, Nageswara S ; ...

The transfer of big data is increasingly supported by dedicated channels in high-performance networks, where transport protocols play an important role in maximizing applicationlevel throughput and link utilization. The performance of transport protocols largely depend on their control parameter settings, but it is prohibitively time consuming to conduct an exhaustive search in a large parameter space to find the best set of parameter values. We propose FastProf, a stochastic approximation-based transport profiler, to quickly determine the optimal operational zone of a given data transfer protocol/method over dedicated channels. We implement and test the proposed method using both emulations based onmore »« less
https://doi.org/10.1109/ICCCN.2016.7568562
Advising Big Data Transfer Over Dedicated Connections Based on Profiling Optimization

Journal Article Yun, Daqing ; Wu, Chase Q. ; Rao, Nageswara V. ; ... - IEEE/ACM Transactions on Networking

Big data transfer in next-generation scientific applications is currently commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to themore »« less
Cited by 4
https://doi.org/10.1109/tnet.2019.2943884

Full Text Available

Similar Records

Title: Optimizing End-to-End Big Data Transfers over Terabits Network Infrastructure

Citation Formats

Optimizing communication performance in scale-out storage system journal, July 2018

Async-LCAM: a lock contention aware messenger for Ceph distributed storage system journal, July 2018

New Bargaining Game Model for Collaborative Vehicular Network Services journal, March 2019

Optimizing communication performance in scale-out storage system
journal, July 2018

Async-LCAM: a lock contention aware messenger for Ceph distributed storage system
journal, July 2018

New Bargaining Game Model for Collaborative Vehicular Network Services
journal, March 2019