NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

Kim, Taeuk; Khan, Awais; Kim, Youngjae; Kasu, Preethika; Atchley, Scott

doi:10.1155/2018/4120561

Title: NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

Abstract

The evergrowing trend of big data has led scientists to share and transfer the simulation and analytical data across the geodistributed research and computing facilities. However, the existing data transfer frameworks used for data sharing lack the capability to adopt the attributes of the underlying parallel file systems (PFS). LADS (Layout-Aware Data Scheduling) is an end-to-end data transfer tool optimized for terabit network using a layout-aware data scheduling via PFS. However, it does not consider the NUMA (Nonuniform Memory Access) architecture. In this paper, we propose a NUMA-aware thread and resource scheduling for optimized data transfer in terabit network. First, we propose distributed RMA buffers to reduce memory controller contention in CPU sockets and then schedule the threads based on CPU socket and NUMA nodes inside CPU socket to reduce memory access latency. We design and implement the proposed resource and thread scheduling in the existing LADS framework. Experimental results showed from 21.7% to 44% improvement with memory-level optimizations in the LADS framework as compared to the baseline without any optimization.

Authors:

^[1];

^[2]; Atchley, Scott ^[3]

Sogang Univ., Seoul (Korea)
Ajou Univ., Suwon (Korea)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Publication Date:: 2018-05-07

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1565699

Resource Type:: Accepted Manuscript

Journal Name:: Scientific Programming

Additional Journal Information:: Journal Volume: 2018; Journal ID: ISSN 1058-9244

Publisher:: Hindawi

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Computer Science

Citation Formats


                    Kim, Taeuk, Khan, Awais, Kim, Youngjae, Kasu, Preethika, and Atchley, Scott. NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure.  United States: N. p., 2018. 
        Web.  doi:10.1155/2018/4120561.

Copy to clipboard


                    Kim, Taeuk, Khan, Awais, Kim, Youngjae, Kasu, Preethika, & Atchley, Scott. NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure.  United States.  doi:https://doi.org/10.1155/2018/4120561

Copy to clipboard


                    Kim, Taeuk, Khan, Awais, Kim, Youngjae, Kasu, Preethika, and Atchley, Scott. Mon .  
        "NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure".  United States.  doi:https://doi.org/10.1155/2018/4120561.  https://www.osti.gov/servlets/purl/1565699.

Copy to clipboard


                    
@article{osti_1565699,

  title        = {NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure},

  author       = {Kim, Taeuk and Khan, Awais and Kim, Youngjae and Kasu, Preethika and Atchley, Scott},

  abstractNote = {The evergrowing trend of big data has led scientists to share and transfer the simulation and analytical data across the geodistributed research and computing facilities. However, the existing data transfer frameworks used for data sharing lack the capability to adopt the attributes of the underlying parallel file systems (PFS). LADS (Layout-Aware Data Scheduling) is an end-to-end data transfer tool optimized for terabit network using a layout-aware data scheduling via PFS. However, it does not consider the NUMA (Nonuniform Memory Access) architecture. In this paper, we propose a NUMA-aware thread and resource scheduling for optimized data transfer in terabit network. First, we propose distributed RMA buffers to reduce memory controller contention in CPU sockets and then schedule the threads based on CPU socket and NUMA nodes inside CPU socket to reduce memory access latency. We design and implement the proposed resource and thread scheduling in the existing LADS framework. Experimental results showed from 21.7% to 44% improvement with memory-level optimizations in the LADS framework as compared to the baseline without any optimization.},

  doi          = {10.1155/2018/4120561},

  journal      = {Scientific Programming},

  number       = ,

  volume       = 2018,

  place        = {United States},

  year         = {2018},

  month        = {5}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

DOI: https://doi.org/10.1155/2018/4120561

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Analysis of NUMA effects in modern multicore systems for the design of high-performance data transfer applications
journal, September 2017

Li, Tan; Ren, Yufei; Yu, Dantong
Future Generation Computer Systems, Vol. 74
DOI: 10.1016/j.future.2017.04.001

RAMSYS: Resource-Aware Asynchronous Data Transfer with Multicore SYStems
journal, May 2017

Li, Tan; Ren, Yufei; Yu, Dantong
IEEE Transactions on Parallel and Distributed Systems, Vol. 28, Issue 5
DOI: 10.1109/TPDS.2016.2619344

Globus Toolkit Version 4: Software for Service-Oriented Systems
journal, July 2006

Foster, Ian
Journal of Computer Science and Technology, Vol. 21, Issue 4
DOI: 10.1007/s11390-006-0513-y

Similar Records in DOE PAGES and OSTI.GOV collections:

Achieving High Performance With TCP Over 40 GbE on NUMA Architectures for CMS Data Acquisition

Conference Bawej, Tomasz ; et al. - IEEE Trans.Nucl.Sci.

TCP and the socket abstraction have barely changed over the last two decades, but at the network layer there has been a giant leap from a few megabits to 100 gigabits in bandwidth. At the same time, CPU architectures have evolved into the multicore era and applications are expected to make full use of all available resources. Applications in the data acquisition domain based on the standard socket library running in a Non-Uniform Memory Access (NUMA) architecture are unable to reach full efficiency and scalability without the software being adequately aware about the IRQ (Interrupt Request), CPU and memory affinities.more »« less
DOI: https://doi.org/10.1109/TNS.2015.2409898

Full Text Available
Critical Path-Based Thread Placement for NUMA Systems

Journal Article Su, Chun-Yi ; Li, Dong ; Nikolopoulos, Dimitrios ; ... - Performance Evaluation Review

Multicore multiprocessors use Non Uniform Memory Architecture (NUMA) to improve their scalability. However,NUMA introduces performance penalties due to remote memory accesses. Without efficiently managing data layout and thread mapping to cores, scientific applications, even if they are optimized for NUMA, may suffer performance loss. In this paper, we present an algorithm that optimizes the placement of OpenMP threads on NUMA processors. By collecting information from hardware counters and defining new metrics to capture the effects of thread placement, the algorithm reduces NUMA performance penalty by minimizing the critical path of OpenMP parallel regions and by avoiding local memory resource contention.more »« less
DOI: https://doi.org/10.1145/2381056.2381079
Critical Path-Based Thread Placement for NUMA Systems

Journal Article Su, Chun-Yi ; Li, Dong ; Nikolopoulos, Dimitrios ; ... - Performance Evaluation Review

Multicore multiprocessors use Non Uniform Memory Architecture (NUMA) to improve their scalability. However,NUMA introduces performance penalties due to remote memory accesses. Without efficiently managing data layout and thread mapping to cores, scientific applications, even if they are optimized for NUMA, may suffer performance loss. In this paper, we present an algorithm that optimizes the placement of OpenMP threads on NUMA processors. By collecting information from hardware counters and defining new metrics to capture the effects of thread placement, the algorithm reduces NUMA performance penalty by minimizing the critical path of OpenMP parallel regions and by avoiding local memory resource contention.more »« less
DOI: https://doi.org/10.1145/2381056.2381079
Layout-Aware I/O Scheduling for Terabits Data Movement

Conference Kim, Youngjae ; Atchley, Scott ; Vallee, Geoffroy R ; ...

Many science facilities, such as the Department of Energy s Leadership Computing Facilities and experimental facilities including the Spallation Neutron Source, Stanford Linear Accelerator Center, and Advanced Photon Source, produce massive amounts of experimental and simulation data. These data are often shared among the facilities and with collaborating institutions. Moving large datasets over the wide- area network (WAN) is a major problem inhibiting collaboration. Next- generation, terabit-networks will help alleviate the problem, however, the parallel storage systems on the end-system hosts at these institutions can become a bottleneck for terabit data movement. The parallel storage system (PFS) is shared bymore »« less
Layout-aware I/O Scheduling for terabits data movement

Journal Article Kim, Youngjae ; Atchley, Scott ; Vallee, Geoffroy R. ; ...

Many science facilities, such as the Department of Energy's Leadership Computing Facilities and experimental facilities including the Spallation Neutron Source, Stanford Linear Accelerator Center, and Advanced Photon Source, produce massive amounts of experimental and simulation data. These data are often shared among the facilities and with collaborating institutions. Moving large datasets over the wide-area network (WAN) is a major problem inhibiting collaboration. Next-generation, terabit-networks will help alleviate the problem, however, the parallel storage systems on the endsystem hosts at these institutions can become a bottleneck for terabit data movement. The parallel storage system (PFS) is shared by simulation systems, experimentalmore »« less
DOI: https://doi.org/10.1109/BigData.2013.6691661

Similar Records

Title: NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure

Abstract

Citation Formats

Analysis of NUMA effects in modern multicore systems for the design of high-performance data transfer applications journal, September 2017

RAMSYS: Resource-Aware Asynchronous Data Transfer with Multicore SYStems journal, May 2017

Globus Toolkit Version 4: Software for Service-Oriented Systems journal, July 2006

Analysis of NUMA effects in modern multicore systems for the design of high-performance data transfer applications
journal, September 2017

RAMSYS: Resource-Aware Asynchronous Data Transfer with Multicore SYStems
journal, May 2017

Globus Toolkit Version 4: Software for Service-Oriented Systems
journal, July 2006