Demystifying asynchronous I/O Interference in HPC applications

Tseng, Shu-Mei; Nicolae, Bogdan; Cappello, Franck; Chandramowlishwaran, Aparna

doi:10.1177/10943420211016511

Title: Demystifying asynchronous I/O Interference in HPC applications

Abstract

With increasing complexity of HPC workflows, data management services need to perform expensive I/O operations asynchronously in the background, aiming to overlap the I/O with the application runtime. However, this may cause interference due to competition for resources: CPU, memory/network bandwidth. The advent of multi-core architectures has exacerbated this problem, as many I/O operations are issued concurrently, thereby competing not only with the application but also among themselves. Furthermore, the interference patterns can dynamically change as a response to variations in application behavior and I/O subsystems (e.g. multiple users sharing a parallel file system). Without a thorough understanding, I/O operations may perform suboptimally, potentially even worse than in the blocking case. To fill this gap, here we investigate the causes and consequences of interference due to asynchronous I/O on HPC systems. Specifically, we focus on multi-core CPUs and memory bandwidth, isolating the interference due to each resource. Then, we perform an in-depth study to explain the interplay and contention in a variety of resource sharing scenarios such as varying priority and number of background I/O threads and different I/O strategies: sendfile, read/write, mmap/write underlining trade-offs. The insights from this study are important both to enable guided optimizations of existing backgroundmore »« less

Authors:

Tseng, Shu-Mei ^[1];

^[2]; Cappello, Franck ^[2]; Chandramowlishwaran, Aparna ^[1]

Univ. of California, Irvine, CA (United States)
Argonne National Lab. (ANL), Argonne, IL (United States)

Publication Date:: 2021-05-13

Research Org.:: Argonne National Lab. (ANL), Argonne, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1831116

Grant/Contract Number:: AC02-06CH11357

Resource Type:: Journal Article: Accepted Manuscript

Journal Name:: International Journal of High Performance Computing Applications

Additional Journal Information:: Journal Volume: 35; Journal Issue: 4; Journal ID: ISSN 1094-3420

Publisher:: SAGE

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; I/O interference; asynchronous and concurrent I/O; checkpointing; HPC applications; performance analysis

Citation Formats


                    Tseng, Shu-Mei, Nicolae, Bogdan, Cappello, Franck, and Chandramowlishwaran, Aparna. Demystifying asynchronous I/O Interference in HPC applications.  United States: N. p., 2021. 
        Web.  doi:10.1177/10943420211016511.

Copy to clipboard


                    Tseng, Shu-Mei, Nicolae, Bogdan, Cappello, Franck, & Chandramowlishwaran, Aparna. Demystifying asynchronous I/O Interference in HPC applications.  United States.  https://doi.org/10.1177/10943420211016511

Copy to clipboard


                    Tseng, Shu-Mei, Nicolae, Bogdan, Cappello, Franck, and Chandramowlishwaran, Aparna. 2021.  
        "Demystifying asynchronous I/O Interference in HPC applications".  United States.  https://doi.org/10.1177/10943420211016511.  https://www.osti.gov/servlets/purl/1831116.

Copy to clipboard


                    
@article{osti_1831116,

  title        = {Demystifying asynchronous I/O Interference in HPC applications},

  author       = {Tseng, Shu-Mei and Nicolae, Bogdan and Cappello, Franck and Chandramowlishwaran, Aparna},

  abstractNote = {With increasing complexity of HPC workflows, data management services need to perform expensive I/O operations asynchronously in the background, aiming to overlap the I/O with the application runtime. However, this may cause interference due to competition for resources: CPU, memory/network bandwidth. The advent of multi-core architectures has exacerbated this problem, as many I/O operations are issued concurrently, thereby competing not only with the application but also among themselves. Furthermore, the interference patterns can dynamically change as a response to variations in application behavior and I/O subsystems (e.g. multiple users sharing a parallel file system). Without a thorough understanding, I/O operations may perform suboptimally, potentially even worse than in the blocking case. To fill this gap, here we investigate the causes and consequences of interference due to asynchronous I/O on HPC systems. Specifically, we focus on multi-core CPUs and memory bandwidth, isolating the interference due to each resource. Then, we perform an in-depth study to explain the interplay and contention in a variety of resource sharing scenarios such as varying priority and number of background I/O threads and different I/O strategies: sendfile, read/write, mmap/write underlining trade-offs. The insights from this study are important both to enable guided optimizations of existing background I/O, as well as to open new opportunities to design advanced asynchronous I/O strategies.},

  doi          = {10.1177/10943420211016511},

  url          = {https://www.osti.gov/biblio/1831116},
  journal      = {International Journal of High Performance Computing Applications},

  issn         = {1094-3420},

  number       = 4,

  volume       = 35,

  place        = {United States},

  year         = {2021},

  month        = {5}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1177/10943420211016511

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

OmpSs: A PROPOSAL FOR PROGRAMMING HETEROGENEOUS MULTI-CORE ARCHITECTURES
journal, June 2011

Duran, Alejandro; AyguadÉ, Eduard; Badia, Rosa M.
Parallel Processing Letters, Vol. 21, Issue 02
https://doi.org/10.1142/S0129626411000151

HACC: extreme scaling and performance across diverse architectures
conference, January 2013

Habib, Salman; Morozov, Vitali; Frontiere, Nicholas
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13
https://doi.org/10.1145/2503210.2504566

Toward Scalable and Asynchronous Object-Centric Data Management for HPC
conference, May 2018

Tang, Houjun; Byna, Suren; Tessier, Francois
2018 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID)
https://doi.org/10.1109/CCGRID.2018.00026

GoldRush: resource efficient in situ scientific data analytics using fine-grained interference aware execution
conference, January 2013

Zheng, Fang; Yu, Hongfeng; Hantas, Can
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13
https://doi.org/10.1145/2503210.2503279

Understanding and Improving Computational Science Storage Access through Continuous Characterization
journal, October 2011

Carns, Philip; Harms, Kevin; Allcock, William
ACM Transactions on Storage, Vol. 7, Issue 3, p. 1-26
https://doi.org/10.1145/2027066.2027068

Improving collective I/O performance using threads
conference, January 1999

Dickens, P. M.; Thakur, R.
Proceedings 13th International Parallel Processing Symposium and 10th Symposium on Parallel and Distributed Processing. IPPS/SPDP 1999
https://doi.org/10.1109/IPPS.1999.760432

Exascale computing and big data
journal, June 2015

Reed, Daniel A.; Dongarra, Jack
Communications of the ACM, Vol. 58, Issue 7
https://doi.org/10.1145/2699414

Reducing I/O variability using dynamic I/O path characterization in petascale storage systems
journal, November 2016

Son, Seung Woo; Sehrish, Saba; Liao, Wei-keng
The Journal of Supercomputing, Vol. 73, Issue 5, p. 2069-2097
https://doi.org/10.1007/s11227-016-1904-7

Understanding the Effects of Communication and Coordination on Checkpointing at Scale
conference, November 2014

Ferreira, Kurt B.; Widener, Patrick; Levy, Scott
SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
https://doi.org/10.1109/SC.2014.77

Managing Variability in the IO Performance of Petascale Storage Systems
conference, November 2010

Lofstead, Jay; Zheng, Fang; Liu, Qing
2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
https://doi.org/10.1109/SC.2010.32

Light-weight parallel Python tools for earth system modeling workflows
conference, October 2015

Paul, Kevin; Mickelson, Sheri; Dennis, John M.
2015 IEEE International Conference on Big Data (Big Data)
https://doi.org/10.1109/BigData.2015.7363979

CHARM++: a portable concurrent object oriented system based on C++
conference, January 1993

Kale, Laxmikant V.; Krishnan, Sanjeev
Proceedings of the eighth annual conference on Object-oriented programming systems, languages, and applications - OOPSLA '93
https://doi.org/10.1145/165854.165874

Rucio: Scientific Data Management
journal, August 2019

Barisits, Martin; Beermann, Thomas; Berghaus, Frank
Computing and Software for Big Science, Vol. 3, Issue 1
https://doi.org/10.1007/s41781-019-0026-3

InterferenceRemoval: removing interference of disk access for MPI programs through data replication
conference, January 2010

Zhang, Xuechen; Jiang, Song
Proceedings of the 24th ACM International Conference on Supercomputing - ICS '10
https://doi.org/10.1145/1810085.1810116

Towards Asynchronous Many-Task in Situ Data Analysis Using Legion
conference, May 2016

Pebay, Philippe; Bennett, Janine C.; Hollman, David
2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
https://doi.org/10.1109/IPDPSW.2016.24

Optimal Cooperative Checkpointing for Shared High-Performance Computing Platforms
conference, May 2018

Herault, Thomas; Robert, Yves; Bouteiller, Aurelien
2018 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
https://doi.org/10.1109/IPDPSW.2018.00127

I/O-Aware Batch Scheduling for Petascale Computing Systems
conference, September 2015

Zhou, Zhou; Yang, Xu; Zhao, Dongfang
2015 IEEE International Conference on Cluster Computing (CLUSTER)
https://doi.org/10.1109/CLUSTER.2015.45

Tuning Object-Centric Data Management Systems for Large Scale Scientific Applications
conference, December 2019

Tang, Houjun; Byna, Suren; Bailey, Stephen
2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC)
https://doi.org/10.1109/HiPC.2019.00023

On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems
conference, May 2016

Yildiz, Orcun; Dorier, Matthieu; Ibrahim, Shadi
2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2016.50

StarPU: a unified platform for task scheduling on heterogeneous multicore architectures
journal, November 2010

Augonnet, Cédric; Thibault, Samuel; Namyst, Raymond
Concurrency and Computation: Practice and Experience, Vol. 23, Issue 2
https://doi.org/10.1002/cpe.1631

Harnessing Data Movement in Virtual Clusters for In-Situ Execution
journal, March 2019

Huang, Dan; Liu, Qing; Klasky, Scott
IEEE Transactions on Parallel and Distributed Systems, Vol. 30, Issue 3
https://doi.org/10.1109/TPDS.2018.2867879

Scheduling the I/O of HPC Applications Under Congestion
conference, May 2015

Gainaru, Ana; Aupy, Guillaume; Benoit, Anne
2015 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2015.116

VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale
conference, May 2019

Nicolae, Bogdan; Moody, Adam; Gonsiorowski, Elsa
2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2019.00099

Enterprise HPC storage systems
conference, September 2014

Petersen, Torben Kling; Fragalla, John
2014 IEEE High Performance Extreme Computing Conference (HPEC)
https://doi.org/10.1109/HPEC.2014.7040948

NiMC: Characterizing and Eliminating Network-Induced Memory Contention
conference, May 2016

Groves, Taylor; Grant, Ryan E.; Arnold, Dorian
2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2016.29

Storage challenges at Los Alamos National Lab
conference, April 2012

Bent, John; Grider, Gary; Kettering, Brett
https://doi.org/10.1109/MSST.2012.6232376

Transferring a petabyte in a day
journal, November 2018

Kettimuthu, Rajkumar; Liu, Zhengchun; Wheeler, David
Future Generation Computer Systems, Vol. 88
https://doi.org/10.1016/j.future.2018.05.051

DAOS and Friends: A Proposal for an Exascale Storage System
conference, November 2016

Lofstead, Jay; Jimenez, Ivo; Maltzahn, Carlos
SC16: International Conference for High Performance Computing, Networking, Storage and Analysis
https://doi.org/10.1109/SC.2016.49

Similar records in OSTI.GOV collections:

Efficient Machine Learning Approach for Optimizing Scientific Computing Applications on Emerging HPC Architectures

Thesis/Dissertation Arumugam, Kamesh

Efficient parallel implementations of scientific applications on multi-core CPUs with accelerators such as GPUs and Xeon Phis is challenging. This requires - exploiting the data parallel architecture of the accelerator along with the vector pipelines of modern x86 CPU architectures, load balancing, and efficient memory transfer between different devices. It is relatively easy to meet these requirements for highly structured scientific applications. In contrast, a number of scientific and engineering applications are unstructured. Getting performance on accelerators for these applications is extremely challenging because many of these applications employ irregular algorithms which exhibit data-dependent control-ow and irregular memory accesses. Furthermore,more »« less
https://doi.org/10.2172/1422715

Full Text Available
Dynamic Adaptable Asynchronous Progress Model for MPI RMA Multiphase Applications

Journal Article Si, Min; Pena, Antonio; Hammond, Jeff; ... - IEEE Transactions on Parallel and Distributed Systems

Casper is a process-based asynchronous progress model for MPI one-sided communication on multi-and many-core architectures. The one-sided communication is not truly one-sided in most MPI implementations: the target process still relies on software progress to complete incoming operations. Casper allows the user to specify an arbitrary number of cores dedicated to background ghost processes and transparently redirects the RMA operations to ghost processes by utilizing the PMPI redirection and MPI-3 shared-memory technologies. Although Casper benefits applications that suffer from lack of asynchronous progress, the operation redirection design might not support complex multiphase applications effectively, which often involve dynamically changing communicationmore »« less
https://doi.org/10.1109/TPDS.2018.2815568

Full Text Available
Final Technical Report - Proactive Data Containers for Scientific Storage

Technical Report Soumagne, Jerome; Warren, Richard; Mu, Jingqing; ...

Emerging HPC systems are expected to be deployed with an unprecedented level of complexity, due to a deep system memory/storage hierarchy and heterogeneity of the storage hardware. This hierarchy is expected to range from CPU cache through several levels of volatile memory to nonvolatile memory, traditional hard disks, and tape. Simple and efficient methods of data management and movement through this hierarchy is critical for scientific applications using exascale systems. Existing storage system and I/O (SSIO) technologies face severe challenges in dealing with these requirements. POSIX and MPI I/O standards that are the basis for existing I/O libraries and parallelmore »« less
https://doi.org/10.2172/1577855

Full Text Available
Scalability under a Power Bound using the GREMLINs Framework

Thesis/Dissertation Maiterth, Matthias

With the move towards exascale, system and software developers will have to deal with issues of extreme parallelism. The system properties affected most by the increase in node and core count are the shared resources on node and across the system. The increase in parallelism leads to reduced memory and bandwidth when regarding individual cores. Since power is a limiting factor for supercomputers, and power is not fully utilized in current systems, overprovisioning compute resources is a viable approach to maximized power utilization. To maximize system performance in regard to these changing conditions, it is necessary to understand how resourcemore »« less
https://doi.org/10.2172/1183550

Full Text Available
HPC Global File System Performance Analysis Using A Scientific-Application Derived Benchmark

Journal Article Borrill, Julian; Oliker, Leonid; Shalf, John; ... - Parallel Computing Systems&Applications

With the exponential growth of high-fidelity sensor and simulated data, the scientific community is increasingly reliant on ultrascale HPC resources to handle its data analysis requirements. However, to use such extreme computing power effectively, the I/O components must be designed in a balanced fashion, as any architectural bottleneck will quickly render the platform intolerably inefficient. To understand I/O performance of data-intensive applications in realistic computational settings, we develop a lightweight, portable benchmark called MADbench2, which is derived directly from a large-scale Cosmic Microwave Background (CMB) data analysis package. Our study represents one of the most comprehensive I/O analyses of modernmore »« less
Full Text Available

Similar Records

Title: Demystifying asynchronous I/O Interference in HPC applications

Abstract

Citation Formats

OmpSs: A PROPOSAL FOR PROGRAMMING HETEROGENEOUS MULTI-CORE ARCHITECTURES journal, June 2011

HACC: extreme scaling and performance across diverse architectures conference, January 2013

Toward Scalable and Asynchronous Object-Centric Data Management for HPC conference, May 2018

GoldRush: resource efficient in situ scientific data analytics using fine-grained interference aware execution conference, January 2013

Understanding and Improving Computational Science Storage Access through Continuous Characterization journal, October 2011

Improving collective I/O performance using threads conference, January 1999

Exascale computing and big data journal, June 2015

Reducing I/O variability using dynamic I/O path characterization in petascale storage systems journal, November 2016

Understanding the Effects of Communication and Coordination on Checkpointing at Scale conference, November 2014

Managing Variability in the IO Performance of Petascale Storage Systems conference, November 2010

Light-weight parallel Python tools for earth system modeling workflows conference, October 2015

CHARM++: a portable concurrent object oriented system based on C++ conference, January 1993

Rucio: Scientific Data Management journal, August 2019

InterferenceRemoval: removing interference of disk access for MPI programs through data replication conference, January 2010

Towards Asynchronous Many-Task in Situ Data Analysis Using Legion conference, May 2016

Optimal Cooperative Checkpointing for Shared High-Performance Computing Platforms conference, May 2018

I/O-Aware Batch Scheduling for Petascale Computing Systems conference, September 2015

Tuning Object-Centric Data Management Systems for Large Scale Scientific Applications conference, December 2019

On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems conference, May 2016

StarPU: a unified platform for task scheduling on heterogeneous multicore architectures journal, November 2010

Harnessing Data Movement in Virtual Clusters for In-Situ Execution journal, March 2019

Scheduling the I/O of HPC Applications Under Congestion conference, May 2015

VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale conference, May 2019

Enterprise HPC storage systems conference, September 2014

NiMC: Characterizing and Eliminating Network-Induced Memory Contention conference, May 2016

Storage challenges at Los Alamos National Lab conference, April 2012

Transferring a petabyte in a day journal, November 2018

DAOS and Friends: A Proposal for an Exascale Storage System conference, November 2016

OmpSs: A PROPOSAL FOR PROGRAMMING HETEROGENEOUS MULTI-CORE ARCHITECTURES
journal, June 2011

HACC: extreme scaling and performance across diverse architectures
conference, January 2013

Toward Scalable and Asynchronous Object-Centric Data Management for HPC
conference, May 2018

GoldRush: resource efficient in situ scientific data analytics using fine-grained interference aware execution
conference, January 2013

Understanding and Improving Computational Science Storage Access through Continuous Characterization
journal, October 2011

Improving collective I/O performance using threads
conference, January 1999

Exascale computing and big data
journal, June 2015

Reducing I/O variability using dynamic I/O path characterization in petascale storage systems
journal, November 2016

Understanding the Effects of Communication and Coordination on Checkpointing at Scale
conference, November 2014

Managing Variability in the IO Performance of Petascale Storage Systems
conference, November 2010

Light-weight parallel Python tools for earth system modeling workflows
conference, October 2015

CHARM++: a portable concurrent object oriented system based on C++
conference, January 1993

Rucio: Scientific Data Management
journal, August 2019

InterferenceRemoval: removing interference of disk access for MPI programs through data replication
conference, January 2010

Towards Asynchronous Many-Task in Situ Data Analysis Using Legion
conference, May 2016

Optimal Cooperative Checkpointing for Shared High-Performance Computing Platforms
conference, May 2018

I/O-Aware Batch Scheduling for Petascale Computing Systems
conference, September 2015

Tuning Object-Centric Data Management Systems for Large Scale Scientific Applications
conference, December 2019

On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems
conference, May 2016

StarPU: a unified platform for task scheduling on heterogeneous multicore architectures
journal, November 2010

Harnessing Data Movement in Virtual Clusters for In-Situ Execution
journal, March 2019

Scheduling the I/O of HPC Applications Under Congestion
conference, May 2015

VeloC: Towards High Performance Adaptive Asynchronous Checkpointing at Large Scale
conference, May 2019

Enterprise HPC storage systems
conference, September 2014

NiMC: Characterizing and Eliminating Network-Induced Memory Contention
conference, May 2016

Storage challenges at Los Alamos National Lab
conference, April 2012

Transferring a petabyte in a day
journal, November 2018

DAOS and Friends: A Proposal for an Exascale Storage System
conference, November 2016