Checkpointing Strategies for Shared High-Performance Computing Platforms

Herault, Thomas; Robert, Yves; Bouteiller, Aurelien; Arnold, Dorian; Ferreira, Kurt Brian; George, George; Dongarra, Jack

doi:10.15803/ijnc.9.1_28

Title: Checkpointing Strategies for Shared High-Performance Computing Platforms

Full Record
Other Related Research

Abstract

Input/output (I/O) from various sources often contend for scarcely available bandwidth. For example, checkpoint/restart (CR) protocols can help to ensure application progress in failure-prone environments. However, CR I/O alongside an application's normal, requisite I/O can increase I/O contention and might negatively impact performance. In this work, we consider different aspects (system-level scheduling policies and hardware) that optimize the overall performance of concurrently executing CR-based applications that share I/O resources. We provide a theoretical model and derive a set of necessary constraints to minimize the global waste on a given platform. Our results demonstrate that Young/Daly's optimal checkpoint interval, despite providing a sensible metric for a single, undisturbed application, is not sufficient to optimally address resource contention at scale. We show that by combining optimal checkpointing periods with contention-aware system-level I/O scheduling strategies, we can significantly improve overall application performance and maximize the platform throughput. Finally, we evaluate how specialized hardware, namely burst buffers, may help to mitigate the I/O contention problem. Altogether, these results provide critical analysis and direct guidance on how to design efficient, CR ready, large -scale platforms without a large investment in the I/O subsystem.

Authors:

Herault, Thomas ^[1]; Robert, Yves ^[2]; Bouteiller, Aurelien ^[1]; Arnold, Dorian ^[3]; Ferreira, Kurt Brian ^[4]; George, George ^[1]; Dongarra, Jack ^[5]

Univ. of Tennessee, Knoxville, TN (United States)
ENS Lyon (France); Univ. of Tennessee, Knoxville, TN (United States)
Emory Univ., Atlanta, GA (United States)
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Univ. of Manchester (United Kingdom); Univ. of Tennessee, Knoxville, TN (United States)

Publication Date:: 2019-01-01

Research Org.:: Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1492861

Report Number(s):: SAND-2018-12751J
Journal ID: ISSN 2185-2839; 669702

Grant/Contract Number:: AC04-94AL85000

Resource Type:: Accepted Manuscript

Journal Name:: International Journal of Networking and Computing

Additional Journal Information:: Journal Volume: 9; Journal Issue: 1; Journal ID: ISSN 2185-2839

Publisher:: IJNC Editorial Committee

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Herault, Thomas, Robert, Yves, Bouteiller, Aurelien, Arnold, Dorian, Ferreira, Kurt Brian, George, George, and Dongarra, Jack. Checkpointing Strategies for Shared High-Performance Computing Platforms.  United States: N. p., 2019. 
Web.  doi:10.15803/ijnc.9.1_28.

Copy to clipboard


                    Herault, Thomas, Robert, Yves, Bouteiller, Aurelien, Arnold, Dorian, Ferreira, Kurt Brian, George, George, & Dongarra, Jack. Checkpointing Strategies for Shared High-Performance Computing Platforms.  United States.  https://doi.org/10.15803/ijnc.9.1_28

Copy to clipboard


                    Herault, Thomas, Robert, Yves, Bouteiller, Aurelien, Arnold, Dorian, Ferreira, Kurt Brian, George, George, and Dongarra, Jack. Tue .  
"Checkpointing Strategies for Shared High-Performance Computing Platforms".  United States.  https://doi.org/10.15803/ijnc.9.1_28.  https://www.osti.gov/servlets/purl/1492861.

Copy to clipboard


                    
@article{osti_1492861,

  title        = {Checkpointing Strategies for Shared High-Performance Computing Platforms},

  author       = {Herault, Thomas and Robert, Yves and Bouteiller, Aurelien and Arnold, Dorian and Ferreira, Kurt Brian and George, George and Dongarra, Jack},

  abstractNote = {Input/output (I/O) from various sources often contend for scarcely available bandwidth. For example, checkpoint/restart (CR) protocols can help to ensure application progress in failure-prone environments. However, CR I/O alongside an application's normal, requisite I/O can increase I/O contention and might negatively impact performance. In this work, we consider different aspects (system-level scheduling policies and hardware) that optimize the overall performance of concurrently executing CR-based applications that share I/O resources. We provide a theoretical model and derive a set of necessary constraints to minimize the global waste on a given platform. Our results demonstrate that Young/Daly's optimal checkpoint interval, despite providing a sensible metric for a single, undisturbed application, is not sufficient to optimally address resource contention at scale. We show that by combining optimal checkpointing periods with contention-aware system-level I/O scheduling strategies, we can significantly improve overall application performance and maximize the platform throughput. Finally, we evaluate how specialized hardware, namely burst buffers, may help to mitigate the I/O contention problem. Altogether, these results provide critical analysis and direct guidance on how to design efficient, CR ready, large -scale platforms without a large investment in the I/O subsystem.},

  doi          = {10.15803/ijnc.9.1_28},

  journal      = {International Journal of Networking and Computing},

  number       = 1,

  volume       = 9,

  place        = {United States},

  year         = {2019},

  month        = {1}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.15803/ijnc.9.1_28

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

A checkpoint compression study for high-performance computing systems

Journal Article Ibtesham, Dewan ; Ferreira, Kurt B. ; Arnold, Dorian - International Journal of High Performance Computing Applications

As high-performance computing systems continue to increase in size and complexity, higher failure rates and increased overheads for checkpoint/restart (CR) protocols have raised concerns about the practical viability of CR protocols for future systems. Previously, compression has proven to be a viable approach for reducing checkpoint data volumes and, thereby, reducing CR protocol overhead leading to improved application performance. In this article, we further explore compression-based CR optimization by exploring its baseline performance and scaling properties, evaluating whether improved compression algorithms might lead to even better application performance and comparing checkpoint compression against and alongside other software- and hardware-based optimizations.more »« less
https://doi.org/10.1177/1094342015570921

Full Text Available
Checkpointing Shared Memory Programs at the Application-level

Conference Bronevetsky, G ; Schulz, M ; Szwed, P ; ...

Trends in high-performance computing are making it necessary for long-running applications to tolerate hardware faults. The most commonly used approach is checkpoint and restart(CPR)-the state of the computation is saved periodically on disk, and when a failure occurs, the computation is restarted from the last saved state. At present, it is the responsibility of the programmer to instrument applications for CPR. Our group is investigating the use of compiler technology to instrument codes to make them self-checkpointing and self-restarting, thereby providing an automatic solution to the problem of making long-running scientific applications resilient to hardware faults. Our previous work focusedmore »« less
Full Text Available
SPARC: Demonstrate burst-buffer-based checkpoint/restart on ATS-1.

Technical Report Oldfield, Ron A. ; Ulmer, Craig D. ; Widener, Patrick ; ...

Recent high-performance computing (HPC) platforms such as the Trinity Advanced Technology System (ATS-1) feature burst buffer resources that can have a dramatic impact on an application’s I/O performance. While these non-volatile memory (NVM) resources provide a new tier in the storage hierarchy, developers must find the right way to incorporate the technology into their applications in order to reap the benefits. Similar to other laboratories, Sandia is actively investigating ways in which these resources can be incorporated into our existing libraries and workflows without burdening our application developers with excessive, platform-specific details. This FY18Q1 milestone summaries our progress in adaptingmore »« less
https://doi.org/10.2172/1417577

Full Text Available
Keeping checkpoint/restart viable for exascale systems.

Technical Report Riesen, Rolf E ; Bridges, Patrick G ; Stearley, Jon R ; ...

Next-generation exascale systems, those capable of performing a quintillion (10{sup 18}) operations per second, are expected to be delivered in the next 8-10 years. These systems, which will be 1,000 times faster than current systems, will be of unprecedented scale. As these systems continue to grow in size, faults will become increasingly common, even over the course of small calculations. Therefore, issues such as fault tolerance and reliability will limit application scalability. Current techniques to ensure progress across faults like checkpoint/restart, the dominant fault tolerance mechanism for the last 25 years, are increasingly problematic at the scales of future systemsmore »« less
https://doi.org/10.2172/1029780

Full Text Available
McrEngine: A Scalable Checkpointing System Using Data-Aware Aggregation and Compression

Journal Article Islam, Tanzima Zerin ; Mohror, Kathryn ; Bagchi, Saurabh ; ... - Scientific Programming

High performance computing (HPC) systems use checkpoint-restart to tolerate failures. Typically, applications store their states in checkpoints on a parallel file system (PFS). As applications scale up, checkpoint-restart incurs high overheads due to contention for PFS resources. The high overheads force large-scale applications to reduce checkpoint frequency, which means more compute time is lost in the event of failure. We alleviate this problem through a scalable checkpoint-restart system, mcrEngine. McrEngine aggregates checkpoints from multiple application processes with knowledge of the data semantics available through widely-used I/O libraries, e.g., HDF5 and netCDF, and compresses them. Our novel scheme improves compressibility ofmore »« less
Cited by 18
https://doi.org/10.1155/2013/341672

Similar Records