Orchestrating Fault Prediction with Live Migration and Checkpointing

Title: Orchestrating Fault Prediction with Live Migration and Checkpointing

Full Record
Other Related Research

Abstract

Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ~20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ~29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.

Authors:

Behera, Subhendu ^[1];

^[2]; Mueller, Frank ^[1];

^[2];

^[2]

North Carolina State University (NCSU), Raleigh
ORNL

Publication Date:: 2020-06-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1648858

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: International Symposium on High-Performance Parallel and Distributed Computing (HPDC '20) - Stokholm, , Sweden - 6/23/2020 4:00:00 AM-6/26/2020 4:00:00 AM

Country of Publication:: United States

Language:: English

Citation Formats


                    Behera, Subhendu, Wan, Lipeng, Mueller, Frank, Wolf, Matthew D., and Klasky, Scott A. Orchestrating Fault Prediction with Live Migration and Checkpointing.  United States: N. p., 2020. 
        Web.

Copy to clipboard


                    Behera, Subhendu, Wan, Lipeng, Mueller, Frank, Wolf, Matthew D., & Klasky, Scott A. Orchestrating Fault Prediction with Live Migration and Checkpointing.  United States.

Copy to clipboard


                    Behera, Subhendu, Wan, Lipeng, Mueller, Frank, Wolf, Matthew D., and Klasky, Scott A. Mon .  
        "Orchestrating Fault Prediction with Live Migration and Checkpointing".  United States.  https://www.osti.gov/servlets/purl/1648858.

Copy to clipboard


                    
@article{osti_1648858,

  title        = {Orchestrating Fault Prediction with Live Migration and Checkpointing},

  author       = {Behera, Subhendu and Wan, Lipeng and Mueller, Frank and Wolf, Matthew D. and Klasky, Scott A.},

  abstractNote = {Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ~20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ~29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/1648858},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2020},

  month        = {6}

}

Copy to clipboard

Conference:

View Conference (4.14 MB)

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Proactive Fault Tolerance for HPC with Xen Virtualization

Conference Nagarajan, Arun Babu ; Mueller, Frank ; Engelmann, Christian ; ...

with thousands of processors. At such large counts of compute nodes, faults are becoming common place. Current techniques to tolerate faults focus on reactive schemes to recover from faults and generally rely on a checkpoint/restart mechanism. Yet, in today's systems, node failures can often be anticipated by detecting a deteriorating health status. Instead of a reactive scheme for fault tolerance (FT), we are promoting a proactive one where processes automatically migrate from unhealthy nodes to healthy ones. Our approach relies on operating system virtualization techniques exemplied by but not limited to Xen. This paper contributes an automatic and transparent mechanismmore »« less
McrEngine: A Scalable Checkpointing System Using Data-Aware Aggregation and Compression

Journal Article Islam, Tanzima Zerin ; Mohror, Kathryn ; Bagchi, Saurabh ; ... - Scientific Programming

High performance computing (HPC) systems use checkpoint-restart to tolerate failures. Typically, applications store their states in checkpoints on a parallel file system (PFS). As applications scale up, checkpoint-restart incurs high overheads due to contention for PFS resources. The high overheads force large-scale applications to reduce checkpoint frequency, which means more compute time is lost in the event of failure. We alleviate this problem through a scalable checkpoint-restart system, mcrEngine. McrEngine aggregates checkpoints from multiple application processes with knowledge of the data semantics available through widely-used I/O libraries, e.g., HDF5 and netCDF, and compresses them. Our novel scheme improves compressibility ofmore »« less
Cited by 17
https://doi.org/10.1155/2013/341672
A Job Pause Service under LAM/MPI+BLCR for Transparent Fault Tolerance

Conference Wang, Chao ; Mueller, Frank ; Engelmann, Christian ; ...

Checkpoint/restart (C/R) has become a requirement for long-running jobs in large-scale clusters due to a meantime- to-failure (MTTF) in the order of hours. After a failure, C/R mechanisms generally require a complete restart of an MPI job from the last checkpoint. A complete restart, however, is unnecessary since all but one node are typically still alive. Furthermore, a restart may result in lengthy job requeuing even though the original job had not exceeded its time quantum. In this paper, we overcome these shortcomings. Instead of job restart, we have developed a transparent mechanism for job pause within LAM/MPI+BLCR. This mechanismmore »« less
https://doi.org/10.1109/IPDPS.2007.370307
Asynchronous Checkpoint Migration with MRNet in the Scalable Checkpoint / Restart Library

Conference Mohror, K ; Moody, A ; de Supinski, B R

Applications running on today's supercomputers tolerate failures by periodically saving their state in checkpoint files on stable storage, such as a parallel file system. Although this approach is simple, the overhead of writing the checkpoints can be prohibitive, especially for large-scale jobs. In this paper, we present initial results of an enhancement to our Scalable Checkpoint/Restart Library (SCR). We employ MRNet, a tree-based overlay network library, to transfer checkpoints from the compute nodes to the parallel file system asynchronously. This enhancement increases application efficiency by removing the need for an application to block while checkpoints are transferred to the parallelmore »« less
https://doi.org/10.1109/DSNW.2012.6264668

Full Text Available
TRIO: Burst Buffer Based I/O Orchestration

Conference Wang, Teng ; Oral, H Sarp ; Pritchard, Michael ; ...

The growing computing power on leadership HPC systems is often accompanied by ever-escalating failure rates. Checkpointing is a common defensive mechanism used by scientific applications for failure recovery. However, directly writing the large and bursty checkpointing dataset to parallel filesystem can incur significant I/O contention on storage servers. Such contention in turn degrades the raw bandwidth utilization of storage servers and prolongs the average job I/O time of concurrent applications. Recently burst buffer has been proposed as an intermediate layer to absorb the bursty I/O traffic from compute nodes to storage backend. But an I/O orchestration mechanism is still desiredmore »« less
Full Text Available

Similar Records