skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Shrink or Substitute: Handling Process Failures in HPC Systems Using In-Situ Recovery

Abstract

Efficient utilization of today's high-performance computing (HPC) systems with complex software and hardware components requires that the HPC applications are designed to tolerate process failures at runtime. With low mean-time-to-failure (MTTF) of current and future HPC systems, long running simulations on these systems requires capabilities for gracefully handling process failures by the applications themselves. In this paper, we explore the use of fault tolerance extensions to Message Passing Interface (MPI) called user-level failure mitigation (ULFM) for handling process failures without the need to discard the progress made by the application. We explore two alternative recovery strategies, which use ULFM along with application-driven in-memory checkpointing. In the first case, the application is recovered with only the surviving processes, and in the second case, spares are used to replace the failed processes, such that the original configuration of the application is restored. Our experimental results demonstrate that graceful degradation is a viable alternative for recovery in environments where spares may not be available.

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]
  1. ORNL
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1454399
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: Euromicro International Conference on Parallel, Distributed and Network-based Processing (PDP) - Cambridge, , United Kingdom - 3/21/2018 8:00:00 AM-3/23/2018 8:00:00 AM
Country of Publication:
United States
Language:
English

Citation Formats

Ashraf, Rizwan A., Hukerikar, Saurabh, and Engelmann, Christian. Shrink or Substitute: Handling Process Failures in HPC Systems Using In-Situ Recovery. United States: N. p., 2018. Web. doi:10.1109/PDP2018.2018.00032.
Ashraf, Rizwan A., Hukerikar, Saurabh, & Engelmann, Christian. Shrink or Substitute: Handling Process Failures in HPC Systems Using In-Situ Recovery. United States. doi:10.1109/PDP2018.2018.00032.
Ashraf, Rizwan A., Hukerikar, Saurabh, and Engelmann, Christian. Thu . "Shrink or Substitute: Handling Process Failures in HPC Systems Using In-Situ Recovery". United States. doi:10.1109/PDP2018.2018.00032. https://www.osti.gov/servlets/purl/1454399.
@article{osti_1454399,
title = {Shrink or Substitute: Handling Process Failures in HPC Systems Using In-Situ Recovery},
author = {Ashraf, Rizwan A. and Hukerikar, Saurabh and Engelmann, Christian},
abstractNote = {Efficient utilization of today's high-performance computing (HPC) systems with complex software and hardware components requires that the HPC applications are designed to tolerate process failures at runtime. With low mean-time-to-failure (MTTF) of current and future HPC systems, long running simulations on these systems requires capabilities for gracefully handling process failures by the applications themselves. In this paper, we explore the use of fault tolerance extensions to Message Passing Interface (MPI) called user-level failure mitigation (ULFM) for handling process failures without the need to discard the progress made by the application. We explore two alternative recovery strategies, which use ULFM along with application-driven in-memory checkpointing. In the first case, the application is recovered with only the surviving processes, and in the second case, spares are used to replace the failed processes, such that the original configuration of the application is restored. Our experimental results demonstrate that graceful degradation is a viable alternative for recovery in environments where spares may not be available.},
doi = {10.1109/PDP2018.2018.00032},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {3}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: