skip to main content


Title: Modeling and Simulating Multiple Failure Masking enabled by Local Recovery for Stencil-based Applications at Extreme Scales

By obtaining multi-process hard failure resilience at the application level is a key challenge that must be overcome before the promise of exascale can be fully realized. Some previous work has shown that online global recovery can dramatically reduce the overhead of failures when compared to the more traditional approach of terminating the job and restarting it from the last stored checkpoint. If online recovery is performed in a local manner further scalability is enabled, not only due to the intrinsic lower costs of recovering locally, but also due to derived effects when using some application types. In this paper we model one such effect, namely multiple failure masking, that manifests when running Stencil parallel computations on an environment when failures are recovered locally. First, the delay propagation shape of one or multiple failures recovered locally is modeled to enable several analyses of the probability of different levels of failure masking under certain Stencil application behaviors. These results indicate that failure masking is an extremely desirable effect at scale which manifestation is more evident and beneficial as the machine size or the failure rate increase.
ORCiD logo [1] ;  [2] ;  [2] ;  [2] ;  [3] ;  [2] ;  [1]
  1. Rutgers Univ., Piscataway, NJ (United States)
  2. Sandia National Lab. (SNL-CA), Livermore, CA (United States)
  3. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Publication Date:
Report Number(s):
Journal ID: ISSN 1045-9219; 652588
Grant/Contract Number:
AC04-94AL85000; FG02-06ER54857; SC0007455
Accepted Manuscript
Journal Name:
IEEE Transactions on Parallel and Distributed Systems
Additional Journal Information:
Journal Volume: 28; Journal Issue: 10; Journal ID: ISSN 1045-9219
Research Org:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Sponsoring Org:
USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); National Science Foundation (NSF)
Country of Publication:
United States
97 MATHEMATICS AND COMPUTING; parallel processing; stencil computation; resilience; fault tolerance; failure masking; modeling
OSTI Identifier: