Failure recovery for bulk synchronous applications with MPI stages

Journal Article · 27 February 2019 · Parallel Computing

DOI: https://doi.org/10.1016/j.parco.2019.02.007 · OSTI ID:1784608

Sultana, Nawrin ^[1]; Rüfenacht, Martin ^[2]; Skjellum, Anthony ^[2]; Laguna, Ignacio ^[3]; Mohror, Kathryn ^[3]

---

1. Auburn Univ., AL (United States)
2. Univ. of Tennessee at Chattanooga, TN (United States)
3. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

When an MPI program experiences a failure, the most common recovery approach is to restart all processes from a previous checkpoint and to re-queue the entire job. A disadvantage of this method is that, although the failure occurred within the main application loop, live processes must start again from the beginning of the program, along with new replacement processes—this incurs unnecessary overhead for live processes. To avoid such overheads and concomitant delays, we introduce the concept of “MPI Stages.” MPI Stages saves internal MPI state in a separate checkpoint in conjunction with application state. Upon failure, both MPI and application state are recovered, respectively, from their last synchronous checkpoints and continue without restarting the overall MPI job. Live processes roll back only a few iterations within the main loop instead of rolling back to the beginning of the program, while a replacement of failed process restarts and reintegrates, thereby achieving faster failure recovery. As a result, this approach integrates well with large-scale, bulk synchronous applications and checkpoint/restart.

• Research Organization: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
• Sponsoring Organization: National Science Foundation; USDOE; USDOE National Nuclear Security Administration (NNSA)
• Grant/Contract Number: AC52-07NA27344
• OSTI ID: 1784608
• Report Number(s): LLNL-JRNL--759751; 948325
• Journal Information: Parallel Computing, Journal Name: Parallel Computing Journal Issue: na Vol. 84; ISSN 0167-8191
• Publisher: ElsevierCopyright Statement
• Country of Publication: United States
• Language: English