skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Partial differential equations preconditioner resilient to soft and hard faults

Abstract

We present a domain-decomposition-based preconditioner for the solution of partial differential equations (PDEs) that is resilient to both soft and hard faults. The algorithm reformulates the PDE as a sampling problem, followed by a solution update through data manipulation that is resilient to both soft and hard faults. This reformulation allows us to recast the problem as a set of independent tasks, and exploit data locality to reduce global communication. We discuss two different parallel implementations: (a) a single program multiple data (SPMD) version based on a one-to-one mapping between subdomain and MPI processes responsible for both state and computation; and (b) an asynchronous server–client implementation where all state information is held by the servers and clients are designed solely as computational units. We present a scalability comparison of both implementations under nominal conditions, showing efficiency within ~80% for up to 12,000 cores. We present a resilience analysis under different fault scenarios based on the server–client implementation. This framework provides resiliency to hard faults such that if a client crashes, it stops asking for work, and the servers simply distribute the work among all of the other clients alive. Erroneous subdomain solves (e.g. due to soft faults) appear as corruptedmore » data, which is either rejected if that causes a task to fail, or is seamlessly filtered out during the regression stage through a suitable noise model. Three different types of faults are modeled: hard faults modeling nodes (or clients) crashing; soft faults occurring during the communication of the tasks between server and clients; and soft faults occurring during task execution. We demonstrate the resiliency of the approach for a 2D elliptic PDE, and explore the effect of the faults at various failure rates.« less

Authors:
 [1];  [1];  [1];  [2];  [1];  [2];  [2];  [1]
  1. Sandia National Lab. (SNL-CA), Livermore, CA (United States)
  2. Duke Univ., Durham, NC (United States)
Publication Date:
Research Org.:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC); Univ. of California, Oakland, CA (United States); Lockheed Martin Corpration, Litteton, CO (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1544016
Grant/Contract Number:  
AC02-05CH11231; AC04-94AL85000
Resource Type:
Accepted Manuscript
Journal Name:
International Journal of High Performance Computing Applications
Additional Journal Information:
Journal Volume: 32; Journal Issue: 5; Journal ID: ISSN 1094-3420
Publisher:
SAGE
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Computer Science

Citation Formats

Rizzi, F., Morris, K., Sargsyan, K., Mycek, P., Safta, C., Le Maître, O., Knio, O., and Debusschere, B. Partial differential equations preconditioner resilient to soft and hard faults. United States: N. p., 2017. Web. doi:10.1177/1094342016684975.
Rizzi, F., Morris, K., Sargsyan, K., Mycek, P., Safta, C., Le Maître, O., Knio, O., & Debusschere, B. Partial differential equations preconditioner resilient to soft and hard faults. United States. doi:10.1177/1094342016684975.
Rizzi, F., Morris, K., Sargsyan, K., Mycek, P., Safta, C., Le Maître, O., Knio, O., and Debusschere, B. Sun . "Partial differential equations preconditioner resilient to soft and hard faults". United States. doi:10.1177/1094342016684975. https://www.osti.gov/servlets/purl/1544016.
@article{osti_1544016,
title = {Partial differential equations preconditioner resilient to soft and hard faults},
author = {Rizzi, F. and Morris, K. and Sargsyan, K. and Mycek, P. and Safta, C. and Le Maître, O. and Knio, O. and Debusschere, B.},
abstractNote = {We present a domain-decomposition-based preconditioner for the solution of partial differential equations (PDEs) that is resilient to both soft and hard faults. The algorithm reformulates the PDE as a sampling problem, followed by a solution update through data manipulation that is resilient to both soft and hard faults. This reformulation allows us to recast the problem as a set of independent tasks, and exploit data locality to reduce global communication. We discuss two different parallel implementations: (a) a single program multiple data (SPMD) version based on a one-to-one mapping between subdomain and MPI processes responsible for both state and computation; and (b) an asynchronous server–client implementation where all state information is held by the servers and clients are designed solely as computational units. We present a scalability comparison of both implementations under nominal conditions, showing efficiency within ~80% for up to 12,000 cores. We present a resilience analysis under different fault scenarios based on the server–client implementation. This framework provides resiliency to hard faults such that if a client crashes, it stops asking for work, and the servers simply distribute the work among all of the other clients alive. Erroneous subdomain solves (e.g. due to soft faults) appear as corrupted data, which is either rejected if that causes a task to fail, or is seamlessly filtered out during the regression stage through a suitable noise model. Three different types of faults are modeled: hard faults modeling nodes (or clients) crashing; soft faults occurring during the communication of the tasks between server and clients; and soft faults occurring during task execution. We demonstrate the resiliency of the approach for a 2D elliptic PDE, and explore the effect of the faults at various failure rates.},
doi = {10.1177/1094342016684975},
journal = {International Journal of High Performance Computing Applications},
number = 5,
volume = 32,
place = {United States},
year = {2017},
month = {1}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Algorithm-based fault tolerance applied to high performance computing
journal, April 2009

  • Bosilca, George; Delmas, Rémi; Dongarra, Jack
  • Journal of Parallel and Distributed Computing, Vol. 69, Issue 4
  • DOI: 10.1016/j.jpdc.2008.12.002

Understanding the propagation of hard errors to software and implications for resilient system design
journal, March 2008

  • Li, Man-Lap; Ramachandran, Pradeep; Sahoo, Swarup Kumar
  • ACM SIGOPS Operating Systems Review, Vol. 42, Issue 2
  • DOI: 10.1145/1353535.1346315

A Large-Scale Study of Failures in High-Performance Computing Systems
journal, October 2010

  • Schroeder, Bianca; Gibson, Garth A.
  • IEEE Transactions on Dependable and Secure Computing, Vol. 7, Issue 4
  • DOI: 10.1109/TDSC.2009.4

Fault Resilient Domain Decomposition Preconditioner for PDEs
journal, January 2015

  • Sargsyan, Khachik; Rizzi, Francesco; Mycek, Paul
  • SIAM Journal on Scientific Computing, Vol. 37, Issue 5
  • DOI: 10.1137/15M1014474

Toward Exascale Resilience
journal, September 2009

  • Cappello, Franck; Geist, Al; Gropp, Bill
  • The International Journal of High Performance Computing Applications, Vol. 23, Issue 4
  • DOI: 10.1177/1094342009347767

Error log analysis: statistical modeling and heuristic trend analysis
journal, January 1990

  • Lin, T. -T. Y.; Siewiorek, D. P.
  • IEEE Transactions on Reliability, Vol. 39, Issue 4
  • DOI: 10.1109/24.58720