skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Scalable domain decomposition solvers for stochastic PDEs in high performance computing

Abstract

Stochastic spectral finite element models of practical engineering systems may involve solutions of linear systems or linearized systems for non-linear problems with billions of unknowns. For stochastic modeling, it is therefore essential to design robust, parallel and scalable algorithms that can efficiently utilize high-performance computing to tackle such large-scale systems. Domain decomposition based iterative solvers can handle such systems. And though these algorithms exhibit excellent scalabilities, significant algorithmic and implementational challenges exist to extend them to solve extreme-scale stochastic systems using emerging computing platforms. Intrusive polynomial chaos expansion based domain decomposition algorithms are extended here to concurrently handle high resolution in both spatial and stochastic domains using an in-house implementation. Sparse iterative solvers with efficient preconditioners are employed to solve the resulting global and subdomain level local systems through multi-level iterative solvers. We also use parallel sparse matrix–vector operations to reduce the floating-point operations and memory requirements. Numerical and parallel scalabilities of these algorithms are presented for the diffusion equation having spatially varying diffusion coefficient modeled by a non-Gaussian stochastic process. Scalability of the solvers with respect to the number of random variables is also investigated.

Authors:
 [1];  [2];  [3];  [4];  [1]
  1. Carleton Univ., Ottawa, ON (Canada). Dept. of Civil and Environmental Engineering
  2. Sandia National Lab. (SNL-CA), Livermore, CA (United States)
  3. United States Naval Academy, Annapolis, MD (United States). Dept. of Aerospace Engineering
  4. Royal Military College of Canada, Kingston, ON (Canada). Dept. of Mechanical and Aerospace Engineering
Publication Date:
Research Org.:
Sandia National Lab. (SNL-CA), Livermore, CA (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1399891
Report Number(s):
SAND-2016-10429J
Journal ID: ISSN 0045-7825; 649571; TRN: US1702857
Grant/Contract Number:
AC04-94AL85000
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Computer Methods in Applied Mechanics and Engineering
Additional Journal Information:
Journal Name: Computer Methods in Applied Mechanics and Engineering; Journal ID: ISSN 0045-7825
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; 37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY; schur complement; parallel preconditioner; balancing domain decomposition by constraints; dual-primal finite element tearing and interconnect method; polynomial chaos expansion; coarse grid

Citation Formats

Desai, Ajit, Khalil, Mohammad, Pettit, Chris, Poirel, Dominique, and Sarkar, Abhijit. Scalable domain decomposition solvers for stochastic PDEs in high performance computing. United States: N. p., 2017. Web. doi:10.1016/j.cma.2017.09.006.
Desai, Ajit, Khalil, Mohammad, Pettit, Chris, Poirel, Dominique, & Sarkar, Abhijit. Scalable domain decomposition solvers for stochastic PDEs in high performance computing. United States. doi:10.1016/j.cma.2017.09.006.
Desai, Ajit, Khalil, Mohammad, Pettit, Chris, Poirel, Dominique, and Sarkar, Abhijit. 2017. "Scalable domain decomposition solvers for stochastic PDEs in high performance computing". United States. doi:10.1016/j.cma.2017.09.006.
@article{osti_1399891,
title = {Scalable domain decomposition solvers for stochastic PDEs in high performance computing},
author = {Desai, Ajit and Khalil, Mohammad and Pettit, Chris and Poirel, Dominique and Sarkar, Abhijit},
abstractNote = {Stochastic spectral finite element models of practical engineering systems may involve solutions of linear systems or linearized systems for non-linear problems with billions of unknowns. For stochastic modeling, it is therefore essential to design robust, parallel and scalable algorithms that can efficiently utilize high-performance computing to tackle such large-scale systems. Domain decomposition based iterative solvers can handle such systems. And though these algorithms exhibit excellent scalabilities, significant algorithmic and implementational challenges exist to extend them to solve extreme-scale stochastic systems using emerging computing platforms. Intrusive polynomial chaos expansion based domain decomposition algorithms are extended here to concurrently handle high resolution in both spatial and stochastic domains using an in-house implementation. Sparse iterative solvers with efficient preconditioners are employed to solve the resulting global and subdomain level local systems through multi-level iterative solvers. We also use parallel sparse matrix–vector operations to reduce the floating-point operations and memory requirements. Numerical and parallel scalabilities of these algorithms are presented for the diffusion equation having spatially varying diffusion coefficient modeled by a non-Gaussian stochastic process. Scalability of the solvers with respect to the number of random variables is also investigated.},
doi = {10.1016/j.cma.2017.09.006},
journal = {Computer Methods in Applied Mechanics and Engineering},
number = ,
volume = ,
place = {United States},
year = 2017,
month = 9
}

Journal Article:
Free Publicly Available Full Text
This content will become publicly available on September 21, 2018
Publisher's Version of Record

Save / Share:
  • Recent advances in high performance computing systems and sensing technologies motivate computational simulations with extremely high resolution models with capabilities to quantify uncertainties for credible numerical predictions. A two-level domain decomposition method is reported in this investigation to devise a linear solver for the large-scale system in the Galerkin spectral stochastic finite element method (SSFEM). In particular, a two-level scalable preconditioner is introduced in order to iteratively solve the large-scale linear system in the intrusive SSFEM using an iterative substructuring based domain decomposition solver. The implementation of the algorithm involves solving a local problem on each subdomain that constructs themore » local part of the preconditioner and a coarse problem that propagates information globally among the subdomains. The numerical and parallel scalabilities of the two-level preconditioner are contrasted with the previously developed one-level preconditioner for two-dimensional flow through porous media and elasticity problems with spatially varying non-Gaussian material properties. A distributed implementation of the parallel algorithm is carried out using MPI and PETSc parallel libraries. The scalabilities of the algorithm are investigated in a Linux cluster.« less
  • Abstract not provided.
  • The first part of this talk provides a basic introduction to the building blocks of domain decomposition solvers. Specific details are given for both the classical overlapping Schwarz (OS) algorithm and a recent iterative substructuring (IS) approach called balancing domain decomposition by constraints (BDDC). A more recent hybrid OS-IS approach is also described. The success of domain decomposition solvers depends critically on the coarse space. Similarities and differences between the coarse spaces for OS and BDDC approaches are discussed, along with how they can be obtained from discrete harmonic extensions. Connections are also made between coarse spaces and multiscale modelingmore » approaches from computational mechanics. As a specific example, details are provided on constructing coarse spaces for incompressible fluid problems. The next part of the talk deals with a variety of implementation details for domain decomposition solvers. These include mesh partitioning options, local and global solver options, reducing the coarse space dimension, dealing with constraint equations, residual weighting to accelerate the convergence of OS methods, and recycling of Krylov spaces to efficiently solve problems with multiple right hand sides. Some potential bottlenecks and remedies for domain decomposition solvers are also discussed. The final part of the talk concerns some recent theoretical advances, new algorithms, and open questions in the analysis of domain decomposition solvers. The focus will be primarily on the work of the speaker and his colleagues on elasticity, fluid mechanics, problems in H(curl), and the analysis of subdomains with irregular boundaries.« less
  • Characterizing the communication behavior of large-scale applications is a difficult and costly task due to code/system complexity and long execution times. While many tools to study this behavior have been developed, these approaches either aggregate information in a lossy way through high-level statistics or produce huge trace files that are hard to handle. We contribute an approach that provides orders of magnitude smaller, if not near-constant size, communication traces regardless of the number of nodes while preserving structural information. We introduce intra- and inter-node compression techniques of MPI events that are capable of extracting an application's communication structure. We furthermore » present a replay mechanism for the traces generated by our approach and discuss results of our implementation for BlueGene/L. Given this novel capability, we discuss its impact on communication tuning and beyond. To the best of our knowledge, such a concise representation of MPI traces in a scalable manner combined with deterministic MPI call replay are without any precedent.« less
  • Energy efficiency and resilience are two crucial challenges for HPC systems to reach exascale. While energy efficiency and resilience issues have been extensively studied individually, little has been done to understand the interplay between energy efficiency and resilience for HPC systems. Decreasing the supply voltage associated with a given operating frequency for processors and other CMOS-based components can significantly reduce power consumption. However, this often raises system failure rates and consequently increases application execution time. In this work, we present an energy saving undervolting approach that leverages the mainstream resilience techniques to tolerate the increased failures caused by undervolting.