DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Scalable line and plane relaxation in a parallel structured multigrid solver

Abstract

The efficient solution of sparse, linear systems that arise through the discretization of partial differential equations remains a key challenge for a range of high performance scientific simulations. One approach for reducing data movement and improving performance is by exposing and exploiting structure in a problem through the use of robust structured multilevel solvers. By choosing coarsening that preserves the structure of the problem, these methods maintain efficient structured computation and communication throughout the multigrid hierarchy. However, when coarsening is not permitted to be dependent on the operator, anisotropy must be addressed by the smoother — producing error compatible for coarse-grid correction with structured coarsening. Here, the components required in a scalable parallel structured solver are described with a focus on memory and communication efficiency of robust smoothers. While the implementation of communication and memory reduction techniques in smoothers integrated in a complete 3D solver present a significant engineering challenge, a novel approach is proposed that addresses these challenges systematically through a change to the solver’s execution model. Enabled by user-level threading paired with a set of data and communication abstractions, this approach permits seamless aggregation of communication in plane smoothers — directly reusing code for a 2D distributed multilevelmore » cycle. Results show an effective reduction in communication costs for coarse-grid problems, and result in a speedup of 8.7x in smoothing routines shown in Fig. 12 using this approach. This produces a significant improvement to strong scalability while maintaining favorable weak scaling behavior. Finally, a parallel scaling study using a series of refined meshes is included that demonstrates the effectiveness of this approach in an application of interest.« less

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1];  [2]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  2. Univ. of Illinois at Urbana-Champaign, IL (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA); National Science Foundation (NSF)
OSTI Identifier:
1688768
Alternate Identifier(s):
OSTI ID: 1776467
Report Number(s):
LA-UR-20-23447
Journal ID: ISSN 0167-8191
Grant/Contract Number:  
89233218CNA000001; NA0002374; AC52-06NA25396; OCI-0725070; ACI-1238993
Resource Type:
Accepted Manuscript
Journal Name:
Parallel Computing
Additional Journal Information:
Journal Volume: 100; Journal ID: ISSN 0167-8191
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; parallel; line relaxation; plane relaxation communication; multigrid; structured

Citation Formats

Reisner, Andrew Ray, Moulton, John David, Berndt, Markus, and Olson, Luke N. Scalable line and plane relaxation in a parallel structured multigrid solver. United States: N. p., 2020. Web. doi:10.1016/j.parco.2020.102705.
Reisner, Andrew Ray, Moulton, John David, Berndt, Markus, & Olson, Luke N. Scalable line and plane relaxation in a parallel structured multigrid solver. United States. https://doi.org/10.1016/j.parco.2020.102705
Reisner, Andrew Ray, Moulton, John David, Berndt, Markus, and Olson, Luke N. Tue . "Scalable line and plane relaxation in a parallel structured multigrid solver". United States. https://doi.org/10.1016/j.parco.2020.102705. https://www.osti.gov/servlets/purl/1688768.
@article{osti_1688768,
title = {Scalable line and plane relaxation in a parallel structured multigrid solver},
author = {Reisner, Andrew Ray and Moulton, John David and Berndt, Markus and Olson, Luke N.},
abstractNote = {The efficient solution of sparse, linear systems that arise through the discretization of partial differential equations remains a key challenge for a range of high performance scientific simulations. One approach for reducing data movement and improving performance is by exposing and exploiting structure in a problem through the use of robust structured multilevel solvers. By choosing coarsening that preserves the structure of the problem, these methods maintain efficient structured computation and communication throughout the multigrid hierarchy. However, when coarsening is not permitted to be dependent on the operator, anisotropy must be addressed by the smoother — producing error compatible for coarse-grid correction with structured coarsening. Here, the components required in a scalable parallel structured solver are described with a focus on memory and communication efficiency of robust smoothers. While the implementation of communication and memory reduction techniques in smoothers integrated in a complete 3D solver present a significant engineering challenge, a novel approach is proposed that addresses these challenges systematically through a change to the solver’s execution model. Enabled by user-level threading paired with a set of data and communication abstractions, this approach permits seamless aggregation of communication in plane smoothers — directly reusing code for a 2D distributed multilevel cycle. Results show an effective reduction in communication costs for coarse-grid problems, and result in a speedup of 8.7x in smoothing routines shown in Fig. 12 using this approach. This produces a significant improvement to strong scalability while maintaining favorable weak scaling behavior. Finally, a parallel scaling study using a series of refined meshes is included that demonstrates the effectiveness of this approach in an application of interest.},
doi = {10.1016/j.parco.2020.102705},
journal = {Parallel Computing},
number = ,
volume = 100,
place = {United States},
year = {Tue Oct 20 00:00:00 EDT 2020},
month = {Tue Oct 20 00:00:00 EDT 2020}
}

Works referenced in this record:

A parallel solver for tridiagonal linear systems for distributed memory parallel computers
journal, November 1991


Argobots: A Lightweight Low-Level Threading and Tasking Framework
journal, March 2018

  • Seo, Sangmin; Amer, Abdelhalim; Balaji, Pavan
  • IEEE Transactions on Parallel and Distributed Systems, Vol. 29, Issue 3
  • DOI: 10.1109/TPDS.2017.2766062

Reducing Parallel Communication in Algebraic Multigrid through Sparsification
journal, January 2016

  • Bienz, Amanda; Falgout, Robert D.; Gropp, William
  • SIAM Journal on Scientific Computing, Vol. 38, Issue 5
  • DOI: 10.1137/15M1026341

A multilevel parallel solver for block tridiagonal and banded linear systems
journal, September 1990


On Stable Parallel Linear System Solvers
journal, January 1978


Black box multigrid
journal, December 1982


An Efficient Parallel Algorithm for the Solution of a Tridiagonal Linear System of Equations
journal, January 1973


Parallelization and vectorization aspects of the solution of tridiagonal linear systems
journal, May 1990


A Fast Direct Solution of Poisson's Equation Using Fourier Analysis
journal, January 1965


A Parallel Method for Tridiagonal Equations
journal, June 1981


Scaling Structured Multigrid to 500K+ Cores Through Coarse-Grid Redistribution
journal, January 2018

  • Reisner, Andrew; Olson, Luke N.; Moulton, J. David
  • SIAM Journal on Scientific Computing, Vol. 40, Issue 4
  • DOI: 10.1137/17M1146440

A parallel hybrid banded system solver: the SPIKE algorithm
journal, February 2006


Parallel Factorizations for Tridiagonal Matrices
journal, June 1993

  • Amodio, P.; Brugnano, L.; Politi, T.
  • SIAM Journal on Numerical Analysis, Vol. 30, Issue 3
  • DOI: 10.1137/0730041