A communication-avoiding 3D sparse triangular solver

Sao, Piyush; Kannan, Ramakrishnan {ramki}; Li, Xiaoye Sherry; Vuduc, Richard

doi:10.1145/3330345.3330357

Title: A communication-avoiding 3D sparse triangular solver

Abstract

We present a novel distributed memory algorithm to improve the strong scalability of the solution of a sparse triangular system. This operation appears in the solve phase of direct methods for solving general sparse linear systems, Ax = b. Our 3D sparse triangular solver employs several techniques, including a 3D MPI process grid, elimination tree parallelism, and data replication, all of which reduce the per-process communication when combined. We present analytical models to understand the communication cost of our algorithm and show that our 3D sparse triangular solver can reduce the per-process communication volume asymptotically by a factor of O(n1/4) and O(n1/6) for problems arising from the finite element discretizations of 2D "planar" and 3D "non-planar" PDEs, respectively. We implement our algorithm for use in SuperLU_DIST3D, using a hybrid MPI+OpenMP programming model. Our 3D triangular solve algorithm, when run on 12k cores of Cray XC30, outperforms the current state-of-the-art 2D algorithm by 7.2x for planar and 2.7x for the non-planar sparse matrices, respectively.

Authors:

^[1];

^[1]; Li, Xiaoye Sherry ^[2]; Vuduc, Richard ^[3]

ORNL
Lawrence Berkeley National Laboratory (LBNL)
Georgia Institute of Technology, Atlanta

Publication Date:: 2019-06-01

Research Org.:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1558528

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: International Conference on Supercomputing (ICS 2019) - Phoenix, Arizona, United States of America - 6/26/2019 8:00:00 AM-6/28/2019 8:00:00 AM

Country of Publication:: United States

Language:: English

Citation Formats


                    Sao, Piyush, Kannan, Ramakrishnan, Li, Xiaoye Sherry, and Vuduc, Richard. A communication-avoiding 3D sparse triangular solver.  United States: N. p., 2019. 
        Web.  doi:10.1145/3330345.3330357.

Copy to clipboard


                    Sao, Piyush, Kannan, Ramakrishnan, Li, Xiaoye Sherry, & Vuduc, Richard. A communication-avoiding 3D sparse triangular solver.  United States.  https://doi.org/10.1145/3330345.3330357

Copy to clipboard


                    Sao, Piyush, Kannan, Ramakrishnan, Li, Xiaoye Sherry, and Vuduc, Richard. 2019.  
        "A communication-avoiding 3D sparse triangular solver".  United States.  https://doi.org/10.1145/3330345.3330357.  https://www.osti.gov/servlets/purl/1558528.

Copy to clipboard


                    
@article{osti_1558528,

  title        = {A communication-avoiding 3D sparse triangular solver},

  author       = {Sao, Piyush and Kannan, Ramakrishnan and Li, Xiaoye Sherry and Vuduc, Richard},

  abstractNote = {We present a novel distributed memory algorithm to improve the strong scalability of the solution of a sparse triangular system. This operation appears in the solve phase of direct methods for solving general sparse linear systems, Ax = b. Our 3D sparse triangular solver employs several techniques, including a 3D MPI process grid, elimination tree parallelism, and data replication, all of which reduce the per-process communication when combined. We present analytical models to understand the communication cost of our algorithm and show that our 3D sparse triangular solver can reduce the per-process communication volume asymptotically by a factor of O(n1/4) and O(n1/6) for problems arising from the finite element discretizations of 2D "planar" and 3D "non-planar" PDEs, respectively. We implement our algorithm for use in SuperLU_DIST3D, using a hybrid MPI+OpenMP programming model. Our 3D triangular solve algorithm, when run on 12k cores of Cray XC30, outperforms the current state-of-the-art 2D algorithm by 7.2x for planar and 2.7x for the non-planar sparse matrices, respectively.},

  doi          = {10.1145/3330345.3330357},

  url          = {https://www.osti.gov/biblio/1558528},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2019},

  month        = {6}

}

Copy to clipboard

Conference:

View Conference (1.42 MB)

https://doi.org/10.1145/3330345.3330357

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Nested Dissection of a Regular Finite Element Mesh
journal, April 1973

George, Alan
SIAM Journal on Numerical Analysis, Vol. 10, Issue 2
https://doi.org/10.1137/0710032

Communication-Avoiding Parallel Sparse-Dense Matrix-Matrix Multiplication
conference, May 2016

Koanantakool, Penporn; Azad, Ariful; Buluc, Aydin
2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2016.117

On asynchronous iterations
journal, November 2000

Frommer, Andreas; Szyld, Daniel B.
Journal of Computational and Applied Mathematics, Vol. 123, Issue 1-2
https://doi.org/10.1016/S0377-0427(00)00409-X

Communication results for parallel sparse Cholesky factorization on a hypercube
journal, May 1989

George, Alan; Liu, Joseph W. H.; Ng, Esmond
Parallel Computing, Vol. 10, Issue 3
https://doi.org/10.1016/0167-8191(89)90101-4

Structure-adaptive parallel solution of sparse triangular linear systems
journal, October 2014

Totoni, Ehsan; Heath, Michael T.; Kale, Laxmikant V.
Parallel Computing, Vol. 40, Issue 9
https://doi.org/10.1016/j.parco.2014.06.006

Convergence Models and Surprising Results for the Asynchronous Jacobi Method
conference, May 2018

Chow, Edmond; Chow, Edmond
2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2018.00103

Highly scalable parallel algorithms for sparse matrix factorization
journal, May 1997

Gupta, A.; Karypis, G.; Kumar, V.
IEEE Transactions on Parallel and Distributed Systems, Vol. 8, Issue 5
https://doi.org/10.1109/71.598277

A Communication-Avoiding 3D LU Factorization Algorithm for Sparse Matrices
conference, May 2018

Sao, Piyush; Li, Xiaoye Sherry; Vuduc, Richard
2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2018.00100

Integrated Model, Batch, and Domain Parallelism in Training Neural Networks
conference, July 2018

Gholami, Amir; Azad, Ariful; Jin, Peter
Proceedings of the 30th on Symposium on Parallelism in Algorithms and Architectures
https://doi.org/10.1145/3210377.3210394

Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations
conference, May 2017

Wicky, Tobias; Solomonik, Edgar; Hoefler, Torsten
2017 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
https://doi.org/10.1109/IPDPS.2017.104

Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication
journal, January 2016

Azad, Ariful; Ballard, Grey; Buluç, Aydin
SIAM Journal on Scientific Computing, Vol. 38, Issue 6
https://doi.org/10.1137/15M104253X

Efficient Parallel Sparse Triangular Solution Using Selective Inversion
journal, March 1998

Raghavan, Padma
Parallel Processing Letters, Vol. 08, Issue 01
https://doi.org/10.1142/S0129626498000067

Trading Replication for Communication in Parallel Distributed-Memory Dense Solvers
journal, March 2002

Irony, Dror; Toledo, Sivan
Parallel Processing Letters, Vol. 12, Issue 01
https://doi.org/10.1142/S0129626402000847

A New Data-Mapping Scheme for Latency-Tolerant Distributed Sparse Triangular Solution
conference, January 2002

Teranishi, K.; Raghavan, P.
ACM/IEEE SC 2002 Conference (SC'02)
https://doi.org/10.1109/SC.2002.10020

Parallel Algorithms for Sparse Linear Systems
journal, September 1991

Heath, Michael T.; Ng, Esmond; Peyton, Barry W.
SIAM Review, Vol. 33, Issue 3
https://doi.org/10.1137/1033099

Avoiding communication in sparse matrix computations
conference, April 2008

Demmel, James; Hoemmen, Mark; Mohiyuddin, Marghoob
Distributed Processing Symposium (IPDPS), 2008 IEEE International Symposium on Parallel and Distributed Processing
https://doi.org/10.1109/IPDPS.2008.4536305

Similar records in OSTI.GOV collections:

A communication-avoiding 3D algorithm for sparse LU factorization on heterogeneous systems

Journal Article Sao, Piyush; Li, Xiaoye; Vuduc, Richard - Journal of Parallel and Distributed Computing

We propose a new algorithm to improve the strong scalability of right-looking sparse LU factorization on distributed memory systems. Our 3D algorithm for sparse LU uses a three-dimensional MPI process grid, exploits elimination tree parallelism, and trades off increased memory for reduced per-process communication. We also analyze the asymptotic improvements for planar graphs (e.g., those arising from 2D grid or mesh discretizations) and certain non-planar graphs (specifically for 3D grids and meshes). For a planar graph with $$n$$ vertices, our algorithm reduces communication volume asymptotically in $$n$$ by a factor of $$\mathscr{O}$$ $$\Big(\sqrt{log \ n}\Big)$$ and latency by a factormore »« less
Cited by 2
https://doi.org/10.1016/j.jpdc.2019.03.004

Full Text Available
A Communication-Avoiding 3D LU Factorization Algorithm for Sparse Matrices

Conference Sao, Piyush; Li, Xiaoye; Vuduc, Richard

We propose a new algorithm to improve the strong scalability of right-looking sparse LU factorization on distributed memory systems. Our 3D sparse LU algorithm uses a three-dimensional MPI process grid, aggressively exploits elimination tree parallelism and trades off increased memory for reduced per-process communication. We also analyze the asymptotic improvements for planar graphs (e.g., from 2D grid or mesh domains) and certain non-planar graphs (specifically for 3D grids and meshes). For planar graphs with n vertices, our algorithm reduces communication volume asymptotically in n by a factor of O{log n} and latency by a factor of O{log n}. For non-planarmore »« less
https://doi.org/10.1109/IPDPS.2018.00100
Highly scalable distributed-memory sparse triangular solution algorithms.

Conference Liu, Yang; Jacquelin, Mathias; Ghysels, Pieter; ...

This paper presents a highly efficient distributed-memory parallel sparse triangular solver. The triangular solution phase is often performed following factorization phase in the sparse linear solvers and has become increasingly computationally expensive for direct solvers with many right hand sides (RHSs) or preconditioned iterative solvers. However, the low arithmetic intensity and sequential nature of the triangular solve algorithm pose performance challenges for its large-scale distributed-memory parallelization. In this work, we propose several strategies to enhance scalability of an algorithm with 2D block cyclic process layout. First, an asynchronous binary-tree-based communication scheme implemented via non-blocking MPI functions is leveraged to broadcastmore »« less
https://doi.org/10.1137/1.9781611975215.9

Full Text Available
Deploy threading in Nalu solver stack

Technical Report Prokopenko, Andrey; Thomas, Stephen; Swirydowicz, Kasia; ...

The goal of the ExaWind project is to enable predictive simulations of wind farms composed of many MW-scale turbines situated in complex terrain. Predictive simulations will require computational fluid dynamics (CFD) simulations for which the mesh resolves the geometry of the turbines, and captures the rotation and large deflections of blades. Whereas such simulations for a single turbine are arguably petascale class, multi-turbine wind farm simulations will require exascale-class resources. The primary code in the ExaWind project is Nalu, which is an unstructured-grid solver for the acousticallyincompressible Navier-Stokes equations, and mass continuity is maintained through pressure projection. The model consistsmore »« less
https://doi.org/10.2172/1481562

Full Text Available
Brief Announcement: Communication Optimal Sparse LU Factorization for Planar Matrices

Conference Sao, Piyush; Li, Xiaoye

We introduce a new parallel algorithm for solving sparse LU factorization of planar matrices, which commonly arise in the finite element method for 2D PDEs. Existing scalable methods, such as the multifrontal approach with subtree-to-subcube mapping by Gupta et al. [1] and right-looking with 3D mapping by Sao et al. [2] fail to achieve optimal communication costs for these matrices. Our new algorithm combines 3D mapping and subtree-to-subcube mapping to minimize communication costs while allowing trade-offs between extra memory and reduced communication. We demonstrate that our proposed algorithm attains the communication lower bound up to a factor of O(log logmore »« less
https://doi.org/10.1145/3558481.3591315

Full Text Available

Similar Records

Title: A communication-avoiding 3D sparse triangular solver

Abstract

Citation Formats

Nested Dissection of a Regular Finite Element Mesh journal, April 1973

Communication-Avoiding Parallel Sparse-Dense Matrix-Matrix Multiplication conference, May 2016

On asynchronous iterations journal, November 2000

Communication results for parallel sparse Cholesky factorization on a hypercube journal, May 1989

Structure-adaptive parallel solution of sparse triangular linear systems journal, October 2014

Convergence Models and Surprising Results for the Asynchronous Jacobi Method conference, May 2018

Highly scalable parallel algorithms for sparse matrix factorization journal, May 1997

A Communication-Avoiding 3D LU Factorization Algorithm for Sparse Matrices conference, May 2018

Integrated Model, Batch, and Domain Parallelism in Training Neural Networks conference, July 2018

Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations conference, May 2017

Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication journal, January 2016

Efficient Parallel Sparse Triangular Solution Using Selective Inversion journal, March 1998

Trading Replication for Communication in Parallel Distributed-Memory Dense Solvers journal, March 2002

A New Data-Mapping Scheme for Latency-Tolerant Distributed Sparse Triangular Solution conference, January 2002

Parallel Algorithms for Sparse Linear Systems journal, September 1991

Avoiding communication in sparse matrix computations conference, April 2008

Nested Dissection of a Regular Finite Element Mesh
journal, April 1973

Communication-Avoiding Parallel Sparse-Dense Matrix-Matrix Multiplication
conference, May 2016

On asynchronous iterations
journal, November 2000

Communication results for parallel sparse Cholesky factorization on a hypercube
journal, May 1989

Structure-adaptive parallel solution of sparse triangular linear systems
journal, October 2014

Convergence Models and Surprising Results for the Asynchronous Jacobi Method
conference, May 2018

Highly scalable parallel algorithms for sparse matrix factorization
journal, May 1997

A Communication-Avoiding 3D LU Factorization Algorithm for Sparse Matrices
conference, May 2018

Integrated Model, Batch, and Domain Parallelism in Training Neural Networks
conference, July 2018

Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations
conference, May 2017

Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication
journal, January 2016

Efficient Parallel Sparse Triangular Solution Using Selective Inversion
journal, March 1998

Trading Replication for Communication in Parallel Distributed-Memory Dense Solvers
journal, March 2002

A New Data-Mapping Scheme for Latency-Tolerant Distributed Sparse Triangular Solution
conference, January 2002

Parallel Algorithms for Sparse Linear Systems
journal, September 1991

Avoiding communication in sparse matrix computations
conference, April 2008