Title: A communication-avoiding 3D algorithm for sparse LU factorization on heterogeneous systems

Abstract

We propose a new algorithm to improve the strong scalability of right-looking sparse LU factorization on distributed memory systems. Our 3D algorithm for sparse LU uses a three-dimensional MPI process grid, exploits elimination tree parallelism, and trades off increased memory for reduced per-process communication. We also analyze the asymptotic improvements for planar graphs (e.g., those arising from 2D grid or mesh discretizations) and certain non-planar graphs (specifically for 3D grids and meshes). For a planar graph with $$n$$ vertices, our algorithm reduces communication volume asymptotically in $$n$$ by a factor of $$\mathscr{O}$$ $$\Big(\sqrt{log \ n}\Big)$$ and latency by a factor of $$\mathscr{O}$$ $$(log \ n)$$. For nonplanar cases, our algorithm can reduce the per-process communication volume by 3× and latency by $$\mathscr{O}$$ $$\Big(n^\frac13\Big)$$ times. In all cases, the memory needed to achieve these gains is a constant factor. We implemented our algorithm by extending the 2D data structure used in SuperLU_DIST. Our new 3D code achieves empirical speedups up to 27× for planar graphs and up to 3.3× for non-planar graphs over the baseline 2D SuperLU_DIST when run on 24,000 cores of a Cray XC30. We extend the 3D algorithm for heterogeneous architectures by adding the Highly Asynchronous Lazy Offload (Halo) algorithm for co-processor offload. On 4096 nodes of a Cray XK7 with 32,768 CPU cores and 4096 Nvidia K20x GPUs, the 3D algorithm achieves empirical speedups up to 24× for planar graphs and 3.5× for non-planar graphs over the baseline 2D SuperLU_DIST with co-processor acceleration.

Authors:

Sao, Piyush  ^[1];

• Search DOE PAGES for author "Sao, Piyush"
• Search DOE PAGES for ORCID "0000-0002-9432-5855"

Li, Xiaoye Sherry ^[2]; Vuduc, Richard ^[3]

---

+ Show Author Affiliations

1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
2. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
3. Georgia Inst. of Technology, Atlanta, GA (United States)

Publication Date:

Mon Aug 19 00:00:00 EDT 2019

Research Org.:

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:

USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:

1559632

Alternate Identifier(s):

OSTI ID: 1547464

Grant/Contract Number:  

AC05-00OR22725

Resource Type:

Accepted Manuscript

Journal Name:

Journal of Parallel and Distributed Computing

Additional Journal Information:

Journal Volume: 131; Journal Issue: 9; Journal ID: ISSN 0743-7315

Publisher:

Elsevier

Country of Publication:

United States

Language:

English

Subject:

97 MATHEMATICS AND COMPUTING