PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion: The Symmetric Case
Abstract
We describe an efficient parallel implementation of the selected inversion algorithm for distributed memory computer systems, which we call PSelInv. The PSelInv method computes selected elements of a general sparse matrix A that can be decomposed as A = LU, where L is lower triangular and U is upper triangular. The implementation described in this article focuses on the case of sparse symmetric matrices. It contains an interface that is compatible with the distributed memory parallel sparse direct factorization SuperLU_DIST. However, the underlying data structure and design of PSelInv allows it to be easily combined with other factorization routines, such as PARDISO. We discuss general parallelization strategies such as data and task distribution schemes. In particular, we describe how to exploit the concurrency exposed by the elimination tree associated with the LU factorization of A. We demonstrate the efficiency and accuracy of PSelInv by presenting several numerical experiments. In particular, we show that PSelInv can run efficiently on more than 4,000 cores for a modestly sized matrix. We also demonstrate how PSelInv can be used to accelerate largescale electronic structure calculations.
 Authors:

 Lawrence Berkeley National Laboratory, Berkeley CA USA
 University of California, Berkeley and Lawrence Berkeley National Laboratory, Berkeley CA
 Publication Date:
 Research Org.:
 Lawrence Berkeley National LaboratoryNational Energy Research Scientific Computing Center
 Sponsoring Org.:
 USDOE
 OSTI Identifier:
 1524063
 DOE Contract Number:
 AC0205CH11231
 Resource Type:
 Journal Article
 Journal Name:
 ACM Transactions on Mathematical Software
 Additional Journal Information:
 Journal Volume: 43; Journal Issue: 3; Journal ID: ISSN 00983500
 Country of Publication:
 United States
 Language:
 English
Citation Formats
Jacquelin, Mathias, Lin, Lin, and Yang, Chao. PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion: The Symmetric Case. United States: N. p., 2016.
Web. doi:10.1145/2786977.
Jacquelin, Mathias, Lin, Lin, & Yang, Chao. PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion: The Symmetric Case. United States. doi:10.1145/2786977.
Jacquelin, Mathias, Lin, Lin, and Yang, Chao. Wed .
"PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion: The Symmetric Case". United States. doi:10.1145/2786977.
@article{osti_1524063,
title = {PSelInv—A Distributed Memory Parallel Algorithm for Selected Inversion: The Symmetric Case},
author = {Jacquelin, Mathias and Lin, Lin and Yang, Chao},
abstractNote = {We describe an efficient parallel implementation of the selected inversion algorithm for distributed memory computer systems, which we call PSelInv. The PSelInv method computes selected elements of a general sparse matrix A that can be decomposed as A = LU, where L is lower triangular and U is upper triangular. The implementation described in this article focuses on the case of sparse symmetric matrices. It contains an interface that is compatible with the distributed memory parallel sparse direct factorization SuperLU_DIST. However, the underlying data structure and design of PSelInv allows it to be easily combined with other factorization routines, such as PARDISO. We discuss general parallelization strategies such as data and task distribution schemes. In particular, we describe how to exploit the concurrency exposed by the elimination tree associated with the LU factorization of A. We demonstrate the efficiency and accuracy of PSelInv by presenting several numerical experiments. In particular, we show that PSelInv can run efficiently on more than 4,000 cores for a modestly sized matrix. We also demonstrate how PSelInv can be used to accelerate largescale electronic structure calculations.},
doi = {10.1145/2786977},
journal = {ACM Transactions on Mathematical Software},
issn = {00983500},
number = 3,
volume = 43,
place = {United States},
year = {2016},
month = {12}
}
Works referenced in this record:
On Computing Inverse Entries of a Sparse Matrix in an OutofCore Environment
journal, January 2012
 Amestoy, Patrick R.; Duff, Iain S.; L'Excellent, JeanYves
 SIAM Journal on Scientific Computing, Vol. 34, Issue 4
On computing certain elements of the inverse of a sparse matrix
journal, March 1975
 Erisman, A. M.; Tinney, W. F.
 Communications of the ACM, Vol. 18, Issue 3
Accelerating atomic orbitalbased electronic structure calculation via pole expansion and selected inversion
journal, June 2013
 Lin, Lin; Chen, Mohan; Yang, Chao
 Journal of Physics: Condensed Matter, Vol. 25, Issue 29
The influence of relaxed supernode partitions on the multifrontal method
journal, December 1989
 Ashcraft, Cleve; Grimes, Roger
 ACM Transactions on Mathematical Software, Vol. 15, Issue 4
SelInvAn Algorithm for Selected Inversion of a Sparse Symmetric Matrix
journal, February 2011
 Lin, Lin; Yang, Chao; Meza, Juan C.
 ACM Transactions on Mathematical Software, Vol. 37, Issue 4
ScaLAPACK's MRRR algorithm
journal, January 2010
 Vömel, Christof
 ACM Transactions on Mathematical Software, Vol. 37, Issue 1
A Fast Parallel Algorithm for Selected Inversion of Structured Sparse Matrices with Application to 2D Electronic Structure Calculations
journal, January 2011
 Lin, Lin; Yang, Chao; Lu, Jianfeng
 SIAM Journal on Scientific Computing, Vol. 33, Issue 3
A hybrid method for the parallel computation of Green’s functions
journal, August 2009
 Petersen, Dan Erik; Li, Song; Stokbro, Kurt
 Journal of Computational Physics, Vol. 228, Issue 14
SuperLU_DIST: A scalable distributedmemory sparse direct solver for unsymmetric linear systems
journal, June 2003
 Li, Xiaoye S.; Demmel, James W.
 ACM Transactions on Mathematical Software, Vol. 29, Issue 2
PoleBased approximation of the FermiDirac function
journal, August 2009
 Lin, Lin; Lu, Jianfeng; Ying, Lexing
 Chinese Annals of Mathematics, Series B, Vol. 30, Issue 6
A Parallel Algorithm for Multilevel Graph Partitioning and Sparse Matrix Ordering
journal, January 1998
 Karypis, George; Kumar, Vipin
 Journal of Parallel and Distributed Computing, Vol. 48, Issue 1
The SIESTA method for ab initio order N materials simulation
journal, March 2002
 Soler, José M.; Artacho, Emilio; Gale, Julian D.
 Journal of Physics: Condensed Matter, Vol. 14, Issue 11
Some stable methods for calculating inertia and solving symmetric linear systems
journal, January 1977
 Bunch, James R.; Kaufman, Linda
 Mathematics of Computation, Vol. 31, Issue 137
A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling
journal, January 2001
 Amestoy, Patrick R.; Duff, Iain S.; L'Excellent, JeanYves
 SIAM Journal on Matrix Analysis and Applications, Vol. 23, Issue 1
The university of Florida sparse matrix collection
journal, November 2011
 Davis, Timothy A.; Hu, Yifan
 ACM Transactions on Mathematical Software, Vol. 38, Issue 1
A probing method for computing the diagonal of a matrix inverse
journal, April 2011
 Tang, Jok M.; Saad, Yousef
 Numerical Linear Algebra with Applications, Vol. 19, Issue 3
A twodimensional domain decomposition technique for the simulation of quantumscale devices
journal, February 2012
 Cauley, Stephen; Balakrishnan, Venkataramanan; Klimeck, Gerhard
 Journal of Computational Physics, Vol. 231, Issue 4
Inhomogeneous Electron Gas
journal, November 1964
 Hohenberg, P.; Kohn, W.
 Physical Review, Vol. 136, Issue 3B, p. B864B871
An iteration method for the solution of the eigenvalue problem of linear differential and integral operators
journal, October 1950
 Lanczos, C.
 Journal of Research of the National Bureau of Standards, Vol. 45, Issue 4
Extension and optimization of the FIND algorithm: Computing Green’s and lessthan Green’s functions
journal, February 2012
 Li, S.; Darve, E.
 Journal of Computational Physics, Vol. 231, Issue 4
Rational approximation to the Fermi–Dirac function with applications in density functional theory
journal, May 2010
 Sidje, Roger B.; Saad, Yousef
 Numerical Algorithms, Vol. 56, Issue 3
Electronic structure calculations with dynamical meanfield theory
journal, August 2006
 Kotliar, G.; Savrasov, S. Y.; Haule, K.
 Reviews of Modern Physics, Vol. 78, Issue 3
The Role of Elimination Trees in Sparse Factorization
journal, January 1990
 Liu, Joseph W. H.
 SIAM Journal on Matrix Analysis and Applications, Vol. 11, Issue 1
An estimator for the diagonal of a matrix
journal, November 2007
 Bekas, C.; Kokiopoulou, E.; Saad, Y.
 Applied Numerical Mathematics, Vol. 57, Issue 1112
Finding offdiagonal entries of the inverse of a large symmetric sparse matrix: FINDING OFFDIAGONAL ENTRIES OF THE INVERSE OF A LARGE SPARSE MATRIX
journal, February 2012
 Eastwood, Shawn; Wan, Justin W. L.
 Numerical Linear Algebra with Applications, Vol. 20, Issue 1
A nested dissection approach to modeling transport in nanodevices: Algorithms and applications: NESTED DISSECTION APPROACH TO MODELING TRANSPORT IN NANODEVICES
journal, June 2013
 Hetmaniuk, U.; Zhao, Y.; Anantram, M. P.
 International Journal for Numerical Methods in Engineering, Vol. 95, Issue 7
Adaptive local basis set for Kohn–Sham density functional theory in a discontinuous Galerkin framework I: Total energy calculation
journal, February 2012
 Lin, Lin; Lu, Jianfeng; Ying, Lexing
 Journal of Computational Physics, Vol. 231, Issue 4
A fast algorithm for sparse matrix computations related to inversion
journal, June 2013
 Li, S.; Wu, W.; Darve, E.
 Journal of Computational Physics, Vol. 242
An Efficient BlockOriented Approach to Parallel Sparse Cholesky Factorization
journal, November 1994
 Rothberg, Edward; Gupta, Anoop
 SIAM Journal on Scientific Computing, Vol. 15, Issue 6
A Shifted Block Lanczos Algorithm for Solving Sparse Symmetric Generalized Eigenproblems
journal, January 1994
 Grimes, Roger G.; Lewis, John G.; Simon, Horst D.
 SIAM Journal on Matrix Analysis and Applications, Vol. 15, Issue 1
Computing entries of the inverse of a sparse matrix using the FIND algorithm
journal, November 2008
 Li, S.; Ahmed, S.; Klimeck, G.
 Journal of Computational Physics, Vol. 227, Issue 22
PTScotch: A tool for efficient parallel graph ordering
journal, July 2008
 Chevalier, C.; Pellegrini, F.
 Parallel Computing, Vol. 34, Issue 68
SelfConsistent Equations Including Exchange and Correlation Effects
journal, November 1965
 Kohn, W.; Sham, L. J.
 Physical Review, Vol. 140, Issue 4A, p. A1133A1138