Block Gram-Schmidt Algorithms and Their Stability Properties

Carson, Erin; Lund, Kathryn; Rozloznik, Miroslav; Thomas, Stephen

doi:10.1016/j.laa.2021.12.017

Block Gram-Schmidt Algorithms and Their Stability Properties

Journal Article · 28 December 2021 · Linear Algebra and its Applications

DOI:https://doi.org/10.1016/j.laa.2021.12.017· OSTI ID:1841142

Carson, Erin; Lund, Kathryn; Rozloznik, Miroslav; Thomas, Stephen

Block Gram-Schmidt algorithms serve as essential kernels in many scientific computing applications, but for many commonly used variants, a rigorous treatment of their stability properties remains open. This work provides a comprehensive categorization of block Gram-Schmidt algorithms, particularly those used in Krylov subspace methods to build orthonormal bases one block vector at a time. Known stability results are assembled, and new results are summarized or conjectured for important communication-reducing variants. Additionally, new block versions of low-synchronization variants are derived, and their efficacy and stability are demonstrated for a wide range of challenging examples. Numerical examples are computed with a versatile Matlab package hosted at https://github.com/katlund/BlockStab, and scripts for reproducing all results in the paper are provided. Block Gram-Schmidt implementations in popular software packages are discussed, along with a number of open problems. An appendix containing all algorithms type-set in a uniform fashion is provided.

Research Organization:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

DOE Contract Number:: AC36-08GO28308

OSTI ID:: 1841142

Report Number(s):: NREL/JA-2C00-80324; MainId:42527; UUID:a7eaad28-5513-4750-84b8-631d1fb70f33; MainAdminID:63642

Journal Information:: Linear Algebra and its Applications, Journal Name: Linear Algebra and its Applications Vol. 638

Country of Publication:: United States

Language:: English

References (42)

Block Gram–Schmidt Orthogonalization Stewart, G. W. SIAM Journal on Scientific Computing, Vol. 31, Issue 1 https://doi.org/10.1137/070682563	journal	January 2008
A Storage-Efficient $WY$ Representation for Products of Householder Transformations Schreiber, Robert; Van Loan, Charles SIAM Journal on Scientific and Statistical Computing, Vol. 10, Issue 1 https://doi.org/10.1137/0910005	journal	January 1989
Iterative algorithms for Gram-Schmidt orthogonalization Hoffmann, Walter Computing, Vol. 41, Issue 4 https://doi.org/10.1007/BF02241222	journal	December 1989
A projection method for generalized eigenvalue problems using numerical integration Sakurai, Tetsuya; Sugiura, Hiroshi Journal of Computational and Applied Mathematics, Vol. 159, Issue 1 https://doi.org/10.1016/S0377-0427(03)00565-X	journal	October 2003
Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method Knyazev, Andrew V. SIAM Journal on Scientific Computing, Vol. 23, Issue 2 https://doi.org/10.1137/S1064827500366124	journal	January 2001
Updating the QR decomposition of block tridiagonal and block Hessenberg matrices Gutknecht, Martin H.; Schmelzer, Thomas Applied Numerical Mathematics, Vol. 58, Issue 6 https://doi.org/10.1016/j.apnum.2007.04.010	journal	June 2008
The Stability of Block Variants of Classical Gram--Schmidt Carson, Erin; Lund, Kathryn; Rozložník, Miroslav SIAM Journal on Matrix Analysis and Applications, Vol. 42, Issue 3 https://doi.org/10.1137/21M1394424	journal	January 2021
Modified Gram-Schmidt (MGS), Least Squares, and Backward Stability of MGS-GMRES Paige, Christopher C.; Rozlozník, Miroslav; Strakos, Zdenvek SIAM Journal on Matrix Analysis and Applications, Vol. 28, Issue 1 https://doi.org/10.1137/050630416	journal	January 2006
Enlarged Krylov Subspace Conjugate Gradient Methods for Reducing Communication Grigori, Laura; Moufawad, Sophie; Nataf, Frederic SIAM Journal on Matrix Analysis and Applications, Vol. 37, Issue 2 https://doi.org/10.1137/140989492	journal	January 2016
Deflated Restarting for Matrix Functions Eiermann, M.; Ernst, O. G.; Güttel, S. SIAM Journal on Matrix Analysis and Applications, Vol. 32, Issue 2 https://doi.org/10.1137/090774665	journal	April 2011
Reorthogonalized block classical Gram–Schmidt Barlow, Jesse L.; Smoktunowicz, Alicja Numerische Mathematik, Vol. 123, Issue 3 https://doi.org/10.1007/s00211-012-0496-2	journal	September 2012
Round off error analysis for Gram-Schmidt method and solution of linear least squares problems Abdelmalek, Nabih N. BIT, Vol. 11, Issue 4 https://doi.org/10.1007/BF01939404	journal	December 1971
Rational Krylov approximation of matrix functions: Numerical methods and optimal pole selection Güttel, Stefan GAMM-Mitteilungen, Vol. 36, Issue 1 https://doi.org/10.1002/gamm.201310002	journal	August 2013
Mixed-Precision Cholesky QR Factorization and Its Case Studies on Multicore CPU with Multiple GPUs Yamazaki, Ichitaro; Tomov, Stanimire; Dongarra, Jack SIAM Journal on Scientific Computing, Vol. 37, Issue 3 https://doi.org/10.1137/14M0973773	journal	January 2015
Efficient and Stable Arnoldi Restarts for Matrix Functions Based on Quadrature Frommer, Andreas; Güttel, Stefan; Schweitzer, Marcel SIAM Journal on Matrix Analysis and Applications, Vol. 35, Issue 2 https://doi.org/10.1137/13093491X	journal	January 2014
A Krylov--Schur Algorithm for Large Eigenproblems Stewart, G. W. SIAM Journal on Matrix Analysis and Applications, Vol. 23, Issue 3 https://doi.org/10.1137/S0895479800371529	journal	January 2002
Recycling Krylov Subspaces for Sequences of Linear Systems Parks, Michael L.; de Sturler, Eric; Mackey, Greg SIAM Journal on Scientific Computing, Vol. 28, Issue 5 https://doi.org/10.1137/040607277	journal	January 2006
Rounding error analysis of the classical Gram-Schmidt orthogonalization process Giraud, Luc; Langou, Julien; Rozložník, Miroslav Numerische Mathematik, Vol. 101, Issue 1 https://doi.org/10.1007/s00211-005-0615-4	journal	May 2005
Low synchronization Gram–Schmidt and generalized minimal residual algorithms Świrydowicz, Katarzyna; Langou, Julien; Ananthan, Shreyas Numerical Linear Algebra with Applications, Vol. 28, Issue 2 https://doi.org/10.1002/nla.2343	journal	October 2020
Communication lower bounds and optimal algorithms for numerical linear algebra Ballard, G.; Carson, E.; Demmel, J. Acta Numerica, Vol. 23 https://doi.org/10.1017/S0962492914000038	journal	May 2014
Loss and Recapture of Orthogonality in the Modified Gram–Schmidt Algorithm Björck, Å.; Paige, C. C. SIAM Journal on Matrix Analysis and Applications, Vol. 13, Issue 1 https://doi.org/10.1137/0613015	journal	January 1992
KSPHPDDM and PCHPDDM: Extending PETSc with advanced Krylov methods and robust multilevel overlapping Schwarz preconditioners Jolivet, Pierre; Roman, Jose E.; Zampini, Stefano Computers & Mathematics with Applications, Vol. 84 https://doi.org/10.1016/j.camwa.2021.01.003	journal	February 2021
Stability Analysis and Improvement of the Block Gram–Schmidt Algorithm Jalby, W.; Philippe, B. SIAM Journal on Scientific and Statistical Computing, Vol. 12, Issue 5 https://doi.org/10.1137/0912056	journal	September 1991
Convergence properties of block GMRES and matrix polynomials Simoncini, V.; Gallopoulos, E. Linear Algebra and its Applications, Vol. 247 https://doi.org/10.1016/0024-3795(95)00093-3	journal	November 1996
Solving linear least squares problems by Gram-Schmidt orthogonalization Björck, Åke BIT, Vol. 7, Issue 1 https://doi.org/10.1007/BF01934122	journal	March 1967
Communication-optimal Parallel and Sequential QR and LU Factorizations Demmel, James; Grigori, Laura; Hoemmen, Mark SIAM Journal on Scientific Computing, Vol. 34, Issue 1 https://doi.org/10.1137/080731992	journal	January 2012
An accurate parallel block Gram-Schmidt algorithm without reorthogonalization Vanderstraeten, Denis Numerical Linear Algebra with Applications, Vol. 7, Issue 4 https://doi.org/10.1002/1099-1506(200005)7:4<219::AID-NLA196>3.0.CO;2-L	journal	January 2000
An adaptive $s$-step conjugate gradient algorithm with dynamic basis updating [english] Carson, Erin Claire Applications of Mathematics, Vol. 65, Issue 2 https://doi.org/10.21136/AM.2020.0136-19	journal	February 2020
Gram-Schmidt orthogonalization: 100 years and more: GRAM-SCHMIDT ORTHOGONALIZATION: 100 YEARS AND MORE Leon, Steven J.; Björck, Åke; Gander, Walter Numerical Linear Algebra with Applications, Vol. 20, Issue 3 https://doi.org/10.1002/nla.1839	journal	June 2012
Restarted block-GMRES with deflation of eigenvalues Morgan, Ronald B. Applied Numerical Mathematics, Vol. 54, Issue 2 https://doi.org/10.1016/j.apnum.2004.09.028	journal	July 2005
Backward error analysis of the AllReduce algorithm for householder QR decomposition Mori, Daisuke; Yamamoto, Yusaku; Zhang, Shao-Liang Japan Journal of Industrial and Applied Mathematics, Vol. 29, Issue 1 https://doi.org/10.1007/s13160-011-0053-x	journal	December 2011
Implementation of the GMRES Method Using Householder Transformations Walker, Homer F. SIAM Journal on Scientific and Statistical Computing, Vol. 9, Issue 1 https://doi.org/10.1137/0909010	journal	January 1988
Block Modified Gram--Schmidt Algorithms and Their Analysis Barlow, Jesse L. SIAM Journal on Matrix Analysis and Applications, Vol. 40, Issue 4 https://doi.org/10.1137/18M1197400	journal	January 2019
A note on the error analysis of classical Gram–Schmidt Smoktunowicz, Alicja; Barlow, Jesse L.; Langou, Julien Numerische Mathematik, Vol. 105, Issue 2 https://doi.org/10.1007/s00211-006-0042-1	journal	November 2006
Numerical aspects of Gram-Schmidt orthogonalization of vectors Ruhe, A. Linear Algebra and its Applications, Vol. 52-53, Issue 1 https://doi.org/10.1016/0024-3795(83)90037-X	journal	July 1983
Block-Arnoldi and Davidson methods for unsymmetric large eigenvalue problems Sadkane, Miloud Numerische Mathematik, Vol. 64, Issue 1 https://doi.org/10.1007/BF01388687	journal	December 1993
Block Krylov Subspace Methods for Functions of Matrices II: Modified Block FOM Frommer, Andreas; Lund, Kathryn; Szyld, Daniel B. SIAM Journal on Matrix Analysis and Applications, Vol. 41, Issue 2 https://doi.org/10.1137/19M1255847	journal	January 2020
On Improving Linear Solver Performance: A Block Variant of GMRES Baker, A. H.; Dennis, J. M.; Jessup, E. R. SIAM Journal on Scientific Computing, Vol. 27, Issue 5 https://doi.org/10.1137/040608088	journal	January 2006
Convergence of Restarted Krylov Subspace Methods for Stieltjes Functions of Matrices Frommer, Andreas; Güttel, Stefan; Schweitzer, Marcel SIAM Journal on Matrix Analysis and Applications, Vol. 35, Issue 4 https://doi.org/10.1137/140973463	journal	January 2014
The block conjugate gradient algorithm and related methods O'Leary, Dianne P. Linear Algebra and its Applications, Vol. 29 https://doi.org/10.1016/0024-3795(80)90247-5	journal	February 1980
Shifted Cholesky QR for Computing the QR Factorization of Ill-Conditioned Matrices Fukaya, Takeshi; Kannan, Ramaseshan; Nakatsukasa, Yuji SIAM Journal on Scientific Computing, Vol. 42, Issue 1 https://doi.org/10.1137/18M1218212	journal	January 2020
When modified Gram-Schmidt generates a well-conditioned set of vectors Giraud, L. IMA Journal of Numerical Analysis, Vol. 22, Issue 4 https://doi.org/10.1093/imanum/22.4.521	journal	October 2002

Similar Records

An approach of orthogonalization within the Gram–Schmidt algorithm

Journal Article · 2018 · Computational and Applied Mathematics · OSTI ID:22783864

Low-synch Gram–Schmidt with delayed reorthogonalization for Krylov solvers

Journal Article · 2022 · Parallel Computing · OSTI ID:1872059

Related Subjects

Gram-Schmidt
Krylov
MATHEMATICS AND COMPUTING
block Krylov subspace methods
loss of orthogonality
stability

Block Gram-Schmidt Algorithms and Their Stability Properties

Citation Formats

References (42)

Similar Records

Related Subjects