Convergence analysis of Anderson-type acceleration of Richardson's iteration

Lupo Pasini, Massimiliano

doi:10.1002/nla.2241

Title: Convergence analysis of Anderson-type acceleration of Richardson's iteration

Abstract

We consider here Anderson extrapolation to accelerate the (stationary) Richardson iterative method for sparse linear systems. Using an Anderson mixing at periodic intervals, we assess how this benefits convergence to a prescribed accuracy. The method, named alternating Anderson–Richardson, has appealing properties for high-performance computing, such as the potential to reduce communication and storage in comparison to more conventional linear solvers. We establish sufficient conditions for convergence, and we evaluate the performance of this technique in combination with various preconditioners through numerical examples. Furthermore, we propose an augmented version of this technique.

Authors:

^[1]

Emory Univ., Atlanta, GA (United States). Dept. of Mathematics and Computer Science; Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). National Center for Computational Sciences

Publication Date:: Wed Apr 03 00:00:00 EDT 2019

Research Org.:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1511931

Alternate Identifier(s):: OSTI ID: 1504993

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Numerical Linear Algebra with Applications

Additional Journal Information:: Journal Volume: 26; Journal Issue: 4; Journal ID: ISSN 1070-5325

Publisher:: Wiley

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Anderson acceleration; fixed-point scheme; projection method; Richardson iteration

Citation Formats


                    Lupo Pasini, Massimiliano. Convergence analysis of Anderson-type acceleration of Richardson's iteration.  United States: N. p., 2019. 
Web.  doi:10.1002/nla.2241.

Copy to clipboard


                    Lupo Pasini, Massimiliano. Convergence analysis of Anderson-type acceleration of Richardson's iteration.  United States.  https://doi.org/10.1002/nla.2241

Copy to clipboard


                    Lupo Pasini, Massimiliano. Wed .  
"Convergence analysis of Anderson-type acceleration of Richardson's iteration".  United States.  https://doi.org/10.1002/nla.2241.  https://www.osti.gov/servlets/purl/1511931.

Copy to clipboard


                    
@article{osti_1511931,

  title        = {Convergence analysis of Anderson-type acceleration of Richardson's iteration},

  author       = {Lupo Pasini, Massimiliano},

  abstractNote = {We consider here Anderson extrapolation to accelerate the (stationary) Richardson iterative method for sparse linear systems. Using an Anderson mixing at periodic intervals, we assess how this benefits convergence to a prescribed accuracy. The method, named alternating Anderson–Richardson, has appealing properties for high-performance computing, such as the potential to reduce communication and storage in comparison to more conventional linear solvers. We establish sufficient conditions for convergence, and we evaluate the performance of this technique in combination with various preconditioners through numerical examples. Furthermore, we propose an augmented version of this technique.},

  doi          = {10.1002/nla.2241},

  journal      = {Numerical Linear Algebra with Applications},

  number       = 4,

  volume       = 26,

  place        = {United States},

  year         = {Wed Apr 03 00:00:00 EDT 2019},

  month        = {Wed Apr 03 00:00:00 EDT 2019}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1002/nla.2241

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 5 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Reducing the bandwidth of sparse symmetric matrices
conference, January 1969

Cuthill, E.; McKee, J.
Proceedings of the 1969 24th national conference on -
DOI: 10.1145/800195.805928

A characterization of the behavior of the Anderson acceleration on linear problems
journal, February 2013

Potra, Florian A.; Engler, Hans
Linear Algebra and its Applications, Vol. 438, Issue 3
DOI: 10.1016/j.laa.2012.09.008

Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods: Part II
journal, December 1961

Golub, Gene H.; Varga, Richard S.
Numerische Mathematik, Vol. 3, Issue 1
DOI: 10.1007/BF01386014

Two classes of multisecant methods for nonlinear acceleration
journal, March 2009

Fang, Haw-ren; Saad, Yousef
Numerical Linear Algebra with Applications, Vol. 16, Issue 3
DOI: 10.1002/nla.617

Benchmarking optimization software with performance profiles
journal, January 2002

Dolan, Elizabeth D.; Moré, Jorge J.
Mathematical Programming, Vol. 91, Issue 2
DOI: 10.1007/s101070100263

Numerical Analysis of Fixed Point Algorithms in the Presence of Hardware Faults
journal, January 2015

Stoyanov, Miroslav; Webster, Clayton
SIAM Journal on Scientific Computing, Vol. 37, Issue 5
DOI: 10.1137/140991406

Iterative Methods for Sparse Linear Systems
book, January 2003

Saad, Yousef
DOI: 10.1137/1.9780898718003

GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems
journal, July 1986

Saad, Youcef; Schultz, Martin H.
SIAM Journal on Scientific and Statistical Computing, Vol. 7, Issue 3
DOI: 10.1137/0907058

Hierarchical Krylov and nested Krylov methods for extreme-scale computing
journal, January 2014

McInnes, Lois Curfman; Smith, Barry; Zhang, Hong
Parallel Computing, Vol. 40, Issue 1
DOI: 10.1016/j.parco.2013.10.001

Alternating Anderson–Richardson method: An efficient alternative to preconditioned Krylov methods for large, sparse linear systems
journal, January 2019

Suryanarayana, Phanish; Pratapa, Phanisri P.; Pask, John E.
Computer Physics Communications, Vol. 234
DOI: 10.1016/j.cpc.2018.07.007

Anderson acceleration of the Jacobi iterative method: An efficient alternative to Krylov methods for large, sparse linear systems
journal, February 2016

Pratapa, Phanisri P.; Suryanarayana, Phanish; Pask, John E.
Journal of Computational Physics, Vol. 306
DOI: 10.1016/j.jcp.2015.11.018

Domain Decomposition Preconditioners for Communication-Avoiding Krylov Methods on a Hybrid CPU/GPU Cluster
conference, November 2014

Yamazaki, Ichitaro; Rajamanickam, Sivasankaran; Boman, Erik G.
SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
DOI: 10.1109/SC.2014.81

Convergence Analysis for Anderson Acceleration
journal, January 2015

Toth, Alex; Kelley, C. T.
SIAM Journal on Numerical Analysis, Vol. 53, Issue 2
DOI: 10.1137/130919398

Preconditioning Techniques for Large Linear Systems: A Survey
journal, November 2002

Benzi, Michele
Journal of Computational Physics, Vol. 182, Issue 2
DOI: 10.1006/jcph.2002.7176

Anderson Acceleration for Fixed-Point Iterations
journal, January 2011

Walker, Homer F.; Ni, Peng
SIAM Journal on Numerical Analysis, Vol. 49, Issue 4
DOI: 10.1137/10078356X

The Tchebychev iteration for nonsymmetric linear systems
journal, September 1977

Manteuffel, Thomas A.
Numerische Mathematik, Vol. 28, Issue 3
DOI: 10.1007/BF01389971

Iterative Procedures for Nonlinear Integral Equations
journal, October 1965

Anderson, Donald G.
Journal of the ACM, Vol. 12, Issue 4
DOI: 10.1145/321296.321305

Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods: Part I
journal, December 1961

Golub, Gene H.; Varga, Richard S.
Numerische Mathematik, Vol. 3, Issue 1
DOI: 10.1007/bf01386013

Benchmarking Optimization Software with Performance Profiles
text, January 2001

Dolan, Elizabeth D.; Moré, Jorge J.
arXiv
DOI: 10.48550/arxiv.cs/0102001

Similar Records in DOE PAGES and OSTI.GOV collections:

Anderson acceleration of the Jacobi iterative method: An efficient alternative to Krylov methods for large, sparse linear systems

Journal Article Pratapa, Phanisri P. ; Suryanarayana, Phanish ; Pask, John E. - Journal of Computational Physics

We employ Anderson extrapolation to accelerate the classical Jacobi iterative method for large, sparse linear systems. Specifically, we utilize extrapolation at periodic intervals within the Jacobi iteration to develop the Alternating Anderson–Jacobi (AAJ) method. We verify the accuracy and efficacy of AAJ in a range of test cases, including nonsymmetric systems of equations. We demonstrate that AAJ possesses a favorable scaling with system size that is accompanied by a small prefactor, even in the absence of a preconditioner. In particular, we show that AAJ is able to accelerate the classical Jacobi iteration by over four orders of magnitude, with speed-upsmore »« less
Cited by 44
https://doi.org/10.1016/j.jcp.2015.11.018

Full Text Available
Analysis of Monte Carlo accelerated iterative methods for sparse linear systems: Analysis of Monte Carlo accelerated iterative methods for sparse linear systems

Journal Article Benzi, Michele ; Evans, Thomas M. ; Hamilton, Steven P. ; ... - Numerical Linear Algebra with Applications

Summary We consider hybrid deterministic‐stochastic iterative algorithms for the solution of large, sparse linear systems. Starting from a convergent splitting of the coefficient matrix, we analyze various types of Monte Carlo acceleration schemes applied to the original preconditioned Richardson (stationary) iteration. These methods are expected to have considerable potential for resiliency to faults when implemented on massively parallel machines. We establish sufficient conditions for the convergence of the hybrid schemes, and we investigate different types of preconditioners including sparse approximate inverses. Numerical experiments on linear systems arising from the discretization of partial differential equations are presented.
Cited by 13
https://doi.org/10.1002/nla.2088

Full Text Available
Alternating Anderson–Richardson method: An efficient alternative to preconditioned Krylov methods for large, sparse linear systems

Journal Article Suryanarayana, Phanish ; Pratapa, Phanisri P. ; Pask, John E. - Computer Physics Communications

Here, we present the Alternating Anderson–Richardson (AAR) method: an efficient and scalable alternative to preconditioned Krylov solvers for the solution of large, sparse linear systems on high performance computing platforms. Specifically, we generalize the recently proposed Alternating Anderson–Jacobi (AAJ) method (Pratapa et al., 2016) to include preconditioning, discuss efficient parallel implementation, and provide serial MATLAB and parallel C/C++ implementations. In serial applications to nonsymmetric systems, we find that AAR is comparably robust to GMRES, using the same preconditioning, while often outperforming it in time to solution; and find AAR to be more robust than Bi-CGSTAB for the problems considered. Inmore »« less
Cited by 24
https://doi.org/10.1016/j.cpc.2018.07.007

Full Text Available
Leveraging Anderson Acceleration for improved convergence of iterative solutions to transport systems

Journal Article Willert, Jeffrey ; Taitano, William T. ; Knoll, Dana - Journal of Computational Physics

In this note we demonstrate that using Anderson Acceleration (AA) in place of a standard Picard iteration can not only increase the convergence rate but also make the iteration more robust for two transport applications. We also compare the convergence acceleration provided by AA to that provided by moment-based acceleration methods. Additionally, we demonstrate that those two acceleration methods can be used together in a nested fashion. We begin by describing the AA algorithm. At this point, we will describe two application problems, one from neutronics and one from plasma physics, on which we will apply AA. We provide computationalmore »« less
https://doi.org/10.1016/J.JCP.2014.05.015
Analysis of Anderson Acceleration on a Simplified Neutronics/Thermal Hydraulics System

Conference Toth, Alex ; Kelley, C. T. ; Slattery, Stuart R ; ...

ABSTRACT A standard method for solving coupled multiphysics problems in light water reactors is Picard iteration, which sequentially alternates between solving single physics applications. This solution approach is appealing due to simplicity of implementation and the ability to leverage existing software packages to accurately solve single physics applications. However, there are several drawbacks in the convergence behavior of this method; namely slow convergence and the necessity of heuristically chosen damping factors to achieve convergence in many cases. Anderson acceleration is a method that has been seen to be more robust and fast converging than Picard iteration for many problems, withoutmore »« less

Similar Records

Title: Convergence analysis of Anderson-type acceleration of Richardson's iteration

Abstract

Citation Formats

Reducing the bandwidth of sparse symmetric matrices conference, January 1969

A characterization of the behavior of the Anderson acceleration on linear problems journal, February 2013

Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods: Part II journal, December 1961

Two classes of multisecant methods for nonlinear acceleration journal, March 2009

Benchmarking optimization software with performance profiles journal, January 2002

Numerical Analysis of Fixed Point Algorithms in the Presence of Hardware Faults journal, January 2015

Iterative Methods for Sparse Linear Systems book, January 2003

GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems journal, July 1986

Hierarchical Krylov and nested Krylov methods for extreme-scale computing journal, January 2014

Alternating Anderson–Richardson method: An efficient alternative to preconditioned Krylov methods for large, sparse linear systems journal, January 2019

Anderson acceleration of the Jacobi iterative method: An efficient alternative to Krylov methods for large, sparse linear systems journal, February 2016

Domain Decomposition Preconditioners for Communication-Avoiding Krylov Methods on a Hybrid CPU/GPU Cluster conference, November 2014

Convergence Analysis for Anderson Acceleration journal, January 2015

Preconditioning Techniques for Large Linear Systems: A Survey journal, November 2002

Anderson Acceleration for Fixed-Point Iterations journal, January 2011

The Tchebychev iteration for nonsymmetric linear systems journal, September 1977

Iterative Procedures for Nonlinear Integral Equations journal, October 1965

Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods: Part I journal, December 1961

Benchmarking Optimization Software with Performance Profiles text, January 2001

Reducing the bandwidth of sparse symmetric matrices
conference, January 1969

A characterization of the behavior of the Anderson acceleration on linear problems
journal, February 2013

Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods: Part II
journal, December 1961

Two classes of multisecant methods for nonlinear acceleration
journal, March 2009

Benchmarking optimization software with performance profiles
journal, January 2002

Numerical Analysis of Fixed Point Algorithms in the Presence of Hardware Faults
journal, January 2015

Iterative Methods for Sparse Linear Systems
book, January 2003

GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems
journal, July 1986

Hierarchical Krylov and nested Krylov methods for extreme-scale computing
journal, January 2014

Alternating Anderson–Richardson method: An efficient alternative to preconditioned Krylov methods for large, sparse linear systems
journal, January 2019

Anderson acceleration of the Jacobi iterative method: An efficient alternative to Krylov methods for large, sparse linear systems
journal, February 2016

Domain Decomposition Preconditioners for Communication-Avoiding Krylov Methods on a Hybrid CPU/GPU Cluster
conference, November 2014

Convergence Analysis for Anderson Acceleration
journal, January 2015

Preconditioning Techniques for Large Linear Systems: A Survey
journal, November 2002

Anderson Acceleration for Fixed-Point Iterations
journal, January 2011

The Tchebychev iteration for nonsymmetric linear systems
journal, September 1977

Iterative Procedures for Nonlinear Integral Equations
journal, October 1965

Chebyshev semi-iterative methods, successive overrelaxation iterative methods, and second order Richardson iterative methods: Part I
journal, December 1961

Benchmarking Optimization Software with Performance Profiles
text, January 2001