Pushing memory bandwidth limitations through efficient implementations of Block-Krylov space solvers on GPUs

doi:https://doi.org/10.1016/j.cpc.2018.06.019

Title: Pushing memory bandwidth limitations through efficient implementations of Block-Krylov space solvers on GPUs

Abstract

Lattice quantum chromodynamics simulations in nuclear physics have benefited from a tremendous number of algorithmic advances such as multigrid and eigenvector deflation. These improve the time to solution but do not alleviate the intrinsic memory-bandwidth constraints of the matrix-vector operation dominating iterative solvers. Batching this operation for multiple vectors and exploiting cache and register blocking can yield a super-linear speed up. Block-Krylov solvers can naturally take advantage of such batched matrix-vector operations, further reducing the iterations to solution by sharing the Krylov space between solves. However, practical implementations typically suffer from the quadratic scaling in the number of vector-vector operations. Here, using the QUDA library, we present an implementation of a block-CG solver on NVIDIA GPUs which reduces the memory-bandwidth complexity of vector-vector operations from quadratic to linear. We present results for the HISQ discretization, showing a 5x speedup compared to highly-optimized independent Krylov solves on NVIDIA's SaturnV cluster.

Authors:

Clark, M. A. ^[1]; Strelchenko, Alexei ^[2];

^[3]; Wagner, Mathias ^[4]; Weinberg, Evan ^[5]

NVIDIA Corporation, Santa Clara, CA (United States)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Univ. of Utah, Salt Lake City, UT (United States). Dept. of Physics and Astronomy
NVIDIA GmbH, Würselen (Germany)
Boston Univ., MA (United States). Dept. of Physics

Publication Date:: 2018-07-02

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP); USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1418147

Alternate Identifier(s):: OSTI ID: 1734408

Report Number(s):: arXiv:1710.09745; FERMILAB-PUB-17-592-CD
Journal ID: ISSN 0010-4655; 1632766

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Journal Article: Accepted Manuscript

Journal Name:: Computer Physics Communications

Additional Journal Information:: Journal Volume: 233; Journal Issue: C; Journal ID: ISSN 0010-4655

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS; 97 MATHEMATICS AND COMPUTING; Block solver; GPU

Citation Formats


                    Clark, M. A., Strelchenko, Alexei, Vaquero, Alejandro, Wagner, Mathias, and Weinberg, Evan. Pushing memory bandwidth limitations through efficient implementations of Block-Krylov space solvers on GPUs.  United States: N. p., 2018. 
        Web.  doi:10.1016/j.cpc.2018.06.019.

Copy to clipboard


                    Clark, M. A., Strelchenko, Alexei, Vaquero, Alejandro, Wagner, Mathias, & Weinberg, Evan. Pushing memory bandwidth limitations through efficient implementations of Block-Krylov space solvers on GPUs.  United States.  https://doi.org/10.1016/j.cpc.2018.06.019

Copy to clipboard


                    Clark, M. A., Strelchenko, Alexei, Vaquero, Alejandro, Wagner, Mathias, and Weinberg, Evan. Mon .  
        "Pushing memory bandwidth limitations through efficient implementations of Block-Krylov space solvers on GPUs".  United States.  https://doi.org/10.1016/j.cpc.2018.06.019.  https://www.osti.gov/servlets/purl/1418147.

Copy to clipboard


                    
@article{osti_1418147,

  title        = {Pushing memory bandwidth limitations through efficient implementations of Block-Krylov space solvers on GPUs},

  author       = {Clark, M. A. and Strelchenko, Alexei and Vaquero, Alejandro and Wagner, Mathias and Weinberg, Evan},

  abstractNote = {Lattice quantum chromodynamics simulations in nuclear physics have benefited from a tremendous number of algorithmic advances such as multigrid and eigenvector deflation. These improve the time to solution but do not alleviate the intrinsic memory-bandwidth constraints of the matrix-vector operation dominating iterative solvers. Batching this operation for multiple vectors and exploiting cache and register blocking can yield a super-linear speed up. Block-Krylov solvers can naturally take advantage of such batched matrix-vector operations, further reducing the iterations to solution by sharing the Krylov space between solves. However, practical implementations typically suffer from the quadratic scaling in the number of vector-vector operations. Here, using the QUDA library, we present an implementation of a block-CG solver on NVIDIA GPUs which reduces the memory-bandwidth complexity of vector-vector operations from quadratic to linear. We present results for the HISQ discretization, showing a 5x speedup compared to highly-optimized independent Krylov solves on NVIDIA's SaturnV cluster.},

  doi          = {10.1016/j.cpc.2018.06.019},

  url          = {https://www.osti.gov/biblio/1418147},
  journal      = {Computer Physics Communications},

  issn         = {0010-4655},

  number       = C,

  volume       = 233,

  place        = {United States},

  year         = {2018},

  month        = {7}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.cpc.2018.06.019

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 1 work

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referencing / citing this record:

Status and future perspectives for lattice gauge theory calculations to the exascale and beyond
journal, November 2019

Joó, Bálint; Jung, Chulwoo; Christ, Norman H.
The European Physical Journal A, Vol. 55, Issue 11
https://doi.org/10.1140/epja/i2019-12919-7

Similar records in OSTI.GOV collections:

Acceleration of GPU-based Krylov solvers via data transfer reduction

Journal Article Anzt, Hartwig ; Tomov, Stanimire ; Luszczek, Piotr ; ... - International Journal of High Performance Computing Applications

Krylov subspace iterative solvers are often the method of choice when solving large sparse linear systems. At the same time, hardware accelerators such as graphics processing units continue to offer significant floating point performance gains for matrix and vector computations through easy-to-use libraries of computational kernels. However, as these libraries are usually composed of a well optimized but limited set of linear algebra operations, applications that use them often fail to reduce certain data communications, and hence fail to leverage the full potential of the accelerator. In this study, we target the acceleration of Krylov subspace iterative methods for graphicsmore »« less
Cited by 6
https://doi.org/10.1177/1094342015580139

Full Text Available
Physics-based preconditioning and the Newton-Krylov method for non-equilibrium radiation diffusion

Journal Article Mousseau, V A ; Knoll, D A ; Rider, W J - Journal of Computational Physics

An algorithm is presented for the solution of the time dependent reaction-diffusion systems which arise in non-equilibrium radiation diffusion applications. This system of nonlinear equations is solved by coupling three numerical methods, Jacobian-free Newton-Krylov, operator splitting, and multigrid linear solvers. An inexact Newton's method is used to solve the system of nonlinear equations. Since building the Jacobian matrix for problems of interest can be challenging, the authors employ a Jacobian-free implementation of Newton's method, where the action of the Jacobian matrix on a vector is approximated by a first order Taylor series expansion. Preconditioned generalized minimal residual (PGMRES) is themore »« less
https://doi.org/10.1006/jcph.2000.6488
Petascale solvers for anisotropic PDEs in atmospheric modelling on GPU clusters

Journal Article Müller, Eike Hermann ; Scheichl, Robert ; Vainikko, Eero - Parallel Computing

Memore »« less
https://doi.org/10.1016/j.parco.2015.10.007
Block-Iterative Methods for 3D Constant-Coefficient Stencils on GPUs and Multicore CPUs

Technical Report Philip, Bobby ; Wang, Zhen ; Berrill, Mark A

Block iterative methods are extremely important as smoothers for multigrid methods, as preconditioners for Krylov methods, and as solvers for diagonally dominant linear systems. Developing robust and efficient smoother algorithms suitable for current and evolving GPU and multicore CPU systems is a significant challenge. We address this issue in the case of constant-coefficient stencils arising in the solution of elliptic partial differential equations on structured 3D uniform and adaptively refined block structured grids. Robust, highly parallel implementations of block Jacobi and chaotic block Gauss-Seidel algorithms with exact inversion of the blocks are developed using different parallelization techniques. Experimental results formore »« less
https://doi.org/10.2172/1134156
s-Step Krylov Subspace Methods as Bottom Solvers for Geometric Multigrid

Conference Williams, Samuel ; Lijewski, Mike ; Almgren, Ann ; ...

Geometric multigrid solvers within adaptive mesh refinement (AMR) applications often reach a point where further coarsening of the grid becomes impractical as individual sub domain sizes approach unity. At this point the most common solution is to use a bottom solver, such as BiCGStab, to reduce the residual by a fixed factor at the coarsest level. Each iteration of BiCGStab requires multiple global reductions (MPI collectives). As the number of BiCGStab iterations required for convergence grows with problem size, and the time for each collective operation increases with machine scale, bottom solves in large-scale applications can constitute a significant fractionmore »« less
https://doi.org/10.1109/IPDPS.2014.119

Full Text Available

Similar Records

Title: Pushing memory bandwidth limitations through efficient implementations of Block-Krylov space solvers on GPUs

Abstract

Citation Formats

Status and future perspectives for lattice gauge theory calculations to the exascale and beyond journal, November 2019

Status and future perspectives for lattice gauge theory calculations to the exascale and beyond
journal, November 2019