Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems

Haidar, Azzam; Bayraktar, Harun; Tomov, Stanimire; Dongarra, Jack; Higham, Nicholas J.

doi:10.1098/rspa.2020.0110

Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems

Journal Article · Wed Nov 25 00:00:00 EST 2020 · Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences

DOI:https://doi.org/10.1098/rspa.2020.0110· OSTI ID:1787013

Haidar, Azzam ^[1]; Bayraktar, Harun ^[1]; Tomov, Stanimire ^[2]; ^[3]; ^[4]

NVIDIA, Santa Clara, CA (United States)
Univ. of Tennessee, Knoxville, TN (United States). Dept. of Electrical Engineering and Computer Science
Univ. of Tennessee, Knoxville, TN (United States). Dept. of Electrical Engineering and Computer Science; Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Computer Science and Mathematics Division; Univ. of Manchester (United Kingdom). Dept. of Mathematics
Univ. of Manchester (United Kingdom). Dept. of Mathematics

Double-precision floating-point arithmetic (FP64) has been the de facto standard for engineering and scientific simulations for several decades. Problem complexity and the sheer volume of data coming from various instruments and sensors motivate researchers to mix and match various approaches to optimize compute resources, including different levels of floating-point precision. In recent years, machine learning has motivated hardware support for half-precision floating-point arithmetic. A primary challenge in high-performance computing is to leverage reduced-precision and mixed-precision hardware. We show how the FP16/FP32 Tensor Cores on NVIDIA GPUs can be exploited to accelerate the solution of linear systems of equations Ax = b without sacrificing numerical stability. The techniques we employ include multiprecision LU factorization, the preconditioned generalized minimal residual algorithm (GMRES), and scaling and auto-adaptive rounding to avoid overflow. We also show how to efficiently handle systems with multiple right-hand sides. On the NVIDIA Quadro GV100 (Volta) GPU, we achieve a 4×-5× performance increase and 5× better energy efficiency versus the standard FP64 implementation while maintaining an FP64 level of numerical stability.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

OSTI ID:: 1787013

Journal Information:: Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences, Journal Name: Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences Journal Issue: 2243 Vol. 476; ISSN 1364-5021

Publisher:: The Royal Society PublishingCopyright Statement

Country of Publication:: United States

Language:: English

References (25)

Iterative refinement enhances the stability ofQR factorization methods for solving linear equations Higham, Nicholas J. BIT, Vol. 31, Issue 3 https://doi.org/10.1007/BF01933262	journal	September 1991
Accelerating scientific computations with mixed precision algorithms Baboulin, Marc; Buttari, Alfredo; Dongarra, Jack Computer Physics Communications, Vol. 180, Issue 12 https://doi.org/10.1016/j.cpc.2008.11.005	journal	December 2009
Towards dense linear algebra for hybrid GPU accelerated manycore systems Tomov, Stanimire; Dongarra, Jack; Baboulin, Marc Parallel Computing, Vol. 36, Issue 5-6 https://doi.org/10.1016/j.parco.2009.12.005	journal	June 2010
Iterative refinement implies numerical stability for Gaussian elimination Skeel, Robert D. Mathematics of Computation, Vol. 35, Issue 151 https://doi.org/10.1090/S0025-5718-1980-0572859-4	journal	September 1980
Iterative refinement for linear systems and LAPACK Higham, N. IMA Journal of Numerical Analysis, Vol. 17, Issue 4 https://doi.org/10.1093/imanum/17.4.495	journal	October 1997
Dense linear algebra solvers for multicore with GPU accelerators Tomov, Stanimire; Nath, Rajib; Ltaief, Hatem 2010 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW) https://doi.org/10.1109/IPDPSW.2010.5470941	conference	April 2010
Exploiting the Performance of 32 bit Floating Point Arithmetic in Obtaining 64 bit Accuracy (Revisiting Iterative Refinement for Linear Systems) Langou, Julie; Langou, Julien; Luszczek, Piotr SC 2006 Proceedings Supercomputing 2006, ACM/IEEE SC 2006 Conference (SC'06) https://doi.org/10.1109/SC.2006.30	conference	November 2006
Harnessing GPU Tensor Cores for Fast FP16 Arithmetic to Speed up Mixed-Precision Iterative Refinement Solvers Haidar, Azzam; Tomov, Stanimire; Dongarra, Jack SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00050	conference	November 2018
Solving Sparse Linear Systems with Sparse Backward Error Arioli, M.; Demmel, J. W.; Duff, I. S. SIAM Journal on Matrix Analysis and Applications, Vol. 10, Issue 2 https://doi.org/10.1137/0610013	journal	April 1989
The Efficient Generation of Random Orthogonal Matrices with an Application to Condition Estimators Stewart, G. W. SIAM Journal on Numerical Analysis, Vol. 17, Issue 3 https://doi.org/10.1137/0717034	journal	June 1980
GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems Saad, Youcef; Schultz, Martin H. SIAM Journal on Scientific and Statistical Computing, Vol. 7, Issue 3 https://doi.org/10.1137/0907058	journal	July 1986
A Flexible Inner-Outer Preconditioned GMRES Algorithm Saad, Youcef SIAM Journal on Scientific Computing, Vol. 14, Issue 2 https://doi.org/10.1137/0914028	journal	March 1993
Accuracy and Stability of Numerical Algorithms Higham, Nicholas J. https://doi.org/10.1137/1.9780898718027	book	January 2002
LAPACK Users' Guide Anderson, E.; Bai, Z.; Bischof, C. https://doi.org/10.1137/1.9780898719604	software	January 1999
Applied Numerical Linear Algebra Demmel, James W. Society for Industrial and Applied Mathematics https://doi.org/10.1137/1.9781611971446	book	January 1997
A New Analysis of Iterative Refinement and Its Application to Accurate Solution of Ill-Conditioned Sparse Linear Systems Carson, Erin; Higham, Nicholas J. SIAM Journal on Scientific Computing, Vol. 39, Issue 6 https://doi.org/10.1137/17M1122918	journal	January 2017
Accelerating the Solution of Linear Systems by Iterative Refinement in Three Precisions Carson, Erin; Higham, Nicholas J. SIAM Journal on Scientific Computing, Vol. 40, Issue 2 https://doi.org/10.1137/17M1140819	journal	January 2018
Squeezing a Matrix into Half Precision, with an Application to Solving Linear Systems Higham, Nicholas J.; Pranesh, Srikara; Zounon, Mawussi SIAM Journal on Scientific Computing, Vol. 41, Issue 4 https://doi.org/10.1137/18M1229511	journal	January 2019
Mixed Precision Block Fused Multiply-Add: Error Analysis and Application to GPU Tensor Cores Blanchard, Pierre; Higham, Nicholas J.; Lopez, Florent SIAM Journal on Scientific Computing, Vol. 42, Issue 3 https://doi.org/10.1137/19M1289546	journal	January 2020
Flexible Inner-Outer Krylov Subspace Methods Simoncini, Valeria; Szyld, Daniel B. SIAM Journal on Numerical Analysis, Vol. 40, Issue 6 https://doi.org/10.1137/S0036142902401074	journal	January 2002
Inexact Preconditioned Conjugate Gradient Method with Inner-Outer Iteration Golub, Gene H.; Ye, Qiang SIAM Journal on Scientific Computing, Vol. 21, Issue 4 https://doi.org/10.1137/S1064827597323415	journal	January 1999
The university of Florida sparse matrix collection Davis, Timothy A.; Hu, Yifan ACM Transactions on Mathematical Software, Vol. 38, Issue 1 https://doi.org/10.1145/2049662.2049663	journal	November 2011
Investigating half precision arithmetic to accelerate dense linear system solvers Haidar, Azzam; Wu, Panruo; Tomov, Stanimire SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis, Proceedings of the 8th Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems https://doi.org/10.1145/3148226.3148237	conference	November 2017
Iterative Refinement in Floating Point Moler, Cleve B. Journal of the ACM, Vol. 14, Issue 2 https://doi.org/10.1145/321386.321394	journal	April 1967
Mixed Precision Iterative Refinement Techniques for the Solution of Dense Linear Systems Buttari, Alfredo; Dongarra, Jack; Langou, Julie The International Journal of High Performance Computing Applications, Vol. 21, Issue 4 https://doi.org/10.1177/1094342007084026	journal	November 2007

Similar Records

FTTN: Feature-Targeted Testing for Numerical Properties of NVIDIA & AMD Matrix Accelerators

Conference · Tue Oct 08 00:00:00 EDT 2024 · OSTI ID:2539787

A GPU accelerated mixed-precision Smoothed Particle Hydrodynamics framework with cell-based relative coordinates

Journal Article · Sun Jan 28 19:00:00 EST 2024 · Engineering Analysis with Boundary Elements · OSTI ID:2283916

Analyzing Deep Learning Model Inferences for Image Classification using OpenVINO

Conference · Tue Dec 31 23:00:00 EST 2019 · OSTI ID:1804060

Related Subjects

GMRES
97 MATHEMATICS AND COMPUTING
GPU computing
LU factorization
half precision arithmetic
iterative refinement
mixed precision solvers

Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems

Citation Formats

References (25)

Similar Records

Related Subjects