skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Improving the Numerical Stability of Fast Matrix Multiplication

Journal Article · · SIAM Journal on Matrix Analysis and Applications
DOI:https://doi.org/10.1137/15M1032168· OSTI ID:1356986
 [1];  [2];  [3];  [4];  [5]
  1. Sandia National Lab. (SNL-CA), Livermore, CA (United States); Wake Forest Univ., Winston-Salem, NC (United States)
  2. Stanford Univ., CA (United States). Inst. for Computational and Mathematical Engineering
  3. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  4. Google, San Bruno, CA (United States)
  5. Hebrew Univ. of Jerusalem (Israel). School of Computer Science and Engineering
+ Show Author Affiliations

Fast algorithms for matrix multiplication, namely those that perform asymptotically fewer scalar operations than the classical algorithm, have been considered primarily of theoretical interest. Apart from Strassen's original algorithm, few fast algorithms have been efficiently implemented or used in practical applications. However, there exist many practical alternatives to Strassen's algorithm with varying performance and numerical properties. Fast algorithms are known to be numerically stable, but because their error bounds are slightly weaker than the classical algorithm, they are not used even in cases where they provide a performance benefit. We argue in this paper that the numerical sacrifice of fast algorithms, particularly for the typical use cases of practical algorithms, is not prohibitive, and we explore ways to improve the accuracy both theoretically and empirically. The numerical accuracy of fast matrix multiplication depends on properties of the algorithm and of the input matrices, and we consider both contributions independently. We generalize and tighten previous error analyses of fast algorithms and compare their properties. We discuss algorithmic techniques for improving the error guarantees from two perspectives: manipulating the algorithms, and reducing input anomalies by various forms of diagonal scaling. Finally, we benchmark performance and demonstrate our improved numerical accuracy.

Research Organization:
Hebrew Univ. of Jerusalem (Israel); Sandia National Lab. (SNL-CA), Livermore, CA (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE National Nuclear Security Administration (NNSA); Israel Science Foundation (ISF); Ministry of Science and Technology (Israel); Einstein Foundation; Minerva Foundation (United States); Intel Collaborative Research Inst. for Computational Intelligence (ICRI-CI) (Israel); United States-Israel Binational Science Foundation (BSF); HUJI Cyber Security Research Center (Israel); Israel National Cyber Bureau
Grant/Contract Number:
AC04-94AL85000; AC02-05CH11231; 1878/14; 1901/14; 3-10891
OSTI ID:
1356986
Alternate ID(s):
OSTI ID: 1458476
Report Number(s):
SAND2015-5246J; 594464
Journal Information:
SIAM Journal on Matrix Analysis and Applications, Vol. 37, Issue 4; ISSN 0895-4798
Publisher:
SIAMCopyright Statement
Country of Publication:
United States
Language:
English
Citation Metrics:
Cited by: 13 works
Citation information provided by
Web of Science

References (15)

Stability of fast algorithms for matrix multiplication journal March 1980
On the complexity of the multiplication of matrices of small formats journal February 2003
Error analysis of algorithms for matrix multiplication and triangular decomposition using Winograd's identity journal November 1970
The Better Accuracy of Strassen-Winograd Algorithms (FastMMW) journal January 2014
Fast linear algebra is stable journal October 2007
Fast matrix multiplication is stable journal February 2007
Improving and estimating the accuracy of Strassen's algorithm journal June 1998
The vec-permutation matrix, the vec operator and Kronecker products: a review journal January 1981
Noncommutative Bilinear Algorithms for $3 \times 3$ Matrix Multiplication journal May 1986
A practical algorithm for faster matrix multiplication journal December 1999
The aggregation and cancellation techniques as a practical tool for faster matrix multiplication journal May 2004
New Lower Bounds for the Rank of Matrix Multiplication journal January 2014
Computational Complexity and Numerical Stability journal June 1975
The bilinear complexity and practical algorithms for matrix multiplication journal December 2013
Gaussian elimination is not optimal journal August 1969

Cited By (1)

Investigating Bayesian Optimization for rail network optimization journal October 2019

Similar Records

NSFnets (Navier-Stokes flow nets): Physics-informed neural networks for the incompressible Navier-Stokes equations
Journal Article · Sun Nov 01 00:00:00 EDT 2020 · Journal of Computational Physics · OSTI ID:1356986
+1 more
Randomized Algorithms for Scientific Computing (RASC)
Technical Report · Sat Jul 10 00:00:00 EDT 2021 · OSTI ID:1356986
+16 more
Far-field compression for fast kernel summation methods in high dimensions
Journal Article · Sat Jul 01 00:00:00 EDT 2017 · Applied and Computational Harmonic Analysis · OSTI ID:1356986

Related Subjects

97 MATHEMATICS AND COMPUTING
practical fast matrix multiplication
error bounds
diagonal scaling