Improving the Numerical Stability of Fast Matrix Multiplication

Ballard, Grey; Benson, Austin R.; Druinsky, Alex; Lipshitz, Benjamin; Schwartz, Oded

doi:10.1137/15M1032168

Improving the Numerical Stability of Fast Matrix Multiplication

Journal Article · Tue Oct 04 00:00:00 EDT 2016 · SIAM Journal on Matrix Analysis and Applications

DOI:https://doi.org/10.1137/15M1032168· OSTI ID:1356986

Ballard, Grey ^[1]; Benson, Austin R. ^[2]; Druinsky, Alex ^[3]; Lipshitz, Benjamin ^[4]; Schwartz, Oded ^[5]

Sandia National Lab. (SNL-CA), Livermore, CA (United States); Wake Forest Univ., Winston-Salem, NC (United States)
Stanford Univ., CA (United States). Inst. for Computational and Mathematical Engineering
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Google, San Bruno, CA (United States)
Hebrew Univ. of Jerusalem (Israel). School of Computer Science and Engineering

Fast algorithms for matrix multiplication, namely those that perform asymptotically fewer scalar operations than the classical algorithm, have been considered primarily of theoretical interest. Apart from Strassen's original algorithm, few fast algorithms have been efficiently implemented or used in practical applications. However, there exist many practical alternatives to Strassen's algorithm with varying performance and numerical properties. Fast algorithms are known to be numerically stable, but because their error bounds are slightly weaker than the classical algorithm, they are not used even in cases where they provide a performance benefit. We argue in this study that the numerical sacrifice of fast algorithms, particularly for the typical use cases of practical algorithms, is not prohibitive, and we explore ways to improve the accuracy both theoretically and empirically. The numerical accuracy of fast matrix multiplication depends on properties of the algorithm and of the input matrices, and we consider both contributions independently. We generalize and tighten previous error analyses of fast algorithms and compare their properties. We discuss algorithmic techniques for improving the error guarantees from two perspectives: manipulating the algorithms, and reducing input anomalies by various forms of diagonal scaling. In conclusion, we benchmark performance and demonstrate our improved numerical accuracy.

Research Organization:: Hebrew Univ. of Jerusalem (Israel); Sandia National Lab. (SNL-CA), Livermore, CA (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE National Nuclear Security Administration (NNSA); Israel Science Foundation (ISF); Ministry of Science and Technology (Israel); Einstein Foundation; Minerva Foundation (United States); Intel Collaborative Research Inst. for Computational Intelligence (ICRI-CI) (Israel); United States-Israel Binational Science Foundation (BSF); HUJI Cyber Security Research Center (Israel); Israel National Cyber Bureau

Grant/Contract Number:: AC04-94AL85000; AC02-05CH11231

OSTI ID:: 1356986

Alternate ID(s):: OSTI ID: 1458476

Report Number(s):: SAND2015--5246J; 594464

Journal Information:: SIAM Journal on Matrix Analysis and Applications, Journal Name: SIAM Journal on Matrix Analysis and Applications Journal Issue: 4 Vol. 37; ISSN 0895-4798

Publisher:: SIAMCopyright Statement

Country of Publication:: United States

Language:: English

References (15)

A practical algorithm for faster matrix multiplication Kaporin, Igor Numerical Linear Algebra with Applications, Vol. 6, Issue 8 https://doi.org/10.1002/(SICI)1099-1506(199912)6:8<687::AID-NLA177>3.0.CO;2-I	journal	December 1999
Stability of fast algorithms for matrix multiplication Bini, Dario; Lotti, Grazia Numerische Mathematik, Vol. 36, Issue 1 https://doi.org/10.1007/BF01395989	journal	March 1980
Gaussian elimination is not optimal Strassen, Volker Numerische Mathematik, Vol. 13, Issue 4 https://doi.org/10.1007/BF02165411	journal	August 1969
Error analysis of algorithms for matrix multiplication and triangular decomposition using Winograd's identity Brent, R. P. Numerische Mathematik, Vol. 16, Issue 2 https://doi.org/10.1007/BF02308867	journal	November 1970
Fast matrix multiplication is stable Demmel, James; Dumitriu, Ioana; Holtz, Olga Numerische Mathematik, Vol. 106, Issue 2 https://doi.org/10.1007/s00211-007-0061-6	journal	February 2007
Fast linear algebra is stable Demmel, James; Dumitriu, Ioana; Holtz, Olga Numerische Mathematik, Vol. 108, Issue 1 https://doi.org/10.1007/s00211-007-0114-x	journal	October 2007
Improving and estimating the accuracy of Strassen's algorithm Dumitrescu, Bogdan Numerische Mathematik, Vol. 79, Issue 4 https://doi.org/10.1007/s002110050348	journal	June 1998
On the complexity of the multiplication of matrices of small formats Bläser, Markus Journal of Complexity, Vol. 19, Issue 1 https://doi.org/10.1016/S0885-064X(02)00007-9	journal	February 2003
The aggregation and cancellation techniques as a practical tool for faster matrix multiplication Kaporin, Igor Theoretical Computer Science, Vol. 315, Issue 2-3 https://doi.org/10.1016/j.tcs.2004.01.004	journal	May 2004
The vec-permutation matrix, the vec operator and Kronecker products: a review Henderson, Harold V.; Searle, S. R. Linear and Multilinear Algebra, Vol. 9, Issue 4 https://doi.org/10.1080/03081088108817379	journal	January 1981
The bilinear complexity and practical algorithms for matrix multiplication Smirnov, A. V. Computational Mathematics and Mathematical Physics, Vol. 53, Issue 12 https://doi.org/10.1134/S0965542513120129	journal	December 2013
Computational Complexity and Numerical Stability Miller, Webb SIAM Journal on Computing, Vol. 4, Issue 2 https://doi.org/10.1137/0204009	journal	June 1975
Noncommutative Bilinear Algorithms for $3 \times 3$ Matrix Multiplication Johnson, Rodney W.; McLoughlin, Aileen M. SIAM Journal on Computing, Vol. 15, Issue 2 https://doi.org/10.1137/0215043	journal	May 1986
New Lower Bounds for the Rank of Matrix Multiplication Landsberg, J. M. SIAM Journal on Computing, Vol. 43, Issue 1 https://doi.org/10.1137/120880276	journal	January 2014
The Better Accuracy of Strassen-Winograd Algorithms (FastMMW) D’Alberto, Paolo Advances in Linear Algebra & Matrix Theory, Vol. 04, Issue 01 https://doi.org/10.4236/alamt.2014.41002	journal	January 2014

Cited By (1)

Investigating Bayesian Optimization for rail network optimization Hickish, Bob; Fletcher, David I.; Harrison, Robert F. International Journal of Rail Transportation, Vol. 8, Issue 4 https://doi.org/10.1080/23248378.2019.1669500	journal	October 2019

Similar Records

A Tensor Product Formulation of Strassen′s Matrix Multiplication Algorithm with Memory Reduction

Journal Article · Mon Apr 17 00:00:00 EDT 1995 · Scientific Programming · OSTI ID:1198030

Using Strassen's algorithm to accelerate the solution of linear systems

Journal Article · Sun Dec 31 23:00:00 EST 1989 · Journal of Supercomputing; (United States) · OSTI ID:6689180

Related Subjects

97 MATHEMATICS AND COMPUTING
diagonal scaling
error bounds
practical fast matrix multiplication

Improving the Numerical Stability of Fast Matrix Multiplication

Citation Formats

References (15)

Cited By (1)

Similar Records

Related Subjects