Parallel Algorithms for Computing the Tensor-Train Decomposition

Shi, Tianyi; Ruth, Maximilian; Townsend, Alex

doi:10.1137/21m146079x

Parallel Algorithms for Computing the Tensor-Train Decomposition

Journal Article · 07 June 2023 · SIAM Journal on Scientific Computing

DOI:https://doi.org/10.1137/21m146079x· OSTI ID:2326148

Shi, Tianyi ^[1]; Ruth, Maximilian ^[2]; ^[2]

Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Cornell Univ., Ithaca, NY (United States)

The tensor-train (TT) decomposition expresses a tensor in a data-sparse format used in molecular simulations, high-order correlation functions, and optimization. In this paper, we propose four parallelizable algorithms that compute the TT format from various tensor inputs: (1) Parallel-TTSVD for traditional format, (2) PSTT and its variants for streaming data, (3) Tucker2TT for Tucker format, and (4) TT-fADI for solutions of Sylvester tensor equations. We provide theoretical guarantees of accuracy, parallelization methods, scaling analysis, and numerical results. For example, for a d-dimension tensor in $$\mathbb{R}$$^{$$n\times∙∙∙$$$$\times$$$$n$} a two-sided sketching algorithm PSTT2 is shown to have a memory complexity of $$O(n^{[d/2]})$$, improving upon $$O(n^{d—1})$$ from previous algorithms.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); National Science Foundation (NSF)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 2326148

Journal Information:: SIAM Journal on Scientific Computing, Journal Name: SIAM Journal on Scientific Computing Journal Issue: 3 Vol. 45; ISSN 1064-8275

Publisher:: Society for Industrial and Applied Mathematics (SIAM)Copyright Statement

Country of Publication:: United States

Language:: English

References (35)

SUMMA: scalable universal matrix multiplication algorithm Van De Geijn, R. A.; Watts, J. Concurrency: Practice and Experience, Vol. 9, Issue 4 https://doi.org/10.1002/(SICI)1096-9128(199704)9:4<255::AID-CPE250>3.0.CO;2-2	journal	April 1997
Low‐rank solution of an optimal control problem constrained by random Navier‐Stokes equations Benner, Peter; Dolgov, Sergey; Onwunta, Akwum International Journal for Numerical Methods in Fluids, Vol. 92, Issue 11 https://doi.org/10.1002/fld.4843	journal	May 2020
A literature survey of low-rank tensor approximation techniques Grasedyck, Lars; Kressner, Daniel; Tobler, Christine GAMM-Mitteilungen, Vol. 36, Issue 1 https://doi.org/10.1002/gamm.201310004	journal	August 2013
Parallel Algorithms for Low Rank Tensor Arithmetic Grasedyck, Lars; Löbbert, Christian Advances in Mechanics and Mathematics https://doi.org/10.1007/978-3-030-02487-1_16	book	January 2019
Tensor Spaces and Numerical Tensor Calculus Hackbusch, Wolfgang Springer Series in Computational Mathematics https://doi.org/10.1007/978-3-642-28027-6	book	February 2012
Randomized algorithms for the approximations of Tucker and the tensor train decompositions Che, Maolin; Wei, Yimin Advances in Computational Mathematics, Vol. 45, Issue 1 https://doi.org/10.1007/s10444-018-9622-8	journal	July 2018
MERACLE: Constructive Layer-Wise Conversion of a Tensor Train into a MERA Batselier, Kim; Cichocki, Andrzej; Wong, Ngai Communications on Applied Mathematics and Computation, Vol. 3, Issue 2 https://doi.org/10.1007/s42967-020-00090-6	journal	October 2020
N-way principal component analysis theory, algorithms and applications Henrion, René Chemometrics and Intelligent Laboratory Systems, Vol. 25, Issue 1 https://doi.org/10.1016/0169-7439(93)E0086-J	journal	September 1994
On the ADI method for Sylvester equations Benner, Peter; Li, Ren-Cang; Truhar, Ninoslav Journal of Computational and Applied Mathematics, Vol. 233, Issue 4 https://doi.org/10.1016/j.cam.2009.08.108	journal	December 2009
A massively parallel tensor contraction framework for coupled-cluster computations Solomonik, Edgar; Matthews, Devin; Hammond, Jeff R. Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.06.002	journal	December 2014
Dimensionality reduction in higher-order signal processing and rank-(R1,R2,…,RN) reduction in multilinear algebra De Lathauwer, Lieven; Vandewalle, Joos Linear Algebra and its Applications, Vol. 391 https://doi.org/10.1016/j.laa.2004.01.016	journal	November 2004
TT-cross approximation for multidimensional arrays Oseledets, Ivan; Tyrtyshnikov, Eugene Linear Algebra and its Applications, Vol. 432, Issue 1 https://doi.org/10.1016/j.laa.2009.07.024	journal	January 2010
On the singular values of matrices with high displacement rank Townsend, Alex; Wilber, Heather Linear Algebra and its Applications, Vol. 548 https://doi.org/10.1016/j.laa.2018.02.025	journal	July 2018
Fast Poisson solvers for spectral methods Fortunato, Daniel; Townsend, Alex IMA Journal of Numerical Analysis, Vol. 40, Issue 3 https://doi.org/10.1093/imanum/drz034	journal	November 2019
Exact NMR simulation of protein-size spin systems using tensor train formalism Savostyanov, D. V.; Dolgov, S. V.; Werner, J. M. Physical Review B, Vol. 90, Issue 8 https://doi.org/10.1103/PhysRevB.90.085139	journal	August 2014
ADTT: A Highly Efficient Distributed Tensor-Train Decomposition Method for IIoT Big Data Wang, Xiaokang; Yang, Laurence T.; Wang, Yihao IEEE Transactions on Industrial Informatics, Vol. 17, Issue 3 https://doi.org/10.1109/TII.2020.2967768	journal	March 2021
Parallelized Tensor Train Learning of Polynomial Classifiers Chen, Zhongming; Batselier, Kim; Suykens, Johan A. K. IEEE Transactions on Neural Networks and Learning Systems, Vol. 29, Issue 10 https://doi.org/10.1109/TNNLS.2017.2771264	journal	October 2018
Tensor Decompositions and Applications Kolda, Tamara G.; Bader, Brett W. SIAM Review, Vol. 51, Issue 3 https://doi.org/10.1137/07070111X	journal	August 2009
Breaking the Curse of Dimensionality, Or How to Use SVD in Many Dimensions Oseledets, I. V.; Tyrtyshnikov, E. E. SIAM Journal on Scientific Computing, Vol. 31, Issue 5 https://doi.org/10.1137/090748330	journal	January 2009
Tensor-Train Decomposition Oseledets, I. V. SIAM Journal on Scientific Computing, Vol. 33, Issue 5 https://doi.org/10.1137/090752286	journal	January 2011
Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions Halko, N.; Martinsson, P. G.; Tropp, J. A. SIAM Review, Vol. 53, Issue 2 https://doi.org/10.1137/090771806	journal	January 2011
Finite Difference Methods for Ordinary and Partial Differential Equations LeVeque, Randall J. https://doi.org/10.1137/1.9780898717839	book	January 2007
Fast Solution of Parabolic Problems in the Tensor Train/Quantized Tensor Train Format with Initial Application to the Fokker--Planck Equation Dolgov, S. V.; Khoromskij, B. N.; Oseledets, I. V. SIAM Journal on Scientific Computing, Vol. 34, Issue 6 https://doi.org/10.1137/120864210	journal	January 2012
Two-Level QTT-Tucker Format for Optimized Tensor Calculus Dolgov, Sergey; Khoromskij, Boris SIAM Journal on Matrix Analysis and Applications, Vol. 34, Issue 2 https://doi.org/10.1137/120882597	journal	January 2013
Computational Methods for Linear Matrix Equations Simoncini, V. SIAM Review, Vol. 58, Issue 3 https://doi.org/10.1137/130912839	journal	January 2016
Alternating Minimal Energy Methods for Linear Systems in Higher Dimensions Dolgov, Sergey V.; Savostyanov, Dmitry V. SIAM Journal on Scientific Computing, Vol. 36, Issue 5 https://doi.org/10.1137/140953289	journal	January 2014
Low-Rank Tensor Approximation for High-Order Correlation Functions of Gaussian Random Fields Kressner, Daniel; Kumar, Rajesh; Nobile, Fabio SIAM/ASA Journal on Uncertainty Quantification, Vol. 3, Issue 1 https://doi.org/10.1137/140968938	journal	January 2015
Low-Rank Solution to an Optimization Problem Constrained by the Navier--Stokes Equations Dolgov, Sergey; Stoll, Martin SIAM Journal on Scientific Computing, Vol. 39, Issue 1 https://doi.org/10.1137/15M1040414	journal	January 2017
Low-Rank Tucker Approximation of a Tensor from Streaming Data Sun, Yiming; Guo, Yang; Luo, Charlene SIAM Journal on Mathematics of Data Science, Vol. 2, Issue 4 https://doi.org/10.1137/19M1257718	journal	January 2020
Adaptive Hierarchical Subtensor Partitioning for Tensor Compression Ehrlacher, Virginie; Grigori, Laura; Lombardi, Damiano SIAM Journal on Scientific Computing, Vol. 43, Issue 1 https://doi.org/10.1137/19M128689X	journal	January 2021
On the Compressibility of Tensors Shi, Tianyi; Townsend, Alex SIAM Journal on Matrix Analysis and Applications, Vol. 42, Issue 1 https://doi.org/10.1137/20M1316639	journal	January 2021
Parallel Algorithms for Tensor Train Arithmetic Daas, Hussam Al; Ballard, Grey; Benner, Peter SIAM Journal on Scientific Computing, Vol. 44, Issue 1 https://doi.org/10.1137/20M1387158	journal	February 2022
A Multilinear Singular Value Decomposition De Lathauwer, Lieven; De Moor, Bart; Vandewalle, Joos SIAM Journal on Matrix Analysis and Applications, Vol. 21, Issue 4 https://doi.org/10.1137/S0895479896305696	journal	January 2000
TuckerMPI Ballard, Grey; Klinvex, Alicia; Kolda, Tamara G. ACM Transactions on Mathematical Software, Vol. 46, Issue 2 https://doi.org/10.1145/3378445	journal	June 2020
TT-GMRES: solution to a linear system in the structured tensor format Dolgov, S. V. Russian Journal of Numerical Analysis and Mathematical Modelling, Vol. 28, Issue 2 https://doi.org/10.1515/rnam-2013-0009	journal	January 2013