Stochastic Gradients for Large-Scale Tensor Decomposition

Kolda, Tamara G.; Hong, David

doi:10.1137/19m1266265

Stochastic Gradients for Large-Scale Tensor Decomposition

Journal Article · Tue Oct 27 00:00:00 EDT 2020 · SIAM Journal on Mathematics of Data Science

DOI:https://doi.org/10.1137/19m1266265· OSTI ID:1738932

^[1]; Hong, David ^[2]

Sandia National Lab. (SNL-CA), Livermore, CA (United States)
Univ. of Michigan, Ann Arbor, MI (United States)

Tensor decomposition is a well-known tool for multiway data analysis. This work proposes using stochastic gradients for efficient generalized canonical polyadic (GCP) tensor decomposition of large-scale tensors. GCP tensor decomposition is a recently proposed version of tensor decomposition that allows for a variety of loss functions such as Bernoulli loss for binary data or Huber loss for robust estimation. Here, the stochastic gradient is formed from randomly sampled elements of the tensor and is efficient because it can be computed using the sparse matricized-tensor times Khatri--Rao product tensor kernel. For dense tensors, we simply use uniform sampling. For sparse tensors, we propose two types of stratified sampling that give precedence to sampling nonzeros. Numerical results demonstrate the advantages of the proposed approach and its scalability to large-scale problems.

View Accepted Manuscript (DOE)

Research Organization:: Sandia National Laboratories (SNL-CA), Livermore, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

Grant/Contract Number:: AC04-94AL85000

OSTI ID:: 1738932

Alternate ID(s):: OSTI ID: 1767900

Report Number(s):: SAND--2020-13780J; 692878

Journal Information:: SIAM Journal on Mathematics of Data Science, Journal Name: SIAM Journal on Mathematics of Data Science Journal Issue: 4 Vol. 2; ISSN 2577-0187

Publisher:: Society for Industrial and Applied Mathematics (SIAM)Copyright Statement

Country of Publication:: United States

Language:: English

References (26)

On the performance of gas sensor arrays in open sampling systems using Inhibitory Support Vector Machines Vergara, Alexander; Fonollosa, Jordi; Mahiques, Jonas Sensors and Actuators B: Chemical, Vol. 185 https://doi.org/10.1016/j.snb.2013.05.027	journal	August 2013
Correcting systematic effects in a large set of photometric light curves Tamuz, O.; Mazeh, T.; Zucker, S. Monthly Notices of the Royal Astronomical Society, Vol. 356, Issue 4 https://doi.org/10.1111/j.1365-2966.2004.08585.x	journal	February 2005
A Practical Randomized CP Tensor Decomposition Battaglino, Casey; Ballard, Grey; Kolda, Tamara G. SIAM Journal on Matrix Analysis and Applications, Vol. 39, Issue 2 https://doi.org/10.1137/17m1112303	journal	January 2018
A scalable optimization approach for fitting canonical tensor decompositions Acar, Evrim; Dunlavy, Daniel M.; Kolda, Tamara G. Journal of Chemometrics, Vol. 25, Issue 2 https://doi.org/10.1002/cem.1335	journal	January 2011
Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition Carroll, J. Douglas; Chang, Jih-Jie Psychometrika, Vol. 35, Issue 3 https://doi.org/10.1007/BF02310791	journal	September 1970
Stochastic gradient descent, weighted sampling, and the randomized Kaczmarz algorithm Needell, Deanna; Srebro, Nathan; Ward, Rachel Mathematical Programming, Vol. 155, Issue 1-2 https://doi.org/10.1007/s10107-015-0864-7	journal	February 2015
Positive tensor factorization Welling, Max; Weber, Markus Pattern Recognition Letters, Vol. 22, Issue 12 https://doi.org/10.1016/S0167-8655(01)00070-8	journal	October 2001
Scalable tensor factorizations for incomplete data Acar, Evrim; Dunlavy, Daniel M.; Kolda, Tamara G. Chemometrics and Intelligent Laboratory Systems, Vol. 106, Issue 1 https://doi.org/10.1016/j.chemolab.2010.08.004	journal	March 2011
Statistically weighted principal component analysis of rapid scanning wavelength kinetics experiments Cochran, Robert N.; Horne, Frederick H. Analytical Chemistry, Vol. 49, Issue 6 https://doi.org/10.1021/ac50014a045	journal	May 1977
Newton-based optimization for Kullback–Leibler nonnegative tensor factorizations Hansen, Samantha; Plantenga, Todd; Kolda, Tamara G. Optimization Methods and Software, Vol. 30, Issue 5 https://doi.org/10.1080/10556788.2015.1009977	journal	February 2015
Analysis of longitudinal metabolomics data Jansen, J. J.; Hoefsloot, H. C. J.; Boelens, H. F. M. Bioinformatics, Vol. 20, Issue 15 https://doi.org/10.1093/bioinformatics/bth268	journal	April 2004
A Randomized Block Sampling Approach to Canonical Polyadic Decomposition of Large-Scale Tensors Vervliet, Nico; De Lathauwer, Lieven IEEE Journal of Selected Topics in Signal Processing, Vol. 10, Issue 2 https://doi.org/10.1109/JSTSP.2015.2503260	journal	March 2016
Matrix Factorization Techniques for Recommender Systems Koren, Yehuda; Bell, Robert; Volinsky, Chris Computer, Vol. 42, Issue 8 https://doi.org/10.1109/MC.2009.263	journal	August 2009
Adaptive Algorithms to Track the PARAFAC Decomposition of a Third-Order Tensor Nion, D.; Sidiropoulos, N. D. IEEE Transactions on Signal Processing, Vol. 57, Issue 6 https://doi.org/10.1109/TSP.2009.2016885	journal	June 2009
Fast Alternating LS Algorithms for High Order CANDECOMP/PARAFAC Tensor Factorizations Phan, Anh-Huy; Tichavsky, Petr; Cichocki, Andrzej IEEE Transactions on Signal Processing, Vol. 61, Issue 19 https://doi.org/10.1109/TSP.2013.2269903	journal	October 2013
Subspace Learning and Imputation for Streaming Big Data Matrices and Tensors Mardani, Morteza; Mateos, Gonzalo; Giannakis, Georgios B. IEEE Transactions on Signal Processing, Vol. 63, Issue 10 https://doi.org/10.1109/TSP.2015.2417491	journal	May 2015
Efficient MATLAB Computations with Sparse and Factored Tensors Bader, Brett W.; Kolda, Tamara G. SIAM Journal on Scientific Computing, Vol. 30, Issue 1 https://doi.org/10.1137/060676489	journal	January 2008
Tensor Decompositions and Applications Kolda, Tamara G.; Bader, Brett W. SIAM Review, Vol. 51, Issue 3 https://doi.org/10.1137/07070111X	journal	August 2009
A Limited Memory Algorithm for Bound Constrained Optimization Byrd, Richard H.; Lu, Peihuang; Nocedal, Jorge SIAM Journal on Scientific Computing, Vol. 16, Issue 5 https://doi.org/10.1137/0916069	journal	September 1995
On Tensors, Sparsity, and Nonnegative Factorizations Chi, Eric C.; Kolda, Tamara G. SIAM Journal on Matrix Analysis and Applications, Vol. 33, Issue 4 https://doi.org/10.1137/110859063	journal	January 2012
A Scalable Generative Graph Model with Community Structure Kolda, Tamara G.; Pinar, Ali; Plantenga, Todd SIAM Journal on Scientific Computing, Vol. 36, Issue 5 https://doi.org/10.1137/130914218	journal	January 2014
A Practical Randomized CP Tensor Decomposition Battaglino, Casey; Ballard, Grey; Kolda, Tamara G. SIAM Journal on Matrix Analysis and Applications, Vol. 39, Issue 2 https://doi.org/10.1137/17M1112303	journal	January 2018
Generalized Canonical Polyadic Tensor Decomposition Hong, David; Kolda, Tamara G.; Duersch, Jed A. SIAM Review, Vol. 62, Issue 1 https://doi.org/10.1137/18M1203626	journal	January 2020
Software for Sparse Tensor Decomposition on Emerging Computing Architectures Phipps, Eric T.; Kolda, Tamara G. SIAM Journal on Scientific Computing, Vol. 41, Issue 3 https://doi.org/10.1137/18M1210691	journal	January 2019
Computational Advertising: Techniques for Targeting Relevant Ads Woodruff, David P. Foundations and Trends® in Theoretical Computer Science, Vol. 10, Issue 1-2 https://doi.org/10.1561/0400000060	journal	January 2014
Generalized Low Rank Models Udell, Madeleine; Horn, Corinne; Zadeh, Reza Foundations and Trends® in Machine Learning, Vol. 9, Issue 1 https://doi.org/10.1561/2200000055	journal	January 2016

Cited By (1)

Practical Leverage-Based Sampling for Low-Rank Tensor Decomposition Larsen, Brett W.; Kolda, Tamara G. arXiv https://doi.org/10.48550/arxiv.2006.16438	preprint	January 2020

Similar Records

GentenMPI: Distributed Memory Sparse Tensor Decomposition

Technical Report · Sat Aug 01 00:00:00 EDT 2020 · OSTI ID:1656940

Streaming Generalized Canonical Polyadic Tensor Decompositions

Technical Report · Fri Oct 01 00:00:00 EDT 2021 · OSTI ID:1832304

Software for Sparse Tensor Decomposition on Emerging Computing Architectures

Journal Article · Thu Jun 20 00:00:00 EDT 2019 · SIAM Journal on Scientific Computing · OSTI ID:1542146

Related Subjects

97 MATHEMATICS AND COMPUTING
stochastic gradients
stochastic optimization
strati fied sampling
tensor decomposition

Stochastic Gradients for Large-Scale Tensor Decomposition

Citation Formats

References (26)

Cited By (1)

Similar Records

Related Subjects