General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification

Zhang, Kaiqi; Hawkins, Cole; Zhang, Zheng

doi:10.3389/frai.2021.668353

General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification

Journal Article · Thu Jan 06 00:00:00 EST 2022 · Frontiers in Artificial Intelligence

DOI:https://doi.org/10.3389/frai.2021.668353· OSTI ID:2953825

Zhang, Kaiqi ^[1]; Hawkins, Cole ^[1]; Zhang, Zheng ^[1]

University of California, Santa Barbara, CA (United States)

A major challenge in many machine learning tasks is that the model expressive power depends on model size. Low-rank tensor methods are an efficient tool for handling the curse of dimensionality in many large-scale machine learning models. The major challenges in training a tensor learning model include how to process the high-volume data, how to determine the tensor rank automatically, and how to estimate the uncertainty of the results. While existing tensor learning focuses on a specific task, this paper proposes a generic Bayesian framework that can be employed to solve a broad class of tensor learning problems such as tensor completion, tensor regression, and tensorized neural networks. We develop a low-rank tensor prior for automatic rank determination in nonlinear problems. Our method is implemented with both stochastic gradient Hamiltonian Monte Carlo (SGHMC) and Stein Variational Gradient Descent (SVGD). We compare the automatic rank determination and uncertainty quantification of these two solvers. We demonstrate that our proposed method can determine the tensor rank automatically and can quantify the uncertainty of the obtained results. We validate our framework on tensor completion tasks and tensorized neural network training tasks.

View Accepted Manuscript (DOE)

Research Organization:: University of California, Santa Barbara, CA (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR). Computational Science Research & Partnerships (CSRP)

Grant/Contract Number:: SC0021323

OSTI ID:: 2953825

Journal Information:: Frontiers in Artificial Intelligence, Journal Name: Frontiers in Artificial Intelligence Vol. 4; ISSN 2624-8212

Publisher:: Frontiers Media SACopyright Statement

Country of Publication:: United States

Language:: English

References (19)

Some mathematical notes on three-mode factor analysis Tucker, Ledyard R. Psychometrika, Vol. 31, Issue 3 https://doi.org/10.1007/BF02289464	journal	September 1966
Hybrid Monte Carlo Duane, Simon; Kennedy, A. D.; Pendleton, Brian J. Physics Letters B, Vol. 195, Issue 2 https://doi.org/10.1016/0370-2693(87)91197-X	journal	September 1987
PARAFAC. Tutorial and applications Bro, Rasmus Chemometrics and Intelligent Laboratory Systems, Vol. 38, Issue 2 https://doi.org/10.1016/S0169-7439(97)00032-4	journal	October 1997
Scalable tensor factorizations for incomplete data Acar, Evrim; Dunlavy, Daniel M.; Kolda, Tamara G. Chemometrics and Intelligent Laboratory Systems, Vol. 106, Issue 1 https://doi.org/10.1016/j.chemolab.2010.08.004	journal	March 2011
Bayesian tensorized neural networks with automatic rank selection Hawkins, Cole; Zhang, Zheng Neurocomputing, Vol. 453 https://doi.org/10.1016/j.neucom.2021.04.117	journal	September 2021
An Introduction to MCMC for Machine Learning Andrieu, Christophe; de Freitas, Nando; Doucet, Arnaud Machine Learning, Vol. 50, Issue 1/2, p. 5-43 https://doi.org/10.1023/A:1020281327116	journal	January 2003
Generalized Leapfrog Methods Iserles, A. IMA Journal of Numerical Analysis, Vol. 6, Issue 4 https://doi.org/10.1093/imanum/6.4.381	journal	January 1986
T-Net: Parametrizing Fully Convolutional Nets With a Single High-Order Tensor Kossaifi, Jean; Bulat, Adrian; Tzimiropoulos, Georgios 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2019.00801	conference	June 2019
Tensor Dropout for Robust Learning Kolbeinsson, Arinbjorn; Kossaifi, Jean; Panagakis, Yannis IEEE Journal of Selected Topics in Signal Processing, Vol. 15, Issue 3 https://doi.org/10.1109/JSTSP.2021.3064182	journal	April 2021
Enabling High-Dimensional Hierarchical Uncertainty Quantification by ANOVA and Tensor-Train Decomposition Zhang, Zheng; Yang, Xiu; Oseledets, Ivan V. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, Vol. 34, Issue 1 https://doi.org/10.1109/TCAD.2014.2369505	journal	January 2015
Big-Data Tensor Recovery for High-Dimensional Uncertainty Quantification of Process Variations Zhang, Zheng; Weng, Tsui-Wei; Daniel, Luca IEEE Transactions on Components, Packaging and Manufacturing Technology, Vol. 7, Issue 5 https://doi.org/10.1109/TCPMT.2016.2628703	journal	May 2017
Bayesian Robust Tensor Factorization for Incomplete Multiway Data Zhao, Qibin; Zhou, Guoxu; Zhang, Liqing IEEE Transactions on Neural Networks and Learning Systems, Vol. 27, Issue 4 https://doi.org/10.1109/TNNLS.2015.2423694	journal	April 2016
Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination Zhao, Qibin; Zhang, Liqing; Cichocki, Andrzej IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, Issue 9 https://doi.org/10.1109/TPAMI.2015.2392756	journal	September 2015
Unbiased Risk Estimates for Singular Value Thresholding and Spectral Estimators Candes, Emmanuel J.; Sing-Long, Carlos A.; Trzasko, Joshua D. IEEE Transactions on Signal Processing, Vol. 61, Issue 19 https://doi.org/10.1109/TSP.2013.2270464	journal	October 2013
Tensor Decomposition for Signal Processing and Machine Learning Sidiropoulos, Nicholas D.; De Lathauwer, Lieven; Fu, Xiao IEEE Transactions on Signal Processing, Vol. 65, Issue 13 https://doi.org/10.1109/TSP.2017.2690524	journal	July 2017
Tensor Decompositions and Applications Kolda, Tamara G.; Bader, Brett W. SIAM Review, Vol. 51, Issue 3 https://doi.org/10.1137/07070111X	journal	August 2009
Tensor-Train Decomposition Oseledets, I. V. SIAM Journal on Scientific Computing, Vol. 33, Issue 5 https://doi.org/10.1137/090752286	journal	January 2011
Most Tensor Problems Are NP-Hard Hillar, Christopher J.; Lim, Lek-Heng Journal of the ACM, Vol. 60, Issue 6 https://doi.org/10.1145/2512329	journal	November 2013
Speeding up Convolutional Neural Networks with Low Rank Expansions Jaderberg, Max; Vedaldi, Andrea; Zisserman, Andrew Proceedings of the British Machine Vision Conference 2014 https://doi.org/10.5244/C.28.88	conference	January 2014

Similar Records

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

Journal Article · Wed Jan 12 19:00:00 EST 2022 · SIAM Journal on Mathematics of Data Science · OSTI ID:2889955

Bayesian tensorized neural networks with automatic rank selection

Journal Article · Mon May 03 20:00:00 EDT 2021 · Neurocomputing · OSTI ID:2875828

Uncertainty quantification of graph convolution neural network models of evolving processes

Journal Article · Tue Jul 02 20:00:00 EDT 2024 · Computer Methods in Applied Mechanics and Engineering · OSTI ID:2440697

Related Subjects

Bayesian inference
deep learning
tensor decomposition
tensor learning
uncertainty quantification

General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification

Citation Formats

References (19)

Similar Records

Related Subjects