Bayesian tensorized neural networks with automatic rank selection

Hawkins, Cole; Zhang, Zheng

doi:10.1016/j.neucom.2021.04.117

Bayesian tensorized neural networks with automatic rank selection

Journal Article · Tue May 04 00:00:00 EDT 2021 · Neurocomputing

DOI:https://doi.org/10.1016/j.neucom.2021.04.117· OSTI ID:2875828

Hawkins, Cole ^[1]; Zhang, Zheng ^[1]

University of California, Santa Barbara, CA (United States)

Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. In order to address this challenge, this paper proposes a low-rank Bayesian tensorized neural network. Our Bayesian method performs automatic model compression via an adaptive tensor rank determination. We also present approaches for posterior density calculation and maximum a posteriori (MAP) estimation for the end-to-end training of our tensorized neural network. Here, we provide experimental validation on a two-layer fully connected neural network, a 6-layer CNN and a 110-layer residual neural network where our work produces 7.4x to 137x more compact neural networks directly from the training while achieving high prediction accuracy.

View Accepted Manuscript (DOE)

Research Organization:: University of California, Santa Barbara, CA (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE

Grant/Contract Number:: SC0021323

OSTI ID:: 2875828

Journal Information:: Neurocomputing, Journal Name: Neurocomputing Vol. 453; ISSN 0925-2312

Publisher:: Elsevier BVCopyright Statement

Country of Publication:: United States

Language:: English

References (16)

Bayesian Learning for Neural Networks Neal, Radford M. Lecture Notes in Statistics https://doi.org/10.1007/978-1-4612-0745-0	book	January 1996
Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition Carroll, J. Douglas; Chang, Jih-Jie Psychometrika, Vol. 35, Issue 3 https://doi.org/10.1007/BF02310791	journal	September 1970
PARAFAC: Parallel factor analysis Harshman, Richard A.; Lundy, Margaret E. Computational Statistics & Data Analysis, Vol. 18, Issue 1 https://doi.org/10.1016/0167-9473(94)90132-5	journal	August 1994
Tensor completion and low-n-rank tensor recovery via convex optimization Gandy, Silvia; Recht, Benjamin; Yamada, Isao Inverse Problems, Vol. 27, Issue 2 https://doi.org/10.1088/0266-5611/27/2/025010	journal	January 2011
Wide Compression: Tensor Ring Nets Aggarwal, Vaneet; Wang, Wenlin; Eriksson, Brian 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2018.00972	conference	June 2018
Bayesian Robust Tensor Factorization for Incomplete Multiway Data Zhao, Qibin; Zhou, Guoxu; Zhang, Liqing IEEE Transactions on Neural Networks and Learning Systems, Vol. 27, Issue 4 https://doi.org/10.1109/TNNLS.2015.2423694	journal	April 2016
Tensor Completion for Estimating Missing Values in Visual Data Liu, Ji; Musialski, Przemyslaw; Wonka, Peter IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, Issue 1 https://doi.org/10.1109/TPAMI.2012.39	journal	January 2013
Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination Zhao, Qibin; Zhang, Liqing; Cichocki, Andrzej IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 37, Issue 9 https://doi.org/10.1109/TPAMI.2015.2392756	journal	September 2015
Rank Regularization and Bayesian Inference for Tensor Completion and Extrapolation Bazerque, Juan Andres; Mateos, Gonzalo; Giannakis, Georgios B. IEEE Transactions on Signal Processing, Vol. 61, Issue 22 https://doi.org/10.1109/TSP.2013.2278516	journal	November 2013
Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization Recht, Benjamin; Fazel, Maryam; Parrilo, Pablo A. SIAM Review, Vol. 52, Issue 3 https://doi.org/10.1137/070697835	journal	January 2010
Tensor Decompositions and Applications Kolda, Tamara G.; Bader, Brett W. SIAM Review, Vol. 51, Issue 3 https://doi.org/10.1137/07070111X	journal	August 2009
Tensor-Train Decomposition Oseledets, I. V. SIAM Journal on Scientific Computing, Vol. 33, Issue 5 https://doi.org/10.1137/090752286	journal	January 2011
The Alternating Linear Scheme for Tensor Optimization in the Tensor Train Format Holtz, Sebastian; Rohwedder, Thorsten; Schneider, Reinhold SIAM Journal on Scientific Computing, Vol. 34, Issue 2 https://doi.org/10.1137/100818893	journal	January 2012
Alternating Minimal Energy Methods for Linear Systems in Higher Dimensions Dolgov, Sergey V.; Savostyanov, Dmitry V. SIAM Journal on Scientific Computing, Vol. 36, Issue 5 https://doi.org/10.1137/140953289	journal	January 2014
Keeping the neural networks simple by minimizing the description length of the weights Hinton, Geoffrey E.; van Camp, Drew Proceedings of the sixth annual conference on Computational learning theory - COLT '93 https://doi.org/10.1145/168304.168306	conference	January 1993
Most Tensor Problems Are NP-Hard Hillar, Christopher J.; Lim, Lek-Heng Journal of the ACM, Vol. 60, Issue 6 https://doi.org/10.1145/2512329	journal	November 2013

Similar Records

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

Journal Article · Wed Jan 12 19:00:00 EST 2022 · SIAM Journal on Mathematics of Data Science · OSTI ID:2889955

General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification

Journal Article · Wed Jan 05 19:00:00 EST 2022 · Frontiers in Artificial Intelligence · OSTI ID:2953825

Geometry-aware training of factorized layers in tensor Tucker format

Conference · Sat Nov 30 23:00:00 EST 2024 · OSTI ID:2573639

Related Subjects

Low-rank tensor
Neural network compression
Tensor rank determination

Bayesian tensorized neural networks with automatic rank selection

Citation Formats

References (16)

Similar Records

Related Subjects