Gradient-based optimization for regression in the functional tensor-train format
Abstract
Predictive analysis of complex computational models, such as uncertainty quantification (UQ), must often rely on using an existing database of simulation runs. In this paper we consider the task of performing low-multilinear-rank regression on such a database. Specifically we develop and analyze an efficient gradient computation that enables gradient-based optimization procedures, including stochastic gradient descent and quasi-Newton methods, for learning the parameters of a functional tensor-train (FT). We compare our algorithms with 22 other nonparametric and parametric regression methods on 10 real-world data sets and show that for many physical systems, exploiting low-rank structure facilitates efficient construction of surrogate models. Here, we use a number of synthetic functions to build insight into behavior of our algorithms, including the rank adaptation and group-sparsity regularization procedures that we developed to reduce overfitting. Finally we conclude the paper by building a surrogate of a physical model of a propulsion plant on a naval vessel.
- Authors:
-
- Univ. of Michigan, Ann Arbor, MI (United States)
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States). Optimization and Uncertainty Quantification
- Publication Date:
- Research Org.:
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Sponsoring Org.:
- USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- OSTI Identifier:
- 1485822
- Alternate Identifier(s):
- OSTI ID: 1702087
- Report Number(s):
- SAND-2018-13399J
Journal ID: ISSN 0021-9991; 670535
- Grant/Contract Number:
- AC04-94AL85000; NA-0003525
- Resource Type:
- Journal Article: Accepted Manuscript
- Journal Name:
- Journal of Computational Physics
- Additional Journal Information:
- Journal Volume: 374; Journal Issue: C; Journal ID: ISSN 0021-9991
- Publisher:
- Elsevier
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; Tensors; Regression; Function approximation; Uncertainty quantification; Alternating least squares; Stochastic gradient descent
Citation Formats
Gorodetsky, Alex A., and Jakeman, John D. Gradient-based optimization for regression in the functional tensor-train format. United States: N. p., 2018.
Web. doi:10.1016/j.jcp.2018.08.010.
Gorodetsky, Alex A., & Jakeman, John D. Gradient-based optimization for regression in the functional tensor-train format. United States. https://doi.org/10.1016/j.jcp.2018.08.010
Gorodetsky, Alex A., and Jakeman, John D. 2018.
"Gradient-based optimization for regression in the functional tensor-train format". United States. https://doi.org/10.1016/j.jcp.2018.08.010. https://www.osti.gov/servlets/purl/1485822.
@article{osti_1485822,
title = {Gradient-based optimization for regression in the functional tensor-train format},
author = {Gorodetsky, Alex A. and Jakeman, John D.},
abstractNote = {Predictive analysis of complex computational models, such as uncertainty quantification (UQ), must often rely on using an existing database of simulation runs. In this paper we consider the task of performing low-multilinear-rank regression on such a database. Specifically we develop and analyze an efficient gradient computation that enables gradient-based optimization procedures, including stochastic gradient descent and quasi-Newton methods, for learning the parameters of a functional tensor-train (FT). We compare our algorithms with 22 other nonparametric and parametric regression methods on 10 real-world data sets and show that for many physical systems, exploiting low-rank structure facilitates efficient construction of surrogate models. Here, we use a number of synthetic functions to build insight into behavior of our algorithms, including the rank adaptation and group-sparsity regularization procedures that we developed to reduce overfitting. Finally we conclude the paper by building a surrogate of a physical model of a propulsion plant on a naval vessel.},
doi = {10.1016/j.jcp.2018.08.010},
url = {https://www.osti.gov/biblio/1485822},
journal = {Journal of Computational Physics},
issn = {0021-9991},
number = C,
volume = 374,
place = {United States},
year = {Tue Aug 14 00:00:00 EDT 2018},
month = {Tue Aug 14 00:00:00 EDT 2018}
}
Web of Science
Works referenced in this record:
A scalable optimization approach for fitting canonical tensor decompositions
journal, January 2011
- Acar, Evrim; Dunlavy, Daniel M.; Kolda, Tamara G.
- Journal of Chemometrics, Vol. 25, Issue 2
Spectral Tensor-Train Decomposition
journal, January 2016
- Bigoni, Daniele; Engsig-Karup, Allan P.; Marzouk, Youssef M.
- SIAM Journal on Scientific Computing, Vol. 38, Issue 4
Adaptive sparse polynomial chaos expansion based on least angle regression
journal, March 2011
- Blatman, Géraud; Sudret, Bruno
- Journal of Computational Physics, Vol. 230, Issue 6
Analysis of individual differences in multidimensional scaling via an n-way generalization of “Eckart-Young” decomposition
journal, September 1970
- Carroll, J. Douglas; Chang, Jih-Jie
- Psychometrika, Vol. 35, Issue 3
A Least-Squares Method for Sparse Low Rank Approximation of Multivariate Functions
journal, January 2015
- Chevreuil, M.; Lebrun, R.; Nouy, A.
- SIAM/ASA Journal on Uncertainty Quantification, Vol. 3, Issue 1
Adaptive Smolyak Pseudospectral Approximations
journal, January 2013
- Conrad, Patrick R.; Marzouk, Youssef M.
- SIAM Journal on Scientific Computing, Vol. 35, Issue 6
Sparse pseudospectral approximation method
journal, July 2012
- Constantine, Paul G.; Eldred, Michael S.; Phipps, Eric T.
- Computer Methods in Applied Mechanics and Engineering, Vol. 229-232
A non-adapted sparse approximation of PDEs with stochastic inputs
journal, April 2011
- Doostan, Alireza; Owhadi, Houman
- Journal of Computational Physics, Vol. 230, Issue 8
Non-intrusive low-rank separated approximation of high-dimensional stochastic models
journal, August 2013
- Doostan, Alireza; Validi, AbdoulAhad; Iaccarino, Gianluca
- Computer Methods in Applied Mechanics and Engineering, Vol. 263
Multi-element probabilistic collocation method in high dimensions
journal, March 2010
- Foo, Jasmine; Karniadakis, George Em
- Journal of Computational Physics, Vol. 229, Issue 5
Sparse grid collocation schemes for stochastic natural convection problems
journal, July 2007
- Ganapathysubramanian, Baskar; Zabaras, Nicholas
- Journal of Computational Physics, Vol. 225, Issue 1
Mercer Kernels and Integrated Variance Experimental Design: Connections Between Gaussian Process Regression and Polynomial Approximation
journal, January 2016
- Gorodetsky, Alex; Marzouk, Youssef
- SIAM/ASA Journal on Uncertainty Quantification, Vol. 4, Issue 1
Hierarchical Singular Value Decomposition of Tensors
journal, January 2010
- Grasedyck, Lars
- SIAM Journal on Matrix Analysis and Applications, Vol. 31, Issue 4
Variants of Alternating Least Squares Tensor Completion in the Tensor Train Format
journal, January 2015
- Grasedyck, Lars; Kluge, Melanie; Krämer, Sebastian
- SIAM Journal on Scientific Computing, Vol. 37, Issue 5
A New Scheme for the Tensor Representation
journal, October 2009
- Hackbusch, W.; Kühn, S.
- Journal of Fourier Analysis and Applications, Vol. 15, Issue 5
Enhancing -minimization estimates of polynomial chaos expansions using basis selection
journal, May 2015
- Jakeman, J. D.; Eldred, M. S.; Sargsyan, K.
- Journal of Computational Physics, Vol. 289
Tensor Decompositions and Applications
journal, August 2009
- Kolda, Tamara G.; Bader, Brett W.
- SIAM Review, Vol. 51, Issue 3
An adaptive high-dimensional stochastic model representation technique for the solution of stochastic partial differential equations
journal, May 2010
- Ma, Xiang; Zabaras, Nicholas
- Journal of Computational Physics, Vol. 229, Issue 10
Quantification of Uncertainty from High-Dimensional Scattered data via Polynomial Approximation
journal, January 2014
- Mathelin, Lionel
- International Journal for Uncertainty Quantification, Vol. 4, Issue 3
A Compressed Sensing Approach for Partial Differential Equations with Random Input Data
journal, October 2012
- Mathelin, L.; Gallivan, K. A.
- Communications in Computational Physics, Vol. 12, Issue 4
High-dimensional additive modeling
journal, December 2009
- Meier, Lukas; van de Geer, Sara; Bühlmann, Peter
- The Annals of Statistics, Vol. 37, Issue 6B
An Anisotropic Sparse Grid Stochastic Collocation Method for Partial Differential Equations with Random Input Data
journal, January 2008
- Nobile, F.; Tempone, R.; Webster, C. G.
- SIAM Journal on Numerical Analysis, Vol. 46, Issue 5
Tensor-Train Decomposition
journal, January 2011
- Oseledets, I. V.
- SIAM Journal on Scientific Computing, Vol. 33, Issue 5
Constructive Representation of Functions in Low-Rank Tensor Formats
journal, December 2012
- Oseledets, I. V.
- Constructive Approximation, Vol. 37, Issue 1
TT-cross approximation for multidimensional arrays
journal, January 2010
- Oseledets, Ivan; Tyrtyshnikov, Eugene
- Linear Algebra and its Applications, Vol. 432, Issue 1
Low rank tensor recovery via iterative hard thresholding
journal, June 2017
- Rauhut, Holger; Schneider, Reinhold; Stojanac, Željka
- Linear Algebra and its Applications, Vol. 523
Polynomial-Chaos-Based Kriging
journal, January 2015
- Schobi, Roland; Sudret, Bruno; Wiart, Joe
- International Journal for Uncertainty Quantification, Vol. 5, Issue 2
Simultaneous Variable Selection
journal, August 2005
- Turlach, Berwin A.; Venables, William N.; Wright, Stephen J.
- Technometrics, Vol. 47, Issue 3
Local Convergence of the Alternating Least Squares Algorithm for Canonical Tensor Approximation
journal, January 2012
- Uschmajew, André
- SIAM Journal on Matrix Analysis and Applications, Vol. 33, Issue 2
The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations
journal, January 2002
- Xiu, Dongbin; Karniadakis, George Em
- SIAM Journal on Scientific Computing, Vol. 24, Issue 2
Model selection and estimation in regression with grouped variables
journal, February 2006
- Yuan, Ming; Lin, Yi
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 68, Issue 1
Non-intrusive low-rank separated approximation of high-dimensional stochastic models
journal, August 2013
- Doostan, Alireza; Validi, AbdoulAhad; Iaccarino, Gianluca
- Computer Methods in Applied Mechanics and Engineering, Vol. 263
Sparse grid collocation schemes for stochastic natural convection problems
journal, July 2007
- Ganapathysubramanian, Baskar; Zabaras, Nicholas
- Journal of Computational Physics, Vol. 225, Issue 1
Multi-element probabilistic collocation method in high dimensions
journal, March 2010
- Foo, Jasmine; Karniadakis, George Em
- Journal of Computational Physics, Vol. 229, Issue 5
Adaptive sparse polynomial chaos expansion based on least angle regression
journal, March 2011
- Blatman, Géraud; Sudret, Bruno
- Journal of Computational Physics, Vol. 230, Issue 6
Enhancing -minimization estimates of polynomial chaos expansions using basis selection
journal, May 2015
- Jakeman, J. D.; Eldred, M. S.; Sargsyan, K.
- Journal of Computational Physics, Vol. 289
Fast adaptive interpolation of multi-dimensional arrays in tensor train format
conference, September 2011
- Savostyanov, Dmitry; Oseledets, Ivan
- The 2011 International Workshop on Multidimensional (nD) Systems
Optimization Methods for Large-Scale Machine Learning
journal, January 2018
- Bottou, Léon; Curtis, Frank E.; Nocedal, Jorge
- SIAM Review, Vol. 60, Issue 2
A non-adapted sparse approximation of PDEs with stochastic inputs
text, January 2010
- Doostan, Alireza; Owhadi, Houman
- arXiv
Quantification of uncertainty from high-dimensional scattered data via polynomial approximation
preprint, January 2013
- Mathelin, Lionel
- arXiv
Alternating Least Squares Tensor Completion in The TT-Format
preprint, January 2015
- Grasedyck, Lars; Kluge, Melanie; Krämer, Sebastian
- arXiv
Works referencing / citing this record:
Sparse low-rank separated representation models for learning from data
journal, January 2019
- Audouze, Christophe; Nair, Prasanth B.
- Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 475, Issue 2221
Adaptive multi‐index collocation for uncertainty quantification and sensitivity analysis
journal, November 2019
- Jakeman, John D.; Eldred, Michael S.; Geraci, Gianluca
- International Journal for Numerical Methods in Engineering, Vol. 121, Issue 6
Adaptive multi‐index collocation for uncertainty quantification and sensitivity analysis
journal, August 2020
- Jakeman, John D.; Eldred, Michael S.; Geraci, Gianluca
- International Journal for Numerical Methods in Engineering, Vol. 121, Issue 19
Adaptive multi‐index collocation for uncertainty quantification and sensitivity analysis
journal, November 2019
- Jakeman, John D.; Eldred, Michael S.; Geraci, Gianluca
- International Journal for Numerical Methods in Engineering, Vol. 121, Issue 6
Introductory overview of identifiability analysis: A guide to evaluating whether you have the right type of data for your modeling purpose
journal, September 2019
- Guillaume, Joseph H. A.; Jakeman, John D.; Marsili-Libelli, Stefano
- Environmental Modelling & Software, Vol. 119
MGA: Momentum Gradient Attack on Network
journal, February 2021
- Chen, Jinyin; Chen, Yixian; Zheng, Haibin
- IEEE Transactions on Computational Social Systems, Vol. 8, Issue 1
Cholesky-based experimental design for Gaussian process and kernel-based emulation and calibration
text, January 2021
- Helmut, Harbrecht,; D., Jakeman, John; Peter, Zaspel,
- Global Science Press