Sequential Learning of Active Subspaces

Wycoff, Nathan; Binois, Mickaël; Wild, Stefan M.

doi:10.1080/10618600.2021.1874962

Sequential Learning of Active Subspaces

Journal Article · Mon Mar 08 00:00:00 EST 2021 · Journal of Computational and Graphical Statistics

DOI:https://doi.org/10.1080/10618600.2021.1874962· OSTI ID:1845719

Wycoff, Nathan ^[1]; ^[1]; ^[1]

Argonne National Lab. (ANL), Lemont, IL (United States)

This study shows that in recent years, active subspace methods (ASMs) have become a popular means of performing subspace sensitivity analysis on black-box functions. Naively applied, however, ASMs require gradient evaluations of the target function. In the event of noisy, expensive, or stochastic simulators, evaluating gradients via finite differencing may be infeasible. In such cases, often a surrogate model is employed, on which finite differencing is performed. When the surrogate model is a Gaussian process (GP), we show that the ASM estimator is available in closed form, rendering the finite-difference approximation unnecessary. We use our closed-form solution to develop acquisition functions focused on sequential learning tailored to sensitivity analysis on top of ASMs. We also show that the traditional ASM estimator may be viewed as a method of moments estimator for a certain class of GPs. We demonstrate how uncertainty on GP hyperparameters may be propagated to uncertainty on the sensitivity analysis, allowing model-based confidence intervals on the active subspace. Our methodological developments are illustrated on several examples.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Lemont, IL (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC02-05CH11231; AC02-06CH11357

OSTI ID:: 1845719

Alternate ID(s):: OSTI ID: 2001243

Journal Information:: Journal of Computational and Graphical Statistics, Journal Name: Journal of Computational and Graphical Statistics Journal Issue: 4 Vol. 30; ISSN 1061-8600

Publisher:: Taylor & FrancisCopyright Statement

Country of Publication:: United States

Language:: English

References (43)

The approximation of one matrix by another of lower rank Eckart, Carl; Young, Gale Psychometrika, Vol. 1, Issue 3 https://doi.org/10.1007/bf02288367	journal	September 1936
Kriging surrogate model with coordinate transformation based on likelihood and gradient Namura, Nobuo; Shimoyama, Koji; Obayashi, Shigeru Journal of Global Optimization, Vol. 68, Issue 4 https://doi.org/10.1007/s10898-017-0516-y	journal	April 2017
Additive Gaussian Processes Duvenaud, David; Nickisch, Hannes; Rasmussen, Carl Edward arXiv https://doi.org/10.48550/arxiv.1112.4394	preprint	January 2011
Data-driven polynomial ridge approximation using variable projection Hokanson, Jeffrey M.; Constantine, Paul G. arXiv https://doi.org/10.48550/arxiv.1702.05859	text	January 2017
Sequential design of experiments for estimating percentiles of black-box functions Picheny, Victor; Labopin-Richard, Tatiana Statistica Sinica https://doi.org/10.5705/ss.202016.0160	journal	January 2018
Engineering Design via Surrogate Modelling Forrester, Alexander I. J.; Sóbester, András; Keane, Andy J. John Wiley & Sons, Ltd https://doi.org/10.1002/9780470770801	book	January 2008
A Review on Global Sensitivity Analysis Methods Iooss, Bertrand; Lemaître, Paul Uncertainty Management in Simulation-Optimization of Complex Systems https://doi.org/10.1007/978-1-4899-7547-8_5	book	January 2015
Gaussian Processes in Machine Learning Rasmussen, Carl Edward Advanced Lectures on Machine Learning https://doi.org/10.1007/978-3-540-28650-9_4	book	January 2004
Inverse regression for ridge recovery: a data-driven approach for parameter reduction in computer experiments Glaws, Andrew; Constantine, Paul G.; Cook, R. Dennis Statistics and Computing, Vol. 30, Issue 2 https://doi.org/10.1007/s11222-019-09876-y	journal	May 2019
Global sensitivity indices for nonlinear mathematical models and their Monte Carlo estimates Sobol′, I. M. Mathematics and Computers in Simulation, Vol. 55, Issue 1-3 https://doi.org/10.1016/S0378-4754(00)00270-6	journal	February 2001
A near-stationary subspace for ridge approximation Constantine, Paul G.; Eftekhari, Armin; Hokanson, Jeffrey Computer Methods in Applied Mechanics and Engineering, Vol. 326 https://doi.org/10.1016/j.cma.2017.07.038	journal	November 2017
Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation Tripathy, Rohit; Bilionis, Ilias; Gonzalez, Marcial Journal of Computational Physics, Vol. 321 https://doi.org/10.1016/j.jcp.2016.05.039	journal	September 2016
Inverse regression-based uncertainty quantification algorithms for high-dimensional models: Theory and practice Li, Weixuan; Lin, Guang; Li, Bing Journal of Computational Physics, Vol. 321 https://doi.org/10.1016/j.jcp.2016.05.040	journal	September 2016
ANOVA kernels and RKHS of zero mean functions for model-based sensitivity analysis Durrande, N.; Ginsbourger, D.; Roustant, O. Journal of Multivariate Analysis, Vol. 115 https://doi.org/10.1016/j.jmva.2012.08.016	journal	March 2013
Derivative-free optimization methods Larson, Jeffrey; Menickelly, Matt; Wild, Stefan M. Acta Numerica, Vol. 28 https://doi.org/10.1017/S0962492919000060	journal	May 2019
Replication or Exploration? Sequential Design for Stochastic Simulation Experiments Binois, Mickaël; Huang, Jiangeng; Gramacy, Robert B. Technometrics, Vol. 61, Issue 1 https://doi.org/10.1080/00401706.2018.1469433	journal	September 2018
Projection Pursuit Regression Friedman, Jerome H.; Stuetzle, Werner Journal of the American Statistical Association, Vol. 76, Issue 376 https://doi.org/10.1080/01621459.1981.10477729	journal	December 1981
Sliced Inverse Regression for Dimension Reduction Li, Ker-Chau Journal of the American Statistical Association, Vol. 86, Issue 414 https://doi.org/10.1080/01621459.1991.10475035	journal	June 1991
On Principal Hessian Directions for Data Visualization and Dimension Reduction: Another Application of Stein's Lemma Li, Ker-Chau Journal of the American Statistical Association, Vol. 87, Issue 420 https://doi.org/10.1080/01621459.1992.10476258	journal	December 1992
Exploring Regression Structure Using Nonparametric Functional Estimation Samarov, Alexander M. Journal of the American Statistical Association, Vol. 88, Issue 423 https://doi.org/10.1080/01621459.1993.10476348	journal	September 1993
Gradient-Based Kernel Dimension Reduction for Regression Fukumizu, Kenji; Leng, Chenlei Journal of the American Statistical Association, Vol. 109, Issue 505 https://doi.org/10.1080/01621459.2013.838167	journal	January 2014
Multiresolution Functional ANOVA for Large-Scale, Many-Input Computer Experiments Sung, Chih-Li; Wang, Wenjia; Plumlee, Matthew Journal of the American Statistical Association, Vol. 115, Issue 530 https://doi.org/10.1080/01621459.2019.1595630	journal	May 2019
New sensitivity analysis subordinated to a contrast Fort, Jean-Claude; Klein, Thierry; Rachdi, Nabil Communications in Statistics - Theory and Methods, Vol. 45, Issue 15 https://doi.org/10.1080/03610926.2014.901369	journal	July 2015
Nonparametric estimates of standard error: The jackknife, the bootstrap and other methods Efron, Bradley Biometrika, Vol. 68, Issue 3 https://doi.org/10.1093/biomet/68.3.589	journal	January 1981
Computing active subspaces efficiently with gradient sketching Constantine, Paul G.; Eftekhari, Armin; Wakin, Michael B. 2015 IEEE 6th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) https://doi.org/10.1109/CAMSAP.2015.7383809	conference	December 2015
A Review on Dimension Reduction: A Review on Dimension Reduction Ma, Yanyuan; Zhu, Liping International Statistical Review, Vol. 81, Issue 1 https://doi.org/10.1111/j.1751-5823.2012.00182.x	journal	December 2012
Active Subspaces Constantine, Paul G. Society for Industrial and Applied Mathematics https://doi.org/10.1137/1.9781611973860	book	January 2015
Active Subspace Methods in Theory and Practice: Applications to Kriging Surfaces Constantine, Paul G.; Dow, Eric; Wang, Qiqi SIAM Journal on Scientific Computing, Vol. 36, Issue 4 https://doi.org/10.1137/130916138	journal	January 2014
Estimation of the Derivative-Based Global Sensitivity Measures Using a Gaussian Process Metamodel De Lozzo, Matthias; Marrel, Amandine SIAM/ASA Journal on Uncertainty Quantification, Vol. 4, Issue 1 https://doi.org/10.1137/15M1013377	journal	January 2016
Data-Driven Polynomial Ridge Approximation Using Variable Projection Hokanson, Jeffrey M.; Constantine, Paul G. SIAM Journal on Scientific Computing, Vol. 40, Issue 3 https://doi.org/10.1137/17M1117690	journal	January 2018
Modified Active Subspaces Using the Average of Gradients Lee, Minyong R. SIAM/ASA Journal on Uncertainty Quantification, Vol. 7, Issue 1 https://doi.org/10.1137/17M1140662	journal	January 2019
A Probabilistic Subspace Bound with Application to Active Subspaces Holodnak, John T.; Ipsen, Ilse C. F.; Smith, Ralph C. SIAM Journal on Matrix Analysis and Applications, Vol. 39, Issue 3 https://doi.org/10.1137/17M1141503	journal	January 2018
Remark on “algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound constrained optimization” Morales, José Luis; Nocedal, Jorge ACM Transactions on Mathematical Software, Vol. 38, Issue 1 https://doi.org/10.1145/2049662.2049669	journal	November 2011
Estimating Derivatives of Noisy Simulations Moré, Jorge J.; Wild, Stefan M. ACM Transactions on Mathematical Software, Vol. 38, Issue 3 https://doi.org/10.1145/2168773.2168777	journal	April 2012
Exploiting active subspaces in global optimization: how complex is your problem? Palar, Pramudita Satria; Shimoyama, Koji GECCO '17: Genetic and Evolutionary Computation Conference, Proceedings of the Genetic and Evolutionary Computation Conference Companion https://doi.org/10.1145/3067695.3082511	conference	July 2017
Bayesian Optimization in a Billion Dimensions via Random Embeddings Wang, Ziyu; Hutter, Frank; Zoghi, Masrour Journal of Artificial Intelligence Research, Vol. 55 https://doi.org/10.1613/jair.4806	journal	January 2016
Dimension Reduction for Aerodynamic Design Optimization Viswanath, Asha; Forrester, A. I. J.; Keane, A. J. AIAA Journal, Vol. 49, Issue 6 https://doi.org/10.2514/1.J050717	journal	June 2011
On Active Subspaces in Car Aerodynamics Othmer, Carsten; Lukaczyk, Trent W.; Constantine, Paul 17th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference https://doi.org/10.2514/6.2016-4294	conference	June 2016
On The Accuracy of Kriging Model in Active Subspaces Palar, Pramudita S.; Shimoyama, Koji 2018 AIAA/ASCE/AHS/ASC Structures, Structural Dynamics, and Materials Conference https://doi.org/10.2514/6.2018-0913	conference	January 2018
Sequential Learning of Active Subspaces Wycoff, Nathan; Binois, Mickaël; Wild, Stefan M. Taylor & Francis https://doi.org/10.6084/m9.figshare.13611284	dataset	January 2021
Sequential Learning of Active Subspaces Wycoff, Nathan; Binois, Mickaël; Wild, Stefan M. Taylor & Francis https://doi.org/10.6084/m9.figshare.13611284.v1	dataset	January 2021
Multi-Resolution Functional ANOVA for Large-Scale, Many-Input Computer Experiments Sung, Chih-Li; Wang, Wenjia; Plumlee, Matthew Taylor & Francis https://doi.org/10.6084/m9.figshare.7869362.v1	dataset	January 2019
Multiresolution Functional ANOVA for Large-Scale, Many-Input Computer Experiments Sung, Chih-Li; Wang, Wenjia; Plumlee, Matthew Taylor & Francis https://doi.org/10.6084/m9.figshare.7869362.v4	dataset	January 2023

Cited By (1)

Batch-sequential design and heteroskedastic surrogate modeling for delta smelt conservation Zhang, Boya; Gramacy, Robert B.; Johnson, Leah arXiv https://doi.org/10.48550/arxiv.2010.06515	preprint	January 2020

Similar Records

Discovering Active Subspaces for High-Dimensional Computer Models

Journal Article · Mon Dec 18 19:00:00 EST 2023 · Journal of Computational and Graphical Statistics · OSTI ID:2217500

Multi-Fidelity Active Subspaces for Wind Farm Uncertainty Quantification

Conference · Sun Jan 03 23:00:00 EST 2021 · OSTI ID:1770896

Related Subjects

97 MATHEMATICS AND COMPUTING
Gaussian processes
active learning
dimension reduction
linear embedding
uncertainty quantification

Sequential Learning of Active Subspaces

Citation Formats

References (43)

Cited By (1)

Linked Research (2)

Similar Records

Related Subjects