Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problems

Abstract

Constructing surrogate models for uncertainty quantification (UQ) on complex partial differential equations (PDEs) having inherently high-dimensional O(10n), n ≥ 2, stochastic inputs (e.g., forcing terms, boundary conditions, initial conditions) poses tremendous challenges. The “curse of dimensionality” can be addressed with suitable unsupervised learning techniques used as a pre-processing tool to encode inputs onto lower-dimensional subspaces while retaining its structural information and meaningful properties. In this work, we review and investigate thirteen dimension reduction methods including linear and nonlinear, spectral, blind source separation, convex and non-convex methods and utilize the resulting embeddings to construct a mapping to quantities of interest via polynomial chaos expansions (PCE). Here, we refer to the general proposed approach as manifold PCE (m-PCE), where manifold corresponds to the latent space resulting from any of the studied dimension reduction methods. To investigate the capabilities and limitations of these methods we conduct numerical tests for three physics-based systems (treated as black-boxes) having high-dimensional stochastic inputs of varying complexity modeled as both Gaussian and non-Gaussian random fields to investigate the effect of the intrinsic dimensionality of input data. We demonstrate both the advantages and limitations of the unsupervised learning methods and we conclude that a suitable m-PCE model provides amore » cost-effective approach compared to alternative algorithms proposed in the literature, including recently proposed expensive deep neural network-based surrogates and can be readily applied for high-dimensional UQ in stochastic PDEs.« less

Authors:
 [1];  [2];  [1];  [1];  [1]
+ Show Author Affiliations
  1. Johns Hopkins Univ., Baltimore, MD (United States)
  2. Darmstadt Univ. of Technology (Germany); Siemens AG, Munich (Germany)
Publication Date:
Research Org.:
Johns Hopkins Univ., Baltimore, MD (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1906444
Alternate Identifier(s):
OSTI ID: 1870935
Grant/Contract Number:  
SC0020428
Resource Type:
Accepted Manuscript
Journal Name:
Journal of Computational Physics
Additional Journal Information:
Journal Volume: 464; Journal ID: ISSN 0021-9991
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
42 ENGINEERING; 97 MATHEMATICS AND COMPUTING; High-dimensional uncertainty quantification; Dimension reduction; Unsupervised learning; Surrogate modeling; Manifold learning; Low-dimensional embedding

Citation Formats

Kontolati, Katiana, Loukrezis, Dimitrios, Giovanis, Dimitrios G., Vandanapu, Lohit, and Shields, Michael D. A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problems. United States: N. p., 2022. Web. doi:10.1016/j.jcp.2022.111313.
Copy to clipboard
Kontolati, Katiana, Loukrezis, Dimitrios, Giovanis, Dimitrios G., Vandanapu, Lohit, & Shields, Michael D. A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problems. United States. https://doi.org/10.1016/j.jcp.2022.111313
Copy to clipboard
Kontolati, Katiana, Loukrezis, Dimitrios, Giovanis, Dimitrios G., Vandanapu, Lohit, and Shields, Michael D. Mon . "A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problems". United States. https://doi.org/10.1016/j.jcp.2022.111313. https://www.osti.gov/servlets/purl/1906444.
Copy to clipboard
@article{osti_1906444,
title = {A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problems},
author = {Kontolati, Katiana and Loukrezis, Dimitrios and Giovanis, Dimitrios G. and Vandanapu, Lohit and Shields, Michael D.},
abstractNote = {Constructing surrogate models for uncertainty quantification (UQ) on complex partial differential equations (PDEs) having inherently high-dimensional O(10n), n ≥ 2, stochastic inputs (e.g., forcing terms, boundary conditions, initial conditions) poses tremendous challenges. The “curse of dimensionality” can be addressed with suitable unsupervised learning techniques used as a pre-processing tool to encode inputs onto lower-dimensional subspaces while retaining its structural information and meaningful properties. In this work, we review and investigate thirteen dimension reduction methods including linear and nonlinear, spectral, blind source separation, convex and non-convex methods and utilize the resulting embeddings to construct a mapping to quantities of interest via polynomial chaos expansions (PCE). Here, we refer to the general proposed approach as manifold PCE (m-PCE), where manifold corresponds to the latent space resulting from any of the studied dimension reduction methods. To investigate the capabilities and limitations of these methods we conduct numerical tests for three physics-based systems (treated as black-boxes) having high-dimensional stochastic inputs of varying complexity modeled as both Gaussian and non-Gaussian random fields to investigate the effect of the intrinsic dimensionality of input data. We demonstrate both the advantages and limitations of the unsupervised learning methods and we conclude that a suitable m-PCE model provides a cost-effective approach compared to alternative algorithms proposed in the literature, including recently proposed expensive deep neural network-based surrogates and can be readily applied for high-dimensional UQ in stochastic PDEs.},
doi = {10.1016/j.jcp.2022.111313},
journal = {Journal of Computational Physics},
number = ,
volume = 464,
place = {United States},
year = {Mon May 23 00:00:00 EDT 2022},
month = {Mon May 23 00:00:00 EDT 2022}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1016/j.jcp.2022.111313
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

Adaptive multi-element polynomial chaos with discrete measure: Algorithms and application to SPDEs
journal, April 2015

Sliced inverse regression-based sparse polynomial chaos expansions for reliability analysis in high dimensions
journal, November 2017

Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
journal, October 2019

  • Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios
  • Journal of Computational Physics, Vol. 394
  • DOI: 10.1016/j.jcp.2019.05.024

ConvPDE-UQ: Convolutional neural networks with quantified uncertainty for heterogeneous elliptic partial differential equations on varied domains
journal, October 2019

Fast and robust fixed-point algorithms for independent component analysis
journal, May 1999

  • Hyvarinen, A.
  • IEEE Transactions on Neural Networks, Vol. 10, Issue 3
  • DOI: 10.1109/72.761722

Resampling Methods for Meta-Model Validation with Recommendations for Evolutionary Computation
journal, June 2012

  • Bischl, B.; Mersmann, O.; Trautmann, H.
  • Evolutionary Computation, Vol. 20, Issue 2
  • DOI: 10.1162/EVCO_a_00069

Simulation of Stationary Non-Gaussian Translation Processes
journal, February 1998

Diffusion maps
journal, July 2006

  • Coifman, Ronald R.; Lafon, Stéphane
  • Applied and Computational Harmonic Analysis, Vol. 21, Issue 1
  • DOI: 10.1016/j.acha.2006.04.006

A Generalized Sampling and Preconditioning Scheme for Sparse Approximation of Polynomial Chaos Expansions
journal, January 2017

  • Jakeman, John D.; Narayan, Akil; Zhou, Tao
  • SIAM Journal on Scientific Computing, Vol. 39, Issue 3
  • DOI: 10.1137/16M1063885

Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
journal, February 2019

Convergence study of the truncated Karhunen–Loeve expansion for simulation of stochastic processes
journal, November 2001

  • Huang, S. P.; Quek, S. T.; Phoon, K. K.
  • International Journal for Numerical Methods in Engineering, Vol. 52, Issue 9
  • DOI: 10.1002/nme.255

Kernel PCA for novelty detection
journal, March 2007

PLS-based adaptation for efficient PCE representation in high dimensions
journal, June 2019

A generalized multi-fidelity simulation method using sparse polynomial chaos expansion
journal, December 2021

Adversarial uncertainty quantification in physics-informed neural networks
journal, October 2019

Latin hypercube sampling and the propagation of uncertainty in analyses of complex systems
journal, July 2003

A fast Fixed-Point Algorithm for Independent Component Analysis of Complex Valued Signals
journal, February 2000

Spectral Tensor-Train Decomposition
journal, January 2016

  • Bigoni, Daniele; Engsig-Karup, Allan P.; Marzouk, Youssef M.
  • SIAM Journal on Scientific Computing, Vol. 38, Issue 4
  • DOI: 10.1137/15M1036919

DPPro: Differentially Private High-Dimensional Data Release via Random Projection
journal, December 2017

  • Xu, Chugui; Ren, Ju; Zhang, Yaoxue
  • IEEE Transactions on Information Forensics and Security, Vol. 12, Issue 12
  • DOI: 10.1109/TIFS.2017.2737966

Multi-output local Gaussian process regression: Applications to uncertainty quantification
journal, July 2012

Surrogate modeling of high-dimensional problems via data-driven polynomial chaos expansions and sparse partial least square
journal, June 2020

  • Zhou, Yicheng; Lu, Zhenzhou; Hu, Jinghan
  • Computer Methods in Applied Mechanics and Engineering, Vol. 364
  • DOI: 10.1016/j.cma.2020.112906

Overview and comparative study of dimensionality reduction techniques for high dimensional data
journal, July 2020

The generalization of Latin hypercube sampling
journal, April 2016

Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation
journal, September 2016

  • Tripathy, Rohit; Bilionis, Ilias; Gonzalez, Marcial
  • Journal of Computational Physics, Vol. 321
  • DOI: 10.1016/j.jcp.2016.05.039

Reducing the Dimensionality of Data with Neural Networks
journal, July 2006

Active Manifold and Model-Order Reduction to Accelerate Multidisciplinary Analysis and Optimization
journal, November 2021

  • Boncoraglio, Gabriele; Farhat, Charbel
  • AIAA Journal, Vol. 59, Issue 11
  • DOI: 10.2514/1.J060581

Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems
journal, November 2019

Learning the parts of objects by non-negative matrix factorization
journal, October 1999

  • Lee, Daniel D.; Seung, H. Sebastian
  • Nature, Vol. 401, Issue 6755
  • DOI: 10.1038/44565

Principal Manifolds and Nonlinear Dimensionality Reduction via Tangent Space Alignment
journal, January 2004

A Fast Fixed-Point Algorithm for Independent Component Analysis
journal, October 1997

Vector diffusion maps and the connection Laplacian
journal, March 2012

  • Singer, A.; Wu, H. -T.
  • Communications on Pure and Applied Mathematics, Vol. 65, Issue 8
  • DOI: 10.1002/cpa.21395

Optimal Model Management for Multifidelity Monte Carlo Estimation
journal, January 2016

  • Peherstorfer, Benjamin; Willcox, Karen; Gunzburger, Max
  • SIAM Journal on Scientific Computing, Vol. 38, Issue 5
  • DOI: 10.1137/15M1046472

A comparative dimensionality reduction study in telecom customer segmentation using deep learning and PCA
journal, February 2020

Normalizing field flows: Solving forward and inverse stochastic differential equations using physics-informed flow models
journal, July 2022

A survey on feature selection methods
journal, January 2014

Functional multi-layer perceptron: a non-linear tool for functional data analysis
journal, January 2005

The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations
journal, January 2002

Non-intrusive low-rank separated approximation of high-dimensional stochastic models
journal, August 2013

  • Doostan, Alireza; Validi, AbdoulAhad; Iaccarino, Gianluca
  • Computer Methods in Applied Mechanics and Engineering, Vol. 263
  • DOI: 10.1016/j.cma.2013.04.003

Nonnegative Matrix Factorization: A Comprehensive Review
journal, June 2013

  • Wang, Yu-Xiong; Zhang, Yu-Jin
  • IEEE Transactions on Knowledge and Data Engineering, Vol. 25, Issue 6
  • DOI: 10.1109/TKDE.2012.51

Multilevel Monte Carlo methods
journal, April 2015

Gradient-based optimization for regression in the functional tensor-train format
journal, December 2018

Polynomial Chaos in Stochastic Finite Elements
journal, March 1990

  • Ghanem, Roger; Spanos, P. D.
  • Journal of Applied Mechanics, Vol. 57, Issue 1
  • DOI: 10.1115/1.2888303

Reduced Wiener Chaos representation of random fields via basis adaptation and projection
journal, July 2017

Sparse representations and compressive sampling approaches in engineering mechanics: A review of theoretical concepts and diverse applications
journal, July 2020

Extending Classical Surrogate Modeling to high Dimensions Through Supervised Dimensionality Reduction: a Data-Driven Approach
journal, January 2020

Independent component analysis: algorithms and applications
journal, June 2000

Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators
journal, March 2021

A gradient enhanced ℓ1-minimization for sparse approximation of polynomial chaos expansions
journal, August 2018

Grassmannian Diffusion Maps--Based Dimension Reduction and Classification for High-Dimensional Data
journal, March 2022

  • dos Santos, Ketson R.; Giovanis, Dimitrios G.; Shields, Michael D.
  • SIAM Journal on Scientific Computing, Vol. 44, Issue 2
  • DOI: 10.1137/20M137001X

Manifold learning for coarse-graining atomistic simulations: Application to amorphous solids
journal, August 2021

Quantifying uncertain system outputs via the multilevel Monte Carlo method — Part I: Central moment estimation
journal, August 2020

Solving parametric PDE problems with artificial neural networks
journal, July 2020

  • Khoo, Yuehaw; Lu, Jianfeng; Ying, Lexing
  • European Journal of Applied Mathematics, Vol. 32, Issue 3
  • DOI: 10.1017/S0956792520000182

An efficient multifidelity ℓ 1 -minimization method for sparse polynomial chaos
journal, June 2018

  • Salehi, Saeed; Raisee, Mehrdad; Cervantes, Michel J.
  • Computer Methods in Applied Mechanics and Engineering, Vol. 334
  • DOI: 10.1016/j.cma.2018.01.055

Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantification
journal, December 2018

Sparse Legendre expansions via <mml:math altimg="si12.gif" display="inline" overflow="scroll" xmlns:xocs="http://www.elsevier.com/xml/xocs/dtd" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.elsevier.com/xml/ja/dtd" xmlns:ja="http://www.elsevier.com/xml/ja/dtd" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:tb="http://www.elsevier.com/xml/common/table/dtd" xmlns:sb="http://www.elsevier.com/xml/common/struct-bib/dtd" xmlns:ce="http://www.elsevier.com/xml/common/dtd" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:cals="http://www.elsevier.com/xml/common/cals/dtd"><mml:msub><mml:mrow><mml:mi>ℓ</mml:mi></mml:mrow><mml:mrow><mml:mn>1</mml:mn></mml:mrow></mml:msub></mml:math>-minimization
journal, May 2012

A note on two problems in connexion with graphs
journal, December 1959

A Global Geometric Framework for Nonlinear Dimensionality Reduction
journal, December 2000

Algorithm 97: Shortest path
journal, June 1962

Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification
journal, August 2018

A Multilinear Singular Value Decomposition
journal, January 2000

  • De Lathauwer, Lieven; De Moor, Bart; Vandewalle, Joos
  • SIAM Journal on Matrix Analysis and Applications, Vol. 21, Issue 4
  • DOI: 10.1137/S0895479896305696

Surrogate modeling of advanced computer simulations using deep Gaussian processes
journal, March 2020

Past review, current progress, and challenges ahead on the cocktail party problem
journal, January 2018

  • Qian, Yan-min; Weng, Chao; Chang, Xuan-kai
  • Frontiers of Information Technology & Electronic Engineering, Vol. 19, Issue 1
  • DOI: 10.1631/FITEE.1700814

Compressive sensing adaptation for polynomial chaos expansions
journal, March 2019

Remarks on a Multivariate Transformation
journal, September 1952

A generalized approximate control variate framework for multifidelity uncertainty quantification
journal, May 2020

  • Gorodetsky, Alex A.; Geraci, Gianluca; Eldred, Michael S.
  • Journal of Computational Physics, Vol. 408
  • DOI: 10.1016/j.jcp.2020.109257

Robust adaptive least squares polynomial chaos expansions in high‐frequency applications
journal, February 2020

  • Loukrezis, Dimitrios; Galetzka, Armin; De Gersem, Herbert
  • International Journal of Numerical Modelling: Electronic Networks, Devices and Fields, Vol. 33, Issue 6
  • DOI: 10.1002/jnm.2725

Advances in surrogate based modeling, feasibility analysis, and optimization: A review
journal, January 2018

Adaptive sparse polynomial chaos expansion based on least angle regression
journal, March 2011

Data-driven uncertainty quantification using the arbitrary polynomial chaos expansion
journal, October 2012

Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data
journal, April 2003

  • Donoho, D. L.; Grimes, C.
  • Proceedings of the National Academy of Sciences, Vol. 100, Issue 10
  • DOI: 10.1073/pnas.1031596100

Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations
journal, January 2018

  • Raissi, Maziar; Perdikaris, Paris; Karniadakis, George Em
  • SIAM Journal on Scientific Computing, Vol. 40, Issue 1
  • DOI: 10.1137/17M1120762

Integrated stochastic analysis of fiber composites manufacturing using adapted polynomial chaos expansions
journal, March 2019

Accelerated basis adaptation in homogeneous chaos spaces
journal, December 2021

  • Zeng, Xiaoshu; Red-Horse, John; Ghanem, Roger
  • Computer Methods in Applied Mechanics and Engineering, Vol. 386
  • DOI: 10.1016/j.cma.2021.114109

Exploiting active subspaces to quantify uncertainty in the numerical simulation of the HyShot II scramjet
journal, December 2015

Random-projection ensemble classification
journal, June 2017

  • Cannings, Timothy I.; Samworth, Richard J.
  • Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 79, Issue 4
  • DOI: 10.1111/rssb.12228

Nonlinear Dimensionality Reduction by Locally Linear Embedding
journal, December 2000

Basis adaptation in homogeneous chaos spaces
journal, February 2014

A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code
journal, February 2000

Uncertainty propagation using infinite mixture of Gaussian processes and variational Bayesian inference
journal, March 2015

An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis
journal, April 2010

Data-driven polynomial chaos expansion for machine learning regression
journal, July 2019

  • Torre, Emiliano; Marelli, Stefano; Embrechts, Paul
  • Journal of Computational Physics, Vol. 388
  • DOI: 10.1016/j.jcp.2019.03.039

Basis adaptive sample efficient polynomial chaos (BASE-PC)
journal, October 2018

Beyond Wiener–Askey Expansions: Handling Arbitrary PDFs
journal, December 2005

Quasi-Monte Carlo and Multilevel Monte Carlo Methods for Computing Posterior Expectations in Elliptic Inverse Problems
journal, January 2017

  • Scheichl, R.; Stuart, A. M.; Teckentrup, A. L.
  • SIAM/ASA Journal on Uncertainty Quantification, Vol. 5, Issue 1
  • DOI: 10.1137/16M1061692

Data-driven surrogates for high dimensional models using Gaussian process regression on the Grassmann manifold
journal, October 2020

  • Giovanis, D. G.; Shields, M. D.
  • Computer Methods in Applied Mechanics and Engineering, Vol. 370
  • DOI: 10.1016/j.cma.2020.113269

Diffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization
journal, September 2006

  • Lafon, S.; Lee, A. B.
  • IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 28, Issue 9, p. 1393-1403
  • DOI: 10.1109/TPAMI.2006.184

Artificial neural networks for solving ordinary and partial differential equations
journal, January 1998

  • Lagaris, I. E.; Likas, A.; Fotiadis, D. I.
  • IEEE Transactions on Neural Networks, Vol. 9, Issue 5
  • DOI: 10.1109/72.712178

Reliability analysis of high-dimensional models using low-rank tensor approximations
journal, October 2016

Importance sampling: a review
journal, December 2009

  • Tokdar, Surya T.; Kass, Robert E.
  • WIREs Computational Statistics, Vol. 2, Issue 1
  • DOI: 10.1002/wics.56

Bayesian neural networks for uncertainty quantification in data-driven materials modeling
journal, December 2021

  • Olivier, Audrey; Shields, Michael D.; Graham-Brady, Lori
  • Computer Methods in Applied Mechanics and Engineering, Vol. 386
  • DOI: 10.1016/j.cma.2021.114079

A hybrid neural network-first principles approach to process modeling
journal, October 1992

Sparse Polynomial Chaos Expansions: Literature Survey and Benchmark
journal, January 2021

  • Lüthen, Nora; Marelli, Stefano; Sudret, Bruno
  • SIAM/ASA Journal on Uncertainty Quantification, Vol. 9, Issue 2
  • DOI: 10.1137/20M1315774

Homogeneous chaos basis adaptation for design optimization under uncertainty: Application to the oil well placement problem
journal, August 2017

  • Thimmisetty, Charanraj; Tsilifis, Panagiotis; Ghanem, Roger
  • Artificial Intelligence for Engineering Design, Analysis and Manufacturing, Vol. 31, Issue 3
  • DOI: 10.1017/S0890060417000166

Laplacian Eigenmaps for Dimensionality Reduction and Data Representation
journal, June 2003

An adaptive polynomial chaos expansion for high-dimensional reliability analysis
journal, July 2020

Aleatory or epistemic? Does it matter?
journal, March 2009

UQpy: A general purpose Python package and development environment for uncertainty quantification
journal, November 2020

Simulation of Multi-Dimensional Gaussian Stochastic Fields by Spectral Representation
journal, January 1996

  • Shinozuka, Masanobu; Deodatis, George
  • Applied Mechanics Reviews, Vol. 49, Issue 1
  • DOI: 10.1115/1.3101883

Physical Systems with Random Uncertainties: Chaos Representations with Arbitrary Probability Measure
journal, January 2004

Least angle and ℓ1 penalized regression: A review
journal, January 2008

  • Hesterberg, Tim; Choi, Nam Hee; Meier, Lukas
  • Statistics Surveys, Vol. 2, Issue none
  • DOI: 10.1214/08-SS035

The Earth Mover's Distance as a Metric for Image Retrieval
journal, November 2000

Deep learning for high-dimensional reliability analysis
journal, May 2020

Learning in high-dimensional multimedia data: the state of the art
journal, October 2015

    Sort by:
    [ × clear filter / sort ]

    Similar Records in DOE PAGES and OSTI.GOV collections: