Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems

Yang, Yibo; Perdikaris, Paris

doi:10.1007/s00466-019-01718-y

Title: Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems

Abstract

Here, we present a probabilistic deep learning methodology that enables the construction of predictive data-driven surrogates for stochastic systems. Leveraging recent advances in variational inference with implicit distributions, we put forth a statistical inference framework that enables the end-to-end training of surrogate models on paired input–output observations that may be stochastic in nature, originate from different information sources of variable fidelity, or be corrupted by complex noise processes. The resulting surrogates can accommodate high-dimensional inputs and outputs and are able to return predictions with quantified uncertainty. Furthermore, the effectiveness our approach is demonstrated through a series of canonical studies, including the regression of noisy data, multi-fidelity modeling of stochastic processes, and uncertainty propagation in high-dimensional dynamical systems.

Authors:

Yang, Yibo ^[1];

^[1]

Univ. of Pennsylvania, Philadelphia, PA (United States)

Publication Date:: 2019-05-21

Research Org.:: Univ. of Pennsylvania, Philadelphia, PA (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1595804

Grant/Contract Number:: SC0019116; HR00111890034

Resource Type:: Accepted Manuscript

Journal Name:: Computational Mechanics

Additional Journal Information:: Journal Volume: 64; Journal Issue: 2; Related Information: https://github.com/PredictiveIntelligenceLab/CADGMs; Journal ID: ISSN 0178-7675

Publisher:: Springer

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Probabilistic deep learning; Generative adversarial networks; Variational inference; Multi- delity modeling; Data-driven surrogates

Citation Formats


                    Yang, Yibo, and Perdikaris, Paris. Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems.  United States: N. p., 2019. 
Web.  doi:10.1007/s00466-019-01718-y.

Copy to clipboard


                    Yang, Yibo, & Perdikaris, Paris. Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems.  United States.  https://doi.org/10.1007/s00466-019-01718-y

Copy to clipboard


                    Yang, Yibo, and Perdikaris, Paris. Tue .  
"Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems".  United States.  https://doi.org/10.1007/s00466-019-01718-y.  https://www.osti.gov/servlets/purl/1595804.

Copy to clipboard


                    
@article{osti_1595804,

  title        = {Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems},

  author       = {Yang, Yibo and Perdikaris, Paris},

  abstractNote = {Here, we present a probabilistic deep learning methodology that enables the construction of predictive data-driven surrogates for stochastic systems. Leveraging recent advances in variational inference with implicit distributions, we put forth a statistical inference framework that enables the end-to-end training of surrogate models on paired input–output observations that may be stochastic in nature, originate from different information sources of variable fidelity, or be corrupted by complex noise processes. The resulting surrogates can accommodate high-dimensional inputs and outputs and are able to return predictions with quantified uncertainty. Furthermore, the effectiveness our approach is demonstrated through a series of canonical studies, including the regression of noisy data, multi-fidelity modeling of stochastic processes, and uncertainty propagation in high-dimensional dynamical systems.},

  doi          = {10.1007/s00466-019-01718-y},

  journal      = {Computational Mechanics},

  number       = 2,

  volume       = 64,

  place        = {United States},

  year         = {2019},

  month        = {5}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1007/s00466-019-01718-y

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 2 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Multi-fidelity optimization via surrogate modelling
journal, September 2007

Forrester, Alexander I. J.; Sóbester, András; Keane, Andy J.
Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 463, Issue 2088
DOI: 10.1098/rspa.2007.1900

Surrogate-Based Optimization Using Multifidelity Models with Variable Parameterization and Corrected Space Mapping
journal, November 2008

Robinson, T. D.; Eldred, M. S.; Willcox, K. E.
AIAA Journal, Vol. 46, Issue 11
DOI: 10.2514/1.36043

Approximation and Model Management in Aerodynamic Optimization with Variable-Fidelity Models
journal, November 2001

Alexandrov, Natalia M.; Lewis, Robert Michael; Gumbert, Clyde R.
Journal of Aircraft, Vol. 38, Issue 6
DOI: 10.2514/2.2877

A two-stage multi-fidelity optimization procedure for honeycomb-type cellular materials
journal, September 2010

Sun, Guangyong; Li, Guangyao; Stone, Michael
Computational Materials Science, Vol. 49, Issue 3
DOI: 10.1016/j.commatsci.2010.05.041

Multi-fidelity optimization for sheet metal forming process
journal, December 2010

Sun, Guangyong; Li, Guangyao; Zhou, Shiwei
Structural and Multidisciplinary Optimization, Vol. 44, Issue 1
DOI: 10.1007/s00158-010-0596-5

DDDAS-based multi-fidelity simulation framework for supply chain systems
journal, February 2010

Celik, Nurcin; Lee, Seungho; Vasudevan, Karthik
IIE Transactions, Vol. 42, Issue 5
DOI: 10.1080/07408170903394306

Model inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond
journal, May 2016

Perdikaris, Paris; Karniadakis, George Em
Journal of The Royal Society Interface, Vol. 13, Issue 118
DOI: 10.1098/rsif.2015.1107

Comparison of Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Quantification
conference, June 2012

Eldred, Michael; Burkardt, John
47th AIAA Aerospace Sciences Meeting including The New Horizons Forum and Aerospace Exposition
DOI: 10.2514/6.2009-976

Multi-Fidelity Uncertainty Quantification: Application to a Vertical Axis Wind Turbine Under an Extreme Gust
conference, June 2014

Padron, Andres S.; Alonso, Juan J.; Palacios, Francisco
15th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference
DOI: 10.2514/6.2014-3013

Towards efficient uncertainty quantification in complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme
journal, September 2014

Biehler, Jonas; Gee, Michael W.; Wall, Wolfgang A.
Biomechanics and Modeling in Mechanobiology, Vol. 14, Issue 3
DOI: 10.1007/s10237-014-0618-0

Optimal Model Management for Multifidelity Monte Carlo Estimation
journal, January 2016

Peherstorfer, Benjamin; Willcox, Karen; Gunzburger, Max
SIAM Journal on Scientific Computing, Vol. 38, Issue 5
DOI: 10.1137/15M1046472

Multifidelity importance sampling
journal, March 2016

Peherstorfer, Benjamin; Cui, Tiangang; Marzouk, Youssef
Computer Methods in Applied Mechanics and Engineering, Vol. 300
DOI: 10.1016/j.cma.2015.12.002

A Stochastic Collocation Algorithm with Multifidelity Models
journal, January 2014

Narayan, Akil; Gittelson, Claude; Xiu, Dongbin
SIAM Journal on Scientific Computing, Vol. 36, Issue 2
DOI: 10.1137/130929461

Computational Aspects of Stochastic Collocation with Multifidelity Models
journal, January 2014

Zhu, Xueyu; Narayan, Akil; Xiu, Dongbin
SIAM/ASA Journal on Uncertainty Quantification, Vol. 2, Issue 1
DOI: 10.1137/130949154

Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification
journal, May 2013

Bilionis, Ilias; Zabaras, Nicholas; Konomi, Bledar A.
Journal of Computational Physics, Vol. 241
DOI: 10.1016/j.jcp.2013.01.011

Multi-fidelity Gaussian process regression for prediction of random fields
journal, May 2017

Parussini, L.; Venturi, D.; Perdikaris, P.
Journal of Computational Physics, Vol. 336
DOI: 10.1016/j.jcp.2017.01.047

Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets
journal, January 2016

Perdikaris, Paris; Venturi, Daniele; Karniadakis, George Em
SIAM Journal on Scientific Computing, Vol. 38, Issue 4
DOI: 10.1137/15M1055164

Extracting and composing robust features with denoising autoencoders
conference, January 2008

Vincent, Pascal; Larochelle, Hugo; Bengio, Yoshua
Proceedings of the 25th international conference on Machine learning - ICML '08
DOI: 10.1145/1390156.1390294

Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach
journal, August 2016

Gómez-Bombarelli, Rafael; Aguilera-Iparraguirre, Jorge; Hirzel, Timothy D.
Nature Materials, Vol. 15, Issue 10
DOI: 10.1038/nmat4717

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules
journal, January 2018

Gómez-Bombarelli, Rafael; Wei, Jennifer N.; Duvenaud, David
ACS Central Science, Vol. 4, Issue 2
DOI: 10.1021/acscentsci.7b00572

Bayesian inference with optimal maps
journal, October 2012

El Moselhy, Tarek A.; Marzouk, Youssef M.
Journal of Computational Physics, Vol. 231, Issue 23
DOI: 10.1016/j.jcp.2012.07.022

Variational Inference: A Review for Statisticians
journal, July 2016

Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
Journal of the American Statistical Association, Vol. 112, Issue 518
DOI: 10.1080/01621459.2017.1285773

Graphical Models, Exponential Families, and Variational Inference
journal, January 2007

Wainwright, Martin J.; Jordan, Michael I.
Foundations and Trends® in Machine Learning, Vol. 1, Issue 1–2
DOI: 10.1561/2200000001

Predictive collective variable discovery with deep Bayesian models
journal, January 2019

Schöberl, Markus; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios
The Journal of Chemical Physics, Vol. 150, Issue 2
DOI: 10.1063/1.5058063

A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
journal, November 2019

Grigo, Constantin; Koutsourelakis, Phaedon-Stelios
Journal of Computational Physics, Vol. 397
DOI: 10.1016/j.jcp.2019.05.053

Predicting the output from a complex computer code when fast approximations are available
journal, March 2000

Kennedy, M.
Biometrika, Vol. 87, Issue 1
DOI: 10.1093/biomet/87.1.1

Fourth-Order Time-Stepping for Stiff PDEs
journal, January 2005

Kassam, Aly-Khan; Trefethen, Lloyd N.
SIAM Journal on Scientific Computing, Vol. 26, Issue 4
DOI: 10.1137/S1064827502410633

Deep learning
journal, May 2015

LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
Nature, Vol. 521, Issue 7553
DOI: 10.1038/nature14539

Understanding deep convolutional networks
journal, April 2016

Mallat, Stéphane
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 374, Issue 2065
DOI: 10.1098/rsta.2015.0203

Active Learning with Statistical Models
journal, January 1996

Cohn, D. A.; Ghahramani, Z.; Jordan, M. I.
Journal of Artificial Intelligence Research, Vol. 4
DOI: 10.1613/jair.295

Taking the Human Out of the Loop: A Review of Bayesian Optimization
journal, January 2016

Shahriari, Bobak; Swersky, Kevin; Wang, Ziyu
Proceedings of the IEEE, Vol. 104, Issue 1
DOI: 10.1109/JPROC.2015.2494218

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules.
text, January 2018

Gómez-Bombarelli, Rafael; Wei, Jennifer N.; Duvenaud, David
Apollo - University of Cambridge Repository
DOI: 10.17863/cam.34134

Variational Inference: A Review for Statisticians
text, January 2017

Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
Taylor & Francis
DOI: 10.6084/m9.figshare.5203696.v1

Variational Inference: A Review for Statisticians
text, January 2017

Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
Taylor & Francis
DOI: 10.6084/m9.figshare.5203696

Deep Learning
text, January 2018

Ganatra, Sanket
Zenodo
DOI: 10.5281/zenodo.1189359

Variational Inference: A Review for Statisticians
text, January 2016

Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
arXiv
DOI: 10.48550/arxiv.1601.00670

Active Learning with Statistical Models
text, January 1996

Cohn, D. A.; Ghahramani, Z.; Jordan, M. I.
arXiv
DOI: 10.48550/arxiv.cs/9603104

Similar Records in DOE PAGES and OSTI.GOV collections:

Adversarial uncertainty quantification in physics-informed neural networks

Journal Article Yang, Yibo ; Perdikaris, Paris - Journal of Computational Physics

Here, we introduce a deep learning framework for quantifying and propagating uncertainty in systems governed by non-linear differential equations using physics-informed neural networks. Specifically, we employ latent variable models to construct probabilistic representations for the system states, and put forth an adversarial inference procedure for training them on data, while constraining their predictions to satisfy given physical laws expressed by partial differential equations. Such physics-informed constraints provide a regularization mechanism for effectively training deep generative models as surrogates of physical systems in which the cost of data acquisition is high, and training data-sets are typically small. This provides a flexiblemore »« less
Cited by 12
https://doi.org/10.1016/j.jcp.2019.05.027

Full Text Available
Probabilistic neural networks for fluid flow surrogate modeling and data recovery

Journal Article Maulik, Romit ; Fukami, Kai ; Ramachandra, Nesar ; ... - Physical Review Fluids

We consider the use of probabilistic neural networks for fluid flow surrogate modeling and data recovery. This framework is constructed by assuming that the target variables are sampled from a Gaussian distribution conditioned on the inputs. Consequently, the overall formulation sets up a procedure to predict the hyperparameters of this distribution which are then used to compute an objective function given training data. We demonstrate that this framework has the ability to provide for prediction confidence intervals based on the assumption of a probabilistic posterior, given an appropriate model architecture and adequate training data. The applicability of the present frameworkmore »« less
https://doi.org/10.1103/physrevfluids.5.104401

Full Text Available
Multi-fidelity information fusion with concatenated neural networks

Journal Article Pawar, Suraj ; San, Omer ; Vedula, Prakash ; ... - Scientific Reports

Abstract Recently, computational modeling has shifted towards the use of statistical inference, deep learning, and other data-driven modeling frameworks. Although this shift in modeling holds promise in many applications like design optimization and real-time control by lowering the computational burden, training deep learning models needs a huge amount of data. This big data is not always available for scientific problems and leads to poorly generalizable data-driven models. This gap can be furnished by leveraging information from physics-based models. Exploiting prior knowledge about the problem at hand, this study puts forth a physics-guided machine learning (PGML) approach to build more tailored,more »« less
https://doi.org/10.1038/s41598-022-09938-8

Full Text Available
Data-Consistent Solutions to Stochastic Inverse Problems Using a Probabilistic Multi-Fidelity Method Based on Conditional Densities

Journal Article Bruder, L. ; Gee, M. W. ; Wildey, Timothy - International Journal for Uncertainty Quantification

In this work, we build upon a recently developed approach for solving stochastic inverse problems based on a combination of measure-theoretic principles and Bayes' rule. We propose a multi-fidelity method to reduce the computational burden of performing uncertainty quantification using high-fidelity models. This approach is based on a Monte Carlo framework for uncertainty quantification that combines information from solvers of various fidelities to obtain statistics on the quantities of interest of the problem. In particular, our goal is to generate samples from a high-fidelity push-forward density at a fraction of the costs of standard Monte Carlo methods, while maintaining flexibilitymore »« less
https://doi.org/10.1615/int.j.uncertaintyquantification.2020030092

Full Text Available
Gaussian processes with built-in dimensionality reduction: Applications to high-dimensional uncertainty propagation

Journal Article Tripathy, Rohit ; Bilionis, Ilias ; Gonzalez, Marcial - Journal of Computational Physics

Uncertainty quantification (UQ) tasks, such as model calibration, uncertainty propagation, and optimization under uncertainty, typically require several thousand evaluations of the underlying computer codes. To cope with the cost of simulations, one replaces the real response surface with a cheap surrogate based, e.g., on polynomial chaos expansions, neural networks, support vector machines, or Gaussian processes (GP). However, the number of simulations required to learn a generic multivariate response grows exponentially as the input dimension increases. This curse of dimensionality can only be addressed, if the response exhibits some special structure that can be discovered and exploited. A wide range ofmore »« less
https://doi.org/10.1016/J.JCP.2016.05.039

Similar Records

Title: Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems

Abstract

Citation Formats

Multi-fidelity optimization via surrogate modelling journal, September 2007

Surrogate-Based Optimization Using Multifidelity Models with Variable Parameterization and Corrected Space Mapping journal, November 2008

Approximation and Model Management in Aerodynamic Optimization with Variable-Fidelity Models journal, November 2001

A two-stage multi-fidelity optimization procedure for honeycomb-type cellular materials journal, September 2010

Multi-fidelity optimization for sheet metal forming process journal, December 2010

DDDAS-based multi-fidelity simulation framework for supply chain systems journal, February 2010

Model inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond journal, May 2016

Comparison of Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Quantification conference, June 2012

Multi-Fidelity Uncertainty Quantification: Application to a Vertical Axis Wind Turbine Under an Extreme Gust conference, June 2014

Towards efficient uncertainty quantification in complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme journal, September 2014

Optimal Model Management for Multifidelity Monte Carlo Estimation journal, January 2016

Multifidelity importance sampling journal, March 2016

A Stochastic Collocation Algorithm with Multifidelity Models journal, January 2014

Computational Aspects of Stochastic Collocation with Multifidelity Models journal, January 2014

Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification journal, May 2013

Multi-fidelity Gaussian process regression for prediction of random fields journal, May 2017

Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets journal, January 2016

Extracting and composing robust features with denoising autoencoders conference, January 2008

Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach journal, August 2016

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules journal, January 2018

Bayesian inference with optimal maps journal, October 2012

Variational Inference: A Review for Statisticians journal, July 2016

Graphical Models, Exponential Families, and Variational Inference journal, January 2007

Predictive collective variable discovery with deep Bayesian models journal, January 2019

A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime journal, November 2019

Predicting the output from a complex computer code when fast approximations are available journal, March 2000

Fourth-Order Time-Stepping for Stiff PDEs journal, January 2005

Deep learning journal, May 2015

Understanding deep convolutional networks journal, April 2016

Active Learning with Statistical Models journal, January 1996

Taking the Human Out of the Loop: A Review of Bayesian Optimization journal, January 2016

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules. text, January 2018

Variational Inference: A Review for Statisticians text, January 2017

Variational Inference: A Review for Statisticians text, January 2017

Deep Learning text, January 2018

Variational Inference: A Review for Statisticians text, January 2016

Active Learning with Statistical Models text, January 1996

Multi-fidelity optimization via surrogate modelling
journal, September 2007

Surrogate-Based Optimization Using Multifidelity Models with Variable Parameterization and Corrected Space Mapping
journal, November 2008

Approximation and Model Management in Aerodynamic Optimization with Variable-Fidelity Models
journal, November 2001

A two-stage multi-fidelity optimization procedure for honeycomb-type cellular materials
journal, September 2010

Multi-fidelity optimization for sheet metal forming process
journal, December 2010

DDDAS-based multi-fidelity simulation framework for supply chain systems
journal, February 2010

Model inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond
journal, May 2016

Comparison of Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Quantification
conference, June 2012

Multi-Fidelity Uncertainty Quantification: Application to a Vertical Axis Wind Turbine Under an Extreme Gust
conference, June 2014

Towards efficient uncertainty quantification in complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme
journal, September 2014

Optimal Model Management for Multifidelity Monte Carlo Estimation
journal, January 2016

Multifidelity importance sampling
journal, March 2016

A Stochastic Collocation Algorithm with Multifidelity Models
journal, January 2014

Computational Aspects of Stochastic Collocation with Multifidelity Models
journal, January 2014

Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification
journal, May 2013

Multi-fidelity Gaussian process regression for prediction of random fields
journal, May 2017

Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets
journal, January 2016

Extracting and composing robust features with denoising autoencoders
conference, January 2008

Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach
journal, August 2016

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules
journal, January 2018

Bayesian inference with optimal maps
journal, October 2012

Variational Inference: A Review for Statisticians
journal, July 2016

Graphical Models, Exponential Families, and Variational Inference
journal, January 2007

Predictive collective variable discovery with deep Bayesian models
journal, January 2019

A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
journal, November 2019

Predicting the output from a complex computer code when fast approximations are available
journal, March 2000

Fourth-Order Time-Stepping for Stiff PDEs
journal, January 2005

Deep learning
journal, May 2015

Understanding deep convolutional networks
journal, April 2016

Active Learning with Statistical Models
journal, January 1996

Taking the Human Out of the Loop: A Review of Bayesian Optimization
journal, January 2016

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules.
text, January 2018

Variational Inference: A Review for Statisticians
text, January 2017

Variational Inference: A Review for Statisticians
text, January 2017

Deep Learning
text, January 2018

Variational Inference: A Review for Statisticians
text, January 2016

Active Learning with Statistical Models
text, January 1996