skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

This content will become publicly available on May 21, 2020

Title: Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems

Abstract

Here, we present a probabilistic deep learning methodology that enables the construction of predictive data-driven surrogates for stochastic systems. Leveraging recent advances in variational inference with implicit distributions, we put forth a statistical inference framework that enables the end-to-end training of surrogate models on paired input–output observations that may be stochastic in nature, originate from different information sources of variable fidelity, or be corrupted by complex noise processes. The resulting surrogates can accommodate high-dimensional inputs and outputs and are able to return predictions with quantified uncertainty. Furthermore, the effectiveness our approach is demonstrated through a series of canonical studies, including the regression of noisy data, multi-fidelity modeling of stochastic processes, and uncertainty propagation in high-dimensional dynamical systems.

Authors:
 [1]; ORCiD logo [1]
  1. Univ. of Pennsylvania, Philadelphia, PA (United States)
Publication Date:
Research Org.:
Univ. of Pennsylvania, Philadelphia, PA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1595804
Grant/Contract Number:  
[SC0019116; HR00111890034]
Resource Type:
Accepted Manuscript
Journal Name:
Computational Mechanics
Additional Journal Information:
[ Journal Volume: 64; Journal Issue: 2; Related Information: https://github.com/PredictiveIntelligenceLab/CADGMs]; Journal ID: ISSN 0178-7675
Publisher:
Springer
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Probabilistic deep learning; Generative adversarial networks; Variational inference; Multi- delity modeling; Data-driven surrogates

Citation Formats

Yang, Yibo, and Perdikaris, Paris. Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems. United States: N. p., 2019. Web. doi:10.1007/s00466-019-01718-y.
Yang, Yibo, & Perdikaris, Paris. Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems. United States. doi:10.1007/s00466-019-01718-y.
Yang, Yibo, and Perdikaris, Paris. Tue . "Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems". United States. doi:10.1007/s00466-019-01718-y.
@article{osti_1595804,
title = {Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems},
author = {Yang, Yibo and Perdikaris, Paris},
abstractNote = {Here, we present a probabilistic deep learning methodology that enables the construction of predictive data-driven surrogates for stochastic systems. Leveraging recent advances in variational inference with implicit distributions, we put forth a statistical inference framework that enables the end-to-end training of surrogate models on paired input–output observations that may be stochastic in nature, originate from different information sources of variable fidelity, or be corrupted by complex noise processes. The resulting surrogates can accommodate high-dimensional inputs and outputs and are able to return predictions with quantified uncertainty. Furthermore, the effectiveness our approach is demonstrated through a series of canonical studies, including the regression of noisy data, multi-fidelity modeling of stochastic processes, and uncertainty propagation in high-dimensional dynamical systems.},
doi = {10.1007/s00466-019-01718-y},
journal = {Computational Mechanics},
number = [2],
volume = [64],
place = {United States},
year = {2019},
month = {5}
}

Journal Article:
Free Publicly Available Full Text
This content will become publicly available on May 21, 2020
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Multi-fidelity optimization via surrogate modelling
journal, September 2007

  • Forrester, Alexander I. J.; Sóbester, András; Keane, Andy J.
  • Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 463, Issue 2088
  • DOI: 10.1098/rspa.2007.1900

Surrogate-Based Optimization Using Multifidelity Models with Variable Parameterization and Corrected Space Mapping
journal, November 2008

  • Robinson, T. D.; Eldred, M. S.; Willcox, K. E.
  • AIAA Journal, Vol. 46, Issue 11
  • DOI: 10.2514/1.36043

Approximation and Model Management in Aerodynamic Optimization with Variable-Fidelity Models
journal, November 2001

  • Alexandrov, Natalia M.; Lewis, Robert Michael; Gumbert, Clyde R.
  • Journal of Aircraft, Vol. 38, Issue 6
  • DOI: 10.2514/2.2877

A two-stage multi-fidelity optimization procedure for honeycomb-type cellular materials
journal, September 2010


Multi-fidelity optimization for sheet metal forming process
journal, December 2010

  • Sun, Guangyong; Li, Guangyao; Zhou, Shiwei
  • Structural and Multidisciplinary Optimization, Vol. 44, Issue 1
  • DOI: 10.1007/s00158-010-0596-5

DDDAS-based multi-fidelity simulation framework for supply chain systems
journal, February 2010


Model inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond
journal, May 2016

  • Perdikaris, Paris; Karniadakis, George Em
  • Journal of The Royal Society Interface, Vol. 13, Issue 118
  • DOI: 10.1098/rsif.2015.1107

Comparison of Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Quantification
conference, June 2012

  • Eldred, Michael; Burkardt, John
  • 47th AIAA Aerospace Sciences Meeting including The New Horizons Forum and Aerospace Exposition
  • DOI: 10.2514/6.2009-976

Multi-Fidelity Uncertainty Quantification: Application to a Vertical Axis Wind Turbine Under an Extreme Gust
conference, June 2014

  • Padron, Andres S.; Alonso, Juan J.; Palacios, Francisco
  • 15th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference
  • DOI: 10.2514/6.2014-3013

Towards efficient uncertainty quantification in complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme
journal, September 2014

  • Biehler, Jonas; Gee, Michael W.; Wall, Wolfgang A.
  • Biomechanics and Modeling in Mechanobiology, Vol. 14, Issue 3
  • DOI: 10.1007/s10237-014-0618-0

Optimal Model Management for Multifidelity Monte Carlo Estimation
journal, January 2016

  • Peherstorfer, Benjamin; Willcox, Karen; Gunzburger, Max
  • SIAM Journal on Scientific Computing, Vol. 38, Issue 5
  • DOI: 10.1137/15M1046472

Multifidelity importance sampling
journal, March 2016

  • Peherstorfer, Benjamin; Cui, Tiangang; Marzouk, Youssef
  • Computer Methods in Applied Mechanics and Engineering, Vol. 300
  • DOI: 10.1016/j.cma.2015.12.002

A Stochastic Collocation Algorithm with Multifidelity Models
journal, January 2014

  • Narayan, Akil; Gittelson, Claude; Xiu, Dongbin
  • SIAM Journal on Scientific Computing, Vol. 36, Issue 2
  • DOI: 10.1137/130929461

Computational Aspects of Stochastic Collocation with Multifidelity Models
journal, January 2014

  • Zhu, Xueyu; Narayan, Akil; Xiu, Dongbin
  • SIAM/ASA Journal on Uncertainty Quantification, Vol. 2, Issue 1
  • DOI: 10.1137/130949154

Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification
journal, May 2013

  • Bilionis, Ilias; Zabaras, Nicholas; Konomi, Bledar A.
  • Journal of Computational Physics, Vol. 241
  • DOI: 10.1016/j.jcp.2013.01.011

Multi-fidelity Gaussian process regression for prediction of random fields
journal, May 2017


Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets
journal, January 2016

  • Perdikaris, Paris; Venturi, Daniele; Karniadakis, George Em
  • SIAM Journal on Scientific Computing, Vol. 38, Issue 4
  • DOI: 10.1137/15M1055164

Extracting and composing robust features with denoising autoencoders
conference, January 2008

  • Vincent, Pascal; Larochelle, Hugo; Bengio, Yoshua
  • Proceedings of the 25th international conference on Machine learning - ICML '08
  • DOI: 10.1145/1390156.1390294

Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach
journal, August 2016

  • Gómez-Bombarelli, Rafael; Aguilera-Iparraguirre, Jorge; Hirzel, Timothy D.
  • Nature Materials, Vol. 15, Issue 10
  • DOI: 10.1038/nmat4717

Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules
journal, January 2018

  • Gómez-Bombarelli, Rafael; Wei, Jennifer N.; Duvenaud, David
  • ACS Central Science, Vol. 4, Issue 2
  • DOI: 10.1021/acscentsci.7b00572

Bayesian inference with optimal maps
journal, October 2012

  • El Moselhy, Tarek A.; Marzouk, Youssef M.
  • Journal of Computational Physics, Vol. 231, Issue 23
  • DOI: 10.1016/j.jcp.2012.07.022

Variational Inference: A Review for Statisticians
journal, July 2016

  • Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
  • Journal of the American Statistical Association, Vol. 112, Issue 518
  • DOI: 10.1080/01621459.2017.1285773

Graphical Models, Exponential Families, and Variational Inference
journal, January 2007

  • Wainwright, Martin J.; Jordan, Michael I.
  • Foundations and Trends® in Machine Learning, Vol. 1, Issue 1–2
  • DOI: 10.1561/2200000001

Predictive collective variable discovery with deep Bayesian models
journal, January 2019

  • Schöberl, Markus; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios
  • The Journal of Chemical Physics, Vol. 150, Issue 2
  • DOI: 10.1063/1.5058063

A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime
journal, November 2019

  • Grigo, Constantin; Koutsourelakis, Phaedon-Stelios
  • Journal of Computational Physics, Vol. 397
  • DOI: 10.1016/j.jcp.2019.05.053

Fourth-Order Time-Stepping for Stiff PDEs
journal, January 2005

  • Kassam, Aly-Khan; Trefethen, Lloyd N.
  • SIAM Journal on Scientific Computing, Vol. 26, Issue 4
  • DOI: 10.1137/S1064827502410633

Deep learning
journal, May 2015

  • LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
  • Nature, Vol. 521, Issue 7553
  • DOI: 10.1038/nature14539

Understanding deep convolutional networks
journal, April 2016

  • Mallat, Stéphane
  • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 374, Issue 2065
  • DOI: 10.1098/rsta.2015.0203

Active Learning with Statistical Models
journal, January 1996

  • Cohn, D. A.; Ghahramani, Z.; Jordan, M. I.
  • Journal of Artificial Intelligence Research, Vol. 4
  • DOI: 10.1613/jair.295

Taking the Human Out of the Loop: A Review of Bayesian Optimization
journal, January 2016


    Works referencing / citing this record:

    Multi-fidelity optimization for sheet metal forming process
    journal, December 2010

    • Sun, Guangyong; Li, Guangyao; Zhou, Shiwei
    • Structural and Multidisciplinary Optimization, Vol. 44, Issue 1
    • DOI: 10.1007/s00158-010-0596-5

    Towards efficient uncertainty quantification in complex and large-scale biomechanical problems based on a Bayesian multi-fidelity scheme
    journal, September 2014

    • Biehler, Jonas; Gee, Michael W.; Wall, Wolfgang A.
    • Biomechanics and Modeling in Mechanobiology, Vol. 14, Issue 3
    • DOI: 10.1007/s10237-014-0618-0

    Multifidelity importance sampling
    journal, March 2016

    • Peherstorfer, Benjamin; Cui, Tiangang; Marzouk, Youssef
    • Computer Methods in Applied Mechanics and Engineering, Vol. 300
    • DOI: 10.1016/j.cma.2015.12.002

    A two-stage multi-fidelity optimization procedure for honeycomb-type cellular materials
    journal, September 2010


    Bayesian inference with optimal maps
    journal, October 2012

    • El Moselhy, Tarek A.; Marzouk, Youssef M.
    • Journal of Computational Physics, Vol. 231, Issue 23
    • DOI: 10.1016/j.jcp.2012.07.022

    Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification
    journal, May 2013

    • Bilionis, Ilias; Zabaras, Nicholas; Konomi, Bledar A.
    • Journal of Computational Physics, Vol. 241
    • DOI: 10.1016/j.jcp.2013.01.011

    Multi-fidelity Gaussian process regression for prediction of random fields
    journal, May 2017


    Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules
    journal, January 2018

    • Gómez-Bombarelli, Rafael; Wei, Jennifer N.; Duvenaud, David
    • ACS Central Science, Vol. 4, Issue 2
    • DOI: 10.1021/acscentsci.7b00572

    Deep learning
    journal, May 2015

    • LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
    • Nature, Vol. 521, Issue 7553
    • DOI: 10.1038/nature14539

    Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach
    journal, August 2016

    • Gómez-Bombarelli, Rafael; Aguilera-Iparraguirre, Jorge; Hirzel, Timothy D.
    • Nature Materials, Vol. 15, Issue 10
    • DOI: 10.1038/nmat4717

    Predictive collective variable discovery with deep Bayesian models
    journal, January 2019

    • Schöberl, Markus; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios
    • The Journal of Chemical Physics, Vol. 150, Issue 2
    • DOI: 10.1063/1.5058063

    Variational Inference: A Review for Statisticians
    journal, July 2016

    • Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
    • Journal of the American Statistical Association, Vol. 112, Issue 518
    • DOI: 10.1080/01621459.2017.1285773

    DDDAS-based multi-fidelity simulation framework for supply chain systems
    journal, February 2010


    Model inversion via multi-fidelity Bayesian optimization: a new paradigm for parameter estimation in haemodynamics, and beyond
    journal, May 2016

    • Perdikaris, Paris; Karniadakis, George Em
    • Journal of The Royal Society Interface, Vol. 13, Issue 118
    • DOI: 10.1098/rsif.2015.1107

    Multi-fidelity optimization via surrogate modelling
    journal, September 2007

    • Forrester, Alexander I. J.; Sóbester, András; Keane, Andy J.
    • Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 463, Issue 2088
    • DOI: 10.1098/rspa.2007.1900

    Understanding deep convolutional networks
    journal, April 2016

    • Mallat, Stéphane
    • Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 374, Issue 2065
    • DOI: 10.1098/rsta.2015.0203

    A Stochastic Collocation Algorithm with Multifidelity Models
    journal, January 2014

    • Narayan, Akil; Gittelson, Claude; Xiu, Dongbin
    • SIAM Journal on Scientific Computing, Vol. 36, Issue 2
    • DOI: 10.1137/130929461

    Computational Aspects of Stochastic Collocation with Multifidelity Models
    journal, January 2014

    • Zhu, Xueyu; Narayan, Akil; Xiu, Dongbin
    • SIAM/ASA Journal on Uncertainty Quantification, Vol. 2, Issue 1
    • DOI: 10.1137/130949154

    Extracting and composing robust features with denoising autoencoders
    conference, January 2008

    • Vincent, Pascal; Larochelle, Hugo; Bengio, Yoshua
    • Proceedings of the 25th international conference on Machine learning - ICML '08
    • DOI: 10.1145/1390156.1390294

    Graphical Models, Exponential Families, and Variational Inference
    journal, January 2007

    • Wainwright, Martin J.; Jordan, Michael I.
    • Foundations and Trends® in Machine Learning, Vol. 1, Issue 1–2
    • DOI: 10.1561/2200000001

    Active Learning with Statistical Models
    journal, January 1996

    • Cohn, D. A.; Ghahramani, Z.; Jordan, M. I.
    • Journal of Artificial Intelligence Research, Vol. 4
    • DOI: 10.1613/jair.295

    Surrogate-Based Optimization Using Multifidelity Models with Variable Parameterization and Corrected Space Mapping
    journal, November 2008

    • Robinson, T. D.; Eldred, M. S.; Willcox, K. E.
    • AIAA Journal, Vol. 46, Issue 11
    • DOI: 10.2514/1.36043

    Approximation and Model Management in Aerodynamic Optimization with Variable-Fidelity Models
    journal, November 2001

    • Alexandrov, Natalia M.; Lewis, Robert Michael; Gumbert, Clyde R.
    • Journal of Aircraft, Vol. 38, Issue 6
    • DOI: 10.2514/2.2877

    Comparison of Non-Intrusive Polynomial Chaos and Stochastic Collocation Methods for Uncertainty Quantification
    conference, June 2012

    • Eldred, Michael; Burkardt, John
    • 47th AIAA Aerospace Sciences Meeting including The New Horizons Forum and Aerospace Exposition
    • DOI: 10.2514/6.2009-976

    Multi-Fidelity Uncertainty Quantification: Application to a Vertical Axis Wind Turbine Under an Extreme Gust
    conference, June 2014

    • Padron, Andres S.; Alonso, Juan J.; Palacios, Francisco
    • 15th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference
    • DOI: 10.2514/6.2014-3013