Adversarial uncertainty quantification in physics-informed neural networks

Yang, Yibo; Perdikaris, Paris

doi:10.1016/j.jcp.2019.05.027

Title: Adversarial uncertainty quantification in physics-informed neural networks

Abstract

Here, we introduce a deep learning framework for quantifying and propagating uncertainty in systems governed by non-linear differential equations using physics-informed neural networks. Specifically, we employ latent variable models to construct probabilistic representations for the system states, and put forth an adversarial inference procedure for training them on data, while constraining their predictions to satisfy given physical laws expressed by partial differential equations. Such physics-informed constraints provide a regularization mechanism for effectively training deep generative models as surrogates of physical systems in which the cost of data acquisition is high, and training data-sets are typically small. This provides a flexible framework for characterizing uncertainty in the outputs of physical systems due to randomness in their inputs or noise in their observations that entirely bypasses the need for repeatedly sampling expensive experiments or numerical simulators. Moreover, we demonstrate the effectiveness of our approach through a series of examples involving uncertainty propagation in non-linear conservation laws, and the discovery of constitutive laws for flow through porous media directly from noisy data.

Authors:

Yang, Yibo ^[1];

^[1]

Univ. of Pennsylvania, Philadelphia, PA (United States)

Publication Date:: Fri May 24 00:00:00 EDT 2019

Research Org.:: Univ. of Pennsylvania, Philadelphia, PA (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1595802

Alternate Identifier(s):: OSTI ID: 1691928

Grant/Contract Number:: SC0019116; HR00111890034

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Computational Physics

Additional Journal Information:: Journal Volume: 394; Journal Issue: C; Related Information: https://github.com/PredictiveIntelligenceLab/UQPINNs; Journal ID: ISSN 0021-9991

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Variational inference; Generative adversarial networks; Probabilistic deep learning; Probabilistic scientific computing; Data-driven modeling

Citation Formats


                    Yang, Yibo, and Perdikaris, Paris. Adversarial uncertainty quantification in physics-informed neural networks.  United States: N. p., 2019. 
Web.  doi:10.1016/j.jcp.2019.05.027.

Copy to clipboard


                    Yang, Yibo, & Perdikaris, Paris. Adversarial uncertainty quantification in physics-informed neural networks.  United States.  https://doi.org/10.1016/j.jcp.2019.05.027

Copy to clipboard


                    Yang, Yibo, and Perdikaris, Paris. Fri .  
"Adversarial uncertainty quantification in physics-informed neural networks".  United States.  https://doi.org/10.1016/j.jcp.2019.05.027.  https://www.osti.gov/servlets/purl/1595802.

Copy to clipboard


                    
@article{osti_1595802,

  title        = {Adversarial uncertainty quantification in physics-informed neural networks},

  author       = {Yang, Yibo and Perdikaris, Paris},

  abstractNote = {Here, we introduce a deep learning framework for quantifying and propagating uncertainty in systems governed by non-linear differential equations using physics-informed neural networks. Specifically, we employ latent variable models to construct probabilistic representations for the system states, and put forth an adversarial inference procedure for training them on data, while constraining their predictions to satisfy given physical laws expressed by partial differential equations. Such physics-informed constraints provide a regularization mechanism for effectively training deep generative models as surrogates of physical systems in which the cost of data acquisition is high, and training data-sets are typically small. This provides a flexible framework for characterizing uncertainty in the outputs of physical systems due to randomness in their inputs or noise in their observations that entirely bypasses the need for repeatedly sampling expensive experiments or numerical simulators. Moreover, we demonstrate the effectiveness of our approach through a series of examples involving uncertainty propagation in non-linear conservation laws, and the discovery of constitutive laws for flow through porous media directly from noisy data.},

  doi          = {10.1016/j.jcp.2019.05.027},

  journal      = {Journal of Computational Physics},

  number       = C,

  volume       = 394,

  place        = {United States},

  year         = {Fri May 24 00:00:00 EDT 2019},

  month        = {Fri May 24 00:00:00 EDT 2019}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.jcp.2019.05.027

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 130 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Deep learning
journal, May 2015

LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey
Nature, Vol. 521, Issue 7553
DOI: 10.1038/nature14539

Human-level concept learning through probabilistic program induction
journal, December 2015

Lake, B. M.; Salakhutdinov, R.; Tenenbaum, J. B.
Science, Vol. 350, Issue 6266
DOI: 10.1126/science.aab3050

Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
journal, July 2015

Alipanahi, Babak; Delong, Andrew; Weirauch, Matthew T.
Nature Biotechnology, Vol. 33, Issue 8
DOI: 10.1038/nbt.3300

Hidden physics models: Machine learning of nonlinear partial differential equations
journal, March 2018

Raissi, Maziar; Karniadakis, George Em
Journal of Computational Physics, Vol. 357
DOI: 10.1016/j.jcp.2017.11.039

A hybrid neural network-first principles approach to process modeling
journal, October 1992

Psichogios, Dimitris C.; Ungar, Lyle H.
AIChE Journal, Vol. 38, Issue 10
DOI: 10.1002/aic.690381003

Artificial neural networks for solving ordinary and partial differential equations
journal, January 1998

Lagaris, I. E.; Likas, A.; Fotiadis, D. I.
IEEE Transactions on Neural Networks, Vol. 9, Issue 5
DOI: 10.1109/72.712178

Multiscale modeling and simulation of brain blood flow
journal, February 2016

Perdikaris, Paris; Grinberg, Leopold; Karniadakis, George Em
Physics of Fluids, Vol. 28, Issue 2
DOI: 10.1063/1.4941315

Polynomial Chaos in Stochastic Finite Elements
journal, March 1990

Ghanem, Roger; Spanos, P. D.
Journal of Applied Mechanics, Vol. 57, Issue 1
DOI: 10.1115/1.2888303

The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations
journal, January 2002

Xiu, Dongbin; Karniadakis, George Em
SIAM Journal on Scientific Computing, Vol. 24, Issue 2
DOI: 10.1137/S1064827501387826

Uncertainty Quantification and Polynomial Chaos Techniques in Computational Fluid Dynamics
journal, January 2009

Najm, Habib N.
Annual Review of Fluid Mechanics, Vol. 41, Issue 1
DOI: 10.1146/annurev.fluid.010908.165248

Multi-level Monte Carlo Finite Element method for elliptic PDEs with stochastic coefficients
journal, April 2011

Barth, Andrea; Schwab, Christoph; Zollinger, Nathaniel
Numerische Mathematik, Vol. 119, Issue 1
DOI: 10.1007/s00211-011-0377-0

Optimal Model Management for Multifidelity Monte Carlo Estimation
journal, January 2016

Peherstorfer, Benjamin; Willcox, Karen; Gunzburger, Max
SIAM Journal on Scientific Computing, Vol. 38, Issue 5
DOI: 10.1137/15M1046472

The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows
journal, January 1993

Berkooz, G.; Holmes, P.; Lumley, J. L.
Annual Review of Fluid Mechanics, Vol. 25, Issue 1
DOI: 10.1146/annurev.fl.25.010193.002543

Multi-output local Gaussian process regression: Applications to uncertainty quantification
journal, July 2012

Bilionis, Ilias; Zabaras, Nicholas
Journal of Computational Physics, Vol. 231, Issue 17
DOI: 10.1016/j.jcp.2012.04.047

Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification
journal, May 2013

Bilionis, Ilias; Zabaras, Nicholas; Konomi, Bledar A.
Journal of Computational Physics, Vol. 241
DOI: 10.1016/j.jcp.2013.01.011

Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets
journal, January 2016

Perdikaris, Paris; Venturi, Daniele; Karniadakis, George Em
SIAM Journal on Scientific Computing, Vol. 38, Issue 4
DOI: 10.1137/15M1055164

Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
journal, February 2019

Raissi, M.; Perdikaris, P.; Karniadakis, G. E.
Journal of Computational Physics, Vol. 378
DOI: 10.1016/j.jcp.2018.10.045

Variational Inference: A Review for Statisticians
journal, July 2016

Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
Journal of the American Statistical Association, Vol. 112, Issue 518
DOI: 10.1080/01621459.2017.1285773

Inferring solutions of differential equations using noisy multi-fidelity data
journal, April 2017

Raissi, Maziar; Perdikaris, Paris; Karniadakis, George Em
Journal of Computational Physics, Vol. 335
DOI: 10.1016/j.jcp.2017.01.060

Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations
journal, January 2018

Raissi, Maziar; Perdikaris, Paris; Karniadakis, George Em
SIAM Journal on Scientific Computing, Vol. 40, Issue 1
DOI: 10.1137/17M1120762

The partial differential equation ut + uux = μxx
journal, September 1950

Hopf, Eberhard
Communications on Pure and Applied Mathematics, Vol. 3, Issue 3
DOI: 10.1002/cpa.3160030302

On the limited memory BFGS method for large scale optimization
journal, August 1989

Liu, Dong C.; Nocedal, Jorge
Mathematical Programming, Vol. 45, Issue 1-3
DOI: 10.1007/BF01589116

Modeling fluid flow and transport in variably saturated porous media with the STOMP simulator. 1. Nonvolatile three-phase model description
journal, January 1995

White, M. D.; Oostrom, M.; Lenhard, R. J.
Advances in Water Resources, Vol. 18, Issue 6
DOI: 10.1016/0309-1708(95)00018-E

A Closed-form Equation for Predicting the Hydraulic Conductivity of Unsaturated Soils1
journal, January 1980

van Genuchten, M. Th.
Soil Science Society of America Journal, Vol. 44, Issue 5
DOI: 10.2136/sssaj1980.03615995004400050002x

The partial differential equation ut + uux = μxx
journal, September 1950

Hopf, Eberhard
Communications on Pure and Applied Mathematics, Vol. 3, Issue 3
DOI: 10.1002/cpa.3160030302

Multiscale modeling and simulation of brain blood flow
journal, February 2016

Perdikaris, Paris; Grinberg, Leopold; Karniadakis, George Em
Physics of Fluids, Vol. 28, Issue 2
DOI: 10.1063/1.4941315

Variational Inference: A Review for Statisticians
journal, July 2016

Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D.
Journal of the American Statistical Association, Vol. 112, Issue 518
DOI: 10.1080/01621459.2017.1285773

Automatic differentiation in machine learning: a survey
text, January 2015

Baydin, Atilim Gunes; Pearlmutter, Barak A.; Radul, Alexey Andreyevich
arXiv
DOI: 10.48550/arxiv.1502.05767

Probabilistic Numerical Methods for PDE-constrained Bayesian Inverse Problems
text, January 2017

Cockayne, Jon; Oates, Chris; Sullivan, Tim
arXiv
DOI: 10.48550/arxiv.1701.04006

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
preprint, January 2017

Mescheder, Lars; Nowozin, Sebastian; Geiger, Andreas
arXiv
DOI: 10.48550/arxiv.1701.04722

Optimal fidelity multi-level Monte Carlo for quantification of uncertainty in simulations of cloud cavitation collapse
preprint, January 2017

Šukys, Jonas; Rasthofer, Ursula; Wermelinger, Fabian
arXiv
DOI: 10.48550/arxiv.1705.04374

Works referencing / citing this record:

Multiscale Modeling Meets Machine Learning: What Can We Learn?
journal, February 2020

Peng, Grace C. Y.; Alber, Mark; Buganza Tepole, Adrian
Archives of Computational Methods in Engineering
DOI: 10.1007/s11831-020-09405-5

Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences
journal, November 2019

Alber, Mark; Buganza Tepole, Adrian; Cannon, William R.
npj Digital Medicine, Vol. 2, Issue 1
DOI: 10.1038/s41746-019-0193-y

Multiscale modeling meets machine learning: What can we learn?
preprint, January 2019

Peng, Grace C. Y.; Alber, Mark; Tepole, Adrian Buganza
arXiv
DOI: 10.48550/arxiv.1911.11958

Similar Records in DOE PAGES and OSTI.GOV collections:

Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks

Journal Article Kissas, Georgios ; Yang, Yibo ; Hwuang, Eileen ; ... - Computer Methods in Applied Mechanics and Engineering

Advances in computational science offer a principled pipeline for predictive modeling of cardiovascular flows and aspire to provide a valuable tool for monitoring, diagnostics and surgical planning. Such models can be nowadays deployed on large patient-specific topologies of systemic arterial networks and return detailed predictions on flow patterns, wall shear stresses, and pulse wave propagation. However, their success heavily relies on tedious pre-processing and calibration procedures that typically induce a significant computational cost, thus hampering their clinical applicability. Here, we put forth a machine learning framework that enables the seamless synthesis of non-invasive in-vivo measurement techniques and computational flow dynamicsmore »« less
Cited by 196
https://doi.org/10.1016/j.cma.2019.112623

Full Text Available
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems

Journal Article Yang, Yibo ; Perdikaris, Paris - Computational Mechanics

Here, we present a probabilistic deep learning methodology that enables the construction of predictive data-driven surrogates for stochastic systems. Leveraging recent advances in variational inference with implicit distributions, we put forth a statistical inference framework that enables the end-to-end training of surrogate models on paired input–output observations that may be stochastic in nature, originate from different information sources of variable fidelity, or be corrupted by complex noise processes. The resulting surrogates can accommodate high-dimensional inputs and outputs and are able to return predictions with quantified uncertainty. Furthermore, the effectiveness our approach is demonstrated through a series of canonical studies, includingmore »« less
Cited by 21
https://doi.org/10.1007/s00466-019-01718-y

Full Text Available
Learning functional priors and posteriors from data and physics

Journal Article Meng, Xuhui ; Yang, Liu ; Mao, Zhiping ; ... - Journal of Computational Physics

In this work, we develop a new Bayesian framework based on deep neural networks to be able to extrapolate in space-time using historical data and to quantify uncertainties arising from both noisy and gappy data in physical problems. Specifically, the proposed approach has two stages: (1) prior learning and (2) posterior estimation. At the first stage, we employ the physics-informed Generative Adversarial Networks (PI-GAN) to learn a functional prior either from a prescribed function distribution, e.g., Gaussian process, or from historical data and physics. At the second stage, we employ the Hamiltonian Monte Carlo (HMC) method to estimate the posteriormore »« less
https://doi.org/10.1016/j.jcp.2022.111073

Full Text Available
Bayesian differential programming for robust systems identification under uncertainty

Journal Article Yang, Yibo ; Aziz Bhouri, Mohamed ; Perdikaris, Paris - Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences

This paper presents a machine learning framework for Bayesian systems identification from noisy, sparse and irregular observations of nonlinear dynamical systems. The proposed method takes advantage of recent developments in differentiable programming to propagate gradient information through ordinary differential equation solvers and perform Bayesian inference with respect to unknown model parameters using Hamiltonian Monte Carlo sampling. This allows an efficient inference of the posterior distributions over plausible models with quantified uncertainty, while the use of sparsity-promoting priors enables the discovery of interpretable and parsimonious representations for the underlying latent dynamics. A series of numerical studies is presented to demonstrate themore »« less
https://doi.org/10.1098/rspa.2020.0290
Multi-fidelity information fusion with concatenated neural networks

Journal Article Pawar, Suraj ; San, Omer ; Vedula, Prakash ; ... - Scientific Reports

Recently, computational modeling has shifted towards the use of statistical inference, deep learning, and other data-driven modeling frameworks. Although this shift in modeling holds promise in many applications like design optimization and real-time control by lowering the computational burden, training deep learning models needs a huge amount of data. This big data is not always available for scientific problems and leads to poorly generalizable data-driven models. This gap can be furnished by leveraging information from physics-based models. Exploiting prior knowledge about the problem at hand, this study puts forth a physics-guided machine learning (PGML) approach to build more tailored, effective,more »« less
https://doi.org/10.1038/s41598-022-09938-8

Full Text Available

Similar Records

Title: Adversarial uncertainty quantification in physics-informed neural networks

Abstract

Citation Formats

Deep learning journal, May 2015

Human-level concept learning through probabilistic program induction journal, December 2015

Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning journal, July 2015

Hidden physics models: Machine learning of nonlinear partial differential equations journal, March 2018

A hybrid neural network-first principles approach to process modeling journal, October 1992

Artificial neural networks for solving ordinary and partial differential equations journal, January 1998

Multiscale modeling and simulation of brain blood flow journal, February 2016

Polynomial Chaos in Stochastic Finite Elements journal, March 1990

The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations journal, January 2002

Uncertainty Quantification and Polynomial Chaos Techniques in Computational Fluid Dynamics journal, January 2009

Multi-level Monte Carlo Finite Element method for elliptic PDEs with stochastic coefficients journal, April 2011

Optimal Model Management for Multifidelity Monte Carlo Estimation journal, January 2016

The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows journal, January 1993

Multi-output local Gaussian process regression: Applications to uncertainty quantification journal, July 2012

Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification journal, May 2013

Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets journal, January 2016

Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations journal, February 2019

Variational Inference: A Review for Statisticians journal, July 2016

Inferring solutions of differential equations using noisy multi-fidelity data journal, April 2017

Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations journal, January 2018

The partial differential equation ut + uux = μxx journal, September 1950

On the limited memory BFGS method for large scale optimization journal, August 1989

Modeling fluid flow and transport in variably saturated porous media with the STOMP simulator. 1. Nonvolatile three-phase model description journal, January 1995

A Closed-form Equation for Predicting the Hydraulic Conductivity of Unsaturated Soils1 journal, January 1980

The partial differential equation ut + uux = μxx journal, September 1950

Multiscale modeling and simulation of brain blood flow journal, February 2016

Variational Inference: A Review for Statisticians journal, July 2016

Automatic differentiation in machine learning: a survey text, January 2015

Probabilistic Numerical Methods for PDE-constrained Bayesian Inverse Problems text, January 2017

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks preprint, January 2017

Optimal fidelity multi-level Monte Carlo for quantification of uncertainty in simulations of cloud cavitation collapse preprint, January 2017

Multiscale Modeling Meets Machine Learning: What Can We Learn? journal, February 2020

Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences journal, November 2019

Multiscale modeling meets machine learning: What can we learn? preprint, January 2019

Deep learning
journal, May 2015

Human-level concept learning through probabilistic program induction
journal, December 2015

Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
journal, July 2015

Hidden physics models: Machine learning of nonlinear partial differential equations
journal, March 2018

A hybrid neural network-first principles approach to process modeling
journal, October 1992

Artificial neural networks for solving ordinary and partial differential equations
journal, January 1998

Multiscale modeling and simulation of brain blood flow
journal, February 2016

Polynomial Chaos in Stochastic Finite Elements
journal, March 1990

The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations
journal, January 2002

Uncertainty Quantification and Polynomial Chaos Techniques in Computational Fluid Dynamics
journal, January 2009

Multi-level Monte Carlo Finite Element method for elliptic PDEs with stochastic coefficients
journal, April 2011

Optimal Model Management for Multifidelity Monte Carlo Estimation
journal, January 2016

The Proper Orthogonal Decomposition in the Analysis of Turbulent Flows
journal, January 1993

Multi-output local Gaussian process regression: Applications to uncertainty quantification
journal, July 2012

Multi-output separable Gaussian process: Towards an efficient, fully Bayesian paradigm for uncertainty quantification
journal, May 2013

Multifidelity Information Fusion Algorithms for High-Dimensional Systems and Massive Data sets
journal, January 2016

Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
journal, February 2019

Variational Inference: A Review for Statisticians
journal, July 2016

Inferring solutions of differential equations using noisy multi-fidelity data
journal, April 2017

Numerical Gaussian Processes for Time-Dependent and Nonlinear Partial Differential Equations
journal, January 2018

The partial differential equation ut + uux = μxx
journal, September 1950

On the limited memory BFGS method for large scale optimization
journal, August 1989

Modeling fluid flow and transport in variably saturated porous media with the STOMP simulator. 1. Nonvolatile three-phase model description
journal, January 1995

A Closed-form Equation for Predicting the Hydraulic Conductivity of Unsaturated Soils1
journal, January 1980

The partial differential equation ut + uux = μxx
journal, September 1950

Multiscale modeling and simulation of brain blood flow
journal, February 2016

Variational Inference: A Review for Statisticians
journal, July 2016

Automatic differentiation in machine learning: a survey
text, January 2015

Probabilistic Numerical Methods for PDE-constrained Bayesian Inverse Problems
text, January 2017

Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
preprint, January 2017

Optimal fidelity multi-level Monte Carlo for quantification of uncertainty in simulations of cloud cavitation collapse
preprint, January 2017

Multiscale Modeling Meets Machine Learning: What Can We Learn?
journal, February 2020

Integrating machine learning and multiscale modeling—perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences
journal, November 2019

Multiscale modeling meets machine learning: What can we learn?
preprint, January 2019