Bayesian differential programming for robust systems identification under uncertainty

Yang, Yibo; Aziz Bhouri, Mohamed; Perdikaris, Paris

doi:10.1098/rspa.2020.0290

Bayesian differential programming for robust systems identification under uncertainty

Journal Article · Sun Nov 01 04:00:00 EST 2020 · Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences

DOI:https://doi.org/10.1098/rspa.2020.0290· OSTI ID:1852841

Yang, Yibo ^[1]; Aziz Bhouri, Mohamed ^[2]; ^[2]

Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, Philadelphia, PA 19104, USA; OSTI
Department of Mechanical Engineering and Applied Mechanics, University of Pennsylvania, Philadelphia, PA 19104, USA

This paper presents a machine learning framework for Bayesian systems identification from noisy, sparse and irregular observations of nonlinear dynamical systems. The proposed method takes advantage of recent developments in differentiable programming to propagate gradient information through ordinary differential equation solvers and perform Bayesian inference with respect to unknown model parameters using Hamiltonian Monte Carlo sampling. This allows an efficient inference of the posterior distributions over plausible models with quantified uncertainty, while the use of sparsity-promoting priors enables the discovery of interpretable and parsimonious representations for the underlying latent dynamics. A series of numerical studies is presented to demonstrate the effectiveness of the proposed methods, including nonlinear oscillators, predator–prey systems and examples from systems biology. Taken together, our findings put forth a flexible and robust workflow for data-driven model discovery under uncertainty. All codes and data accompanying this article are available athttps://bit.ly/34FOJMj.

Research Organization:: Univ. of Pennsylvania, Philadelphia, PA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

DOE Contract Number:: SC0019116

OSTI ID:: 1852841

Journal Information:: Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences, Journal Name: Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences Journal Issue: 2243 Vol. 476; ISSN 1364-5021

Publisher:: The Royal Society Publishing

Country of Publication:: United States

Language:: English

References (45)

Nonparametric inference of interaction laws in systems of agents from trajectory data Lu, Fei; Zhong, Ming; Tang, Sui Proceedings of the National Academy of Sciences, Vol. 116, Issue 29 https://doi.org/10.1073/pnas.1822012116	journal	June 2019
Large-Scale Machine Learning with Stochastic Gradient Descent Bottou, Léon Proceedings of COMPSTAT'2010 https://doi.org/10.1007/978-3-7908-2604-3_16	book	January 2010
Bayesian treatment of the independent student-t linear model Geweke, J. Journal of Applied Econometrics, Vol. 8, Issue S1 https://doi.org/10.1002/jae.3950080504	journal	December 1993
Lagrangian coherent structures from approximate velocity data Haller, G. Physics of Fluids, Vol. 14, Issue 6 https://doi.org/10.1063/1.1477449	journal	June 2002
Data-driven discovery of coordinates and governing equations Champion, Kathleen; Lusch, Bethany; Kutz, J. Nathan Proceedings of the National Academy of Sciences, Vol. 116, Issue 45 https://doi.org/10.1073/pnas.1906995116	journal	October 2019
Non‐linear state space modelling of fisheries biomass dynamics by using Metropolis‐Hastings within‐Gibbs sampling Millar, Russell B.; Meyer, Renate Journal of the Royal Statistical Society: Series C (Applied Statistics), Vol. 49, Issue 3 https://doi.org/10.1111/1467-9876.00195	journal	January 2000
Data-driven closures for stochastic dynamical systems Brennan, Catherine; Venturi, Daniele Journal of Computational Physics, Vol. 372 https://doi.org/10.1016/j.jcp.2018.06.038	journal	November 2018
Probabilistic machine learning and artificial intelligence Ghahramani, Zoubin Nature, Vol. 521, Issue 7553 https://doi.org/10.1038/nature14541	journal	May 2015
Temperature dependency and temperature compensation in a model of yeast glycolytic oscillations Ruoff, Peter; Christensen, Melinda K.; Wolf, Jana Biophysical Chemistry, Vol. 106, Issue 2 https://doi.org/10.1016/S0301-4622(03)00191-1	journal	November 2003
The Elements of Statistical Learning Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome Springer Series in Statistics https://doi.org/10.1007/978-0-387-84858-7	book	January 2009
Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks Kissas, Georgios; Yang, Yibo; Hwuang, Eileen Computer Methods in Applied Mechanics and Engineering, Vol. 358 https://doi.org/10.1016/j.cma.2019.112623	journal	January 2020
Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems Yang, Yibo; Perdikaris, Paris Computational Mechanics, Vol. 64, Issue 2 https://doi.org/10.1007/s00466-019-01718-y	journal	May 2019
A First Course in the Numerical Analysis of Differential Equations Iserles, Arieh Cambridge University Press https://doi.org/10.1017/CBO9780511995569	book	January 2008
MCMC Using Hamiltonian Dynamics Neal, Radford; Brooks, Steve; Gelman, Andrew Handbook of Markov Chain Monte Carlo https://doi.org/10.1201/b10905-6	book	May 2011
Deep learning for universal linear embeddings of nonlinear dynamics Lusch, Bethany; Kutz, J. Nathan; Brunton, Steven L. Nature Communications, Vol. 9, Issue 1 https://doi.org/10.1038/s41467-018-07210-0	journal	November 2018
Dynamics of open chemical systems and the algebraic structure of the underlying reaction network Feinberg, Martin; Horn, Friedrich J. M. Chemical Engineering Science, Vol. 29, Issue 3 https://doi.org/10.1016/0009-2509(74)80195-8	journal	March 1974
Control of systems integrating logic, dynamics, and constraints Bemporad, Alberto; Morari, Manfred Automatica, Vol. 35, Issue 3 https://doi.org/10.1016/S0005-1098(98)00178-2	journal	March 1999
Non-intrusive reduced order modeling of unsteady flows using artificial neural networks with application to a combustion problem Wang, Qian; Hesthaven, Jan S.; Ray, Deep Journal of Computational Physics, Vol. 384 https://doi.org/10.1016/j.jcp.2019.01.031	journal	May 2019
Simple chemical reaction systems with limit cycle behaviour Schnakenberg, J. Journal of Theoretical Biology, Vol. 81, Issue 3 https://doi.org/10.1016/0022-5193(79)90042-0	journal	December 1979
A Nonlinear Dynamical Perspective on Climate Prediction Palmer, T. N. Journal of Climate, Vol. 12, Issue 2 https://doi.org/10.1175/1520-0442(1999)012<0575:ANDPOC>2.0.CO;2	journal	February 1999
The Assessment of Prior Distributions in Bayesian Analysis Winkler, Robert L. Journal of the American Statistical Association, Vol. 62, Issue 319 https://doi.org/10.1080/01621459.1967.10500894	journal	September 1967
Systems biology informed deep learning for inferring parameters and hidden dynamics Yazdani, Alireza; Lu, Lu; Raissi, Maziar PLOS Computational Biology, Vol. 16, Issue 11 https://doi.org/10.1371/journal.pcbi.1007575	journal	November 2020
Bayesian Data Analysis Gelman, Andrew; Carlin, John B.; Stern, Hal S. https://doi.org/10.1201/b16018	book	November 2013
Automated adaptive inference of phenomenological dynamical models Daniels, Bryan C.; Nemenman, Ilya Nature Communications, Vol. 6, Issue 1 https://doi.org/10.1038/ncomms9133	journal	August 2015
Adversarial uncertainty quantification in physics-informed neural networks Yang, Yibo; Perdikaris, Paris Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.027	journal	October 2019
Data-driven discovery of partial differential equations Rudy, Samuel H.; Brunton, Steven L.; Proctor, Joshua L. Science Advances, Vol. 3, Issue 4 https://doi.org/10.1126/sciadv.1602614	journal	April 2017
Robust Bayesian analysis: sensitivity to the prior Berger, James O. Journal of Statistical Planning and Inference, Vol. 25, Issue 3 https://doi.org/10.1016/0378-3758(90)90079-A	journal	July 1990
Universal Differential Equations for Scientific Machine Learning Rackauckas, Christopher; Ma, Yingbo; Martensen, Julius https://doi.org/10.21203/rs.3.rs-55125/v1	preprint	August 2020
Automated refinement and inference of analytical models for metabolic networks Schmidt, Michael D.; Vallabhajosyula, Ravishankar R.; Jenkins, Jerry W. Physical Biology, Vol. 8, Issue 5 https://doi.org/10.1088/1478-3975/8/5/055011	journal	August 2011
Bayesian Regularization and Pruning Using a Laplace Prior Williams, Peter M. Neural Computation, Vol. 7, Issue 1 https://doi.org/10.1162/neco.1995.7.1.117	journal	January 1995
Discovering governing equations from data by sparse identification of nonlinear dynamical systems Brunton, Steven L.; Proctor, Joshua L.; Kutz, J. Nathan Proceedings of the National Academy of Sciences, Vol. 113, Issue 15 https://doi.org/10.1073/pnas.1517384113	journal	March 2016
Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations Raissi, Maziar; Yazdani, Alireza; Karniadakis, George Em Science, Vol. 367, Issue 6481 https://doi.org/10.1126/science.aaw4741	journal	January 2020
Data driven governing equations approximation using deep neural networks Qin, Tong; Wu, Kailiang; Xiu, Dongbin Journal of Computational Physics, Vol. 395 https://doi.org/10.1016/j.jcp.2019.06.042	journal	October 2019
Physics-Informed Generative Adversarial Networks for Stochastic Differential Equations Yang, Liu; Zhang, Dongkun; Karniadakis, George Em SIAM Journal on Scientific Computing, Vol. 42, Issue 1 https://doi.org/10.1137/18M1225409	journal	January 2020
Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios Journal of Computational Physics, Vol. 394 https://doi.org/10.1016/j.jcp.2019.05.024	journal	October 2019
Inferring solutions of differential equations using noisy multi-fidelity data Raissi, Maziar; Perdikaris, Paris; Karniadakis, George Em Journal of Computational Physics, Vol. 335 https://doi.org/10.1016/j.jcp.2017.01.060	journal	April 2017
Physics-Informed Neural Networks for Cardiac Activation Mapping Sahli Costabal, Francisco; Yang, Yibo; Perdikaris, Paris Frontiers in Physics, Vol. 8 https://doi.org/10.3389/fphy.2020.00042	journal	February 2020
Solving high-dimensional partial differential equations using deep learning Han, Jiequn; Jentzen, Arnulf; E., Weinan Proceedings of the National Academy of Sciences, Vol. 115, Issue 34 https://doi.org/10.1073/pnas.1718942115	journal	August 2018
Markov Chain Monte Carlo Convergence Diagnostics: A Comparative Review Cowles, Mary Kathryn; Carlin, Bradley P. Journal of the American Statistical Association, Vol. 91, Issue 434 https://doi.org/10.1080/01621459.1996.10476956	journal	June 1996
The geometric foundations of Hamiltonian Monte Carlo Betancourt, Michael; Byrne, Simon; Livingstone, Sam Bernoulli, Vol. 23, Issue 4A https://doi.org/10.3150/16-BEJ810	journal	November 2017
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations Raissi, M.; Perdikaris, P.; Karniadakis, G. E. Journal of Computational Physics, Vol. 378 https://doi.org/10.1016/j.jcp.2018.10.045	journal	February 2019
Compressed sensing Donoho, D. L. IEEE Transactions on Information Theory, Vol. 52, Issue 4 https://doi.org/10.1109/TIT.2006.871582	journal	April 2006
Principles of Computerized Tomographic Imaging Kak, Avinash C.; Slaney, Malcolm; Wang, Ge Medical Physics, Vol. 29, Issue 1 https://doi.org/10.1118/1.1455742	journal	January 2002
Crisis of the chaotic attractor of a climate model: a transfer operator approach Tantet, Alexis; Lucarini, Valerio; Lunkeit, Frank Nonlinearity, Vol. 31, Issue 5 https://doi.org/10.1088/1361-6544/aaaf42	journal	April 2018
Hidden physics models: Machine learning of nonlinear partial differential equations Raissi, Maziar; Karniadakis, George Em Journal of Computational Physics, Vol. 357 https://doi.org/10.1016/j.jcp.2017.11.039	journal	March 2018

Similar Records

Gaussian processes meet NeuralODEs: a Bayesian framework for learning the dynamics of partially observed systems from scarce and noisy data

Journal Article · Mon Jun 20 00:00:00 EDT 2022 · Philosophical Transactions of the Royal Society. A, Mathematical, Physical and Engineering Sciences · OSTI ID:1979597

Sparsifying priors for Bayesian uncertainty quantification in model discovery

Journal Article · Tue Feb 22 19:00:00 EST 2022 · Royal Society Open Science · OSTI ID:1861893

B-PINNs: Bayesian Physics-informal Neural Networks for Forward and Inverse PDE Problems with Noisy Data

Journal Article · Thu Jan 14 23:00:00 EST 2021 · Journal of Computational Physics · OSTI ID:1763962

Related Subjects

Science & Technology - Other Topics

Bayesian differential programming for robust systems identification under uncertainty

Citation Formats

References (45)

Similar Records

Related Subjects