hIPPYlib-MUQ: A Bayesian Inference Software Framework for Integration of Data with Complex Predictive Models under Uncertainty

Kim, Ki-Tae; Villa, Umberto; Parno, Matthew D.; Marzouk, Youssef; Ghattas, Omar; Petra, Noemi

doi:10.1145/3580278

hIPPYlib-MUQ: A Bayesian Inference Software Framework for Integration of Data with Complex Predictive Models under Uncertainty

Journal Article · Thu Jun 15 00:00:00 EDT 2023 · ACM Transactions on Mathematical Software

DOI:https://doi.org/10.1145/3580278· OSTI ID:2421116

^[1]; ^[2]; ^[3]; ^[4]; ^[2]; ^[5]

Univ. of California, Merced, CA (United States); OSTI
Univ. of Texas, Austin, TX (United States)
Dartmouth College, Hanover, NH (United States)
Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)
Univ. of California, Merced, CA (United States)

Bayesian inference provides a systematic framework for integration of data with mathematical models to quantify the uncertainty in the solution of the inverse problem. However, the solution of Bayesian inverse problems governed by complex forward models described by partial differential equations (PDEs) remains prohibitive with black-box Markov chain Monte Carlo (MCMC) methods. We present hIPPYlib-MUQ, an extensible and scalable software framework that contains implementations of state-of-the art algorithms aimed to overcome the challenges of high-dimensional, PDE-constrained Bayesian inverse problems. These algorithms accelerate MCMC sampling by exploiting the geometry and intrinsic low-dimensionality of parameter space via derivative information and low rank approximation. The software integrates two complementary open-source software packages, hIPPYlib and MUQ. hIPPYlib solves PDE-constrained inverse problems using automatically-generated adjoint-based derivatives, but it lacks full Bayesian capabilities. MUQ provides a spectrum of powerful Bayesian inversion models and algorithms, but expects forward models to come equipped with gradients and Hessians to permit large-scale solution. By combining these two complementary libraries, we created a robust, scalable, and efficient software framework that realizes the benefits of each and allows us to tackle complex large-scale Bayesian inverse problems across a broad spectrum of scientific and engineering disciplines. To illustrate the capabilities of hIPPYlib-MUQ, we present a comparison of a number of MCMC methods available in the integrated software on several high-dimensional Bayesian inverse problems. These include problems characterized by both linear and nonlinear PDEs, various noise models, and different parameter dimensions. The results demonstrate that large (~ 50×) speedups over conventional black box and gradient-based MCMC algorithms can be obtained by exploiting Hessian information (from the log-posterior), underscoring the power of the integrated hIPPYlib-MUQ framework.

View Accepted Manuscript (DOE)

Research Organization:: Univ. of Texas, Austin, TX (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); US Department of the Navy, Office of Naval Research (ONR); National Science Foundation (NSF)

Grant/Contract Number:: SC0019303; SC0021239

OSTI ID:: 2421116

Journal Information:: ACM Transactions on Mathematical Software, Journal Name: ACM Transactions on Mathematical Software Journal Issue: 2 Vol. 49; ISSN 0098-3500

Publisher:: Association for Computing MachineryCopyright Statement

Country of Publication:: United States

Language:: English

References (51)

hIPPYlib User Manual: Version 2 Villa, Umberto; Petra, Noemi; Ghattas, Omar figshare https://doi.org/10.6084/m9.figshare.12510578.v1	other	January 2020
Sampling via Measure Transport: An Introduction Marzouk, Youssef; Moselhy, Tarek; Parno, Matthew Handbook of Uncertainty Quantification https://doi.org/10.1007/978-3-319-11259-6_23-1	book	January 2016
Automated Solution of Differential Equations by the Finite Element Method Logg, Anders; Mardal, Kent-Andre; Wells, Garth Lecture Notes in Computational Science and Engineering https://doi.org/10.1007/978-3-642-23099-8	book	January 2012
Statistical and Computational Inverse Problems Kaipio, Jari P.; Somersalo, Erkki Applied Mathematical Sciences https://doi.org/10.1007/b138659	book	January 2005
On a Generalization of the Preconditioned Crank–Nicolson Metropolis Algorithm Rudolf, Daniel; Sprungk, Björn Foundations of Computational Mathematics, Vol. 18, Issue 2 https://doi.org/10.1007/s10208-016-9340-x	journal	November 2016
Efficient Nonlinear Solvers for Nodal High-Order Finite Elements in 3D Brown, Jed Journal of Scientific Computing, Vol. 45, Issue 1-3 https://doi.org/10.1007/s10915-010-9396-8	journal	July 2010
An Adaptive Version for the Metropolis Adjusted Langevin Algorithm with a Truncated Drift Atchadé, Yves F. Methodology and Computing in Applied Probability, Vol. 8, Issue 2 https://doi.org/10.1007/s11009-006-8550-0	journal	June 2006
An adaptive approach to Langevin MCMC Marshall, Tristan; Roberts, Gareth Statistics and Computing, Vol. 22, Issue 5 https://doi.org/10.1007/s11222-011-9276-6	journal	September 2011
Monte Carlo errors with less errors Wolff, Ulli Computer Physics Communications, Vol. 156, Issue 2 https://doi.org/10.1016/S0010-4655(03)00467-3	journal	January 2004
Scalable and efficient algorithms for the propagation of uncertainty from data through inference to prediction for large-scale problems, with application to flow of the Antarctic ice sheet Isaac, Tobin; Petra, Noemi; Stadler, Georg Journal of Computational Physics, Vol. 296 https://doi.org/10.1016/j.jcp.2015.04.047	journal	September 2015
Dimension-independent likelihood-informed MCMC Cui, Tiangang; Law, Kody J. H.; Marzouk, Youssef M. Journal of Computational Physics, Vol. 304 https://doi.org/10.1016/j.jcp.2015.10.008	journal	January 2016
Scalable posterior approximations for large-scale Bayesian inverse problems via likelihood-informed parameter and state reduction Cui, Tiangang; Marzouk, Youssef; Willcox, Karen Journal of Computational Physics, Vol. 315 https://doi.org/10.1016/j.jcp.2016.03.055	journal	June 2016
Geometric MCMC for infinite-dimensional inverse problems Beskos, Alexandros; Girolami, Mark; Lan, Shiwei Journal of Computational Physics, Vol. 335 https://doi.org/10.1016/j.jcp.2016.12.041	journal	April 2017
Inverse problems: A Bayesian perspective Stuart, A. M. Acta Numerica, Vol. 19 https://doi.org/10.1017/S0962492910000061	journal	May 2010
Langevin Diffusions and Metropolis-Hastings Algorithms Roberts, G. O. Methodology and Computing in Applied Probability, Vol. 4, Issue 4, p. 337-357 https://doi.org/10.1023/A:1023562417138	journal	December 2002
Equation of State Calculations by Fast Computing Machines Metropolis, Nicholas; Rosenbluth, Arianna W.; Rosenbluth, Marshall N. The Journal of Chemical Physics, Vol. 21, Issue 6 https://doi.org/10.1063/1.1699114	journal	June 1953
A general construction for parallelizing Metropolis−Hastings algorithms Calderhead, Ben Proceedings of the National Academy of Sciences, Vol. 111, Issue 49 https://doi.org/10.1073/pnas.1408184111	journal	November 2014
Explaining the Gibbs Sampler Casella, George; George, Edward I. The American Statistician, Vol. 46, Issue 3 https://doi.org/10.1080/00031305.1992.10475878	journal	August 1992
Accurate Approximations for Posterior Moments and Marginal Densities Tierney, Luke; Kadane, Joseph B. Journal of the American Statistical Association, Vol. 81, Issue 393 https://doi.org/10.1080/01621459.1986.10478240	journal	March 1986
General Methods for Monitoring Convergence of Iterative Simulations Brooks, Stephen P.; Gelman, Andrew Journal of Computational and Graphical Statistics, Vol. 7, Issue 4 https://doi.org/10.1080/10618600.1998.10474787	journal	December 1998
Data-free likelihood-informed dimension reduction of Bayesian inverse problems Cui, Tiangang; Zahm, Olivier Inverse Problems, Vol. 37, Issue 4 https://doi.org/10.1088/1361-6420/abeafb	journal	March 2021
Certified dimension reduction in nonlinear Bayesian inverse problems Zahm, Olivier; Cui, Tiangang; Law, Kody Mathematics of Computation, Vol. 91, Issue 336 https://doi.org/10.1090/mcom/3737	journal	April 2022
Monte Carlo sampling methods using Markov chains and their applications Hastings, W. K. Biometrika, Vol. 57, Issue 1 https://doi.org/10.1093/biomet/57.1.97	journal	April 1970
Multivariate output analysis for Markov chain Monte Carlo Vats, Dootika; Flegal, James M.; Jones, Galin L. Biometrika, Vol. 106, Issue 2 https://doi.org/10.1093/biomet/asz002	journal	April 2019
Kalman filtering and smoothing solutions to temporal Gaussian process regression models Hartikainen, Jouni; Sarkka, Simo 2010 IEEE International Workshop on Machine Learning for Signal Processing https://doi.org/10.1109/MLSP.2010.5589113	conference	August 2010
Riemann manifold Langevin and Hamiltonian Monte Carlo methods: Riemann Manifold Langevin and Hamiltonian Monte Carlo Methods Girolami, Mark; Calderhead, Ben Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 73, Issue 2 https://doi.org/10.1111/j.1467-9868.2010.00765.x	journal	March 2011
An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach: Link between Gaussian Fields and Gaussian Markov Random Fields Lindgren, Finn; Rue, Håvard; Lindström, Johan Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 73, Issue 4 https://doi.org/10.1111/j.1467-9868.2011.00777.x	journal	August 2011
Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions Halko, N.; Martinsson, P. G.; Tropp, J. A. SIAM Review, Vol. 53, Issue 2 https://doi.org/10.1137/090771806	journal	January 2011
Inverse Problem Theory and Methods for Model Parameter Estimation Tarantola, Albert https://doi.org/10.1137/1.9780898717921	book	January 2005
A Stochastic Newton MCMC Method for Large-Scale Statistical Inverse Problems with Application to Seismic Inversion Martin, James; Wilcox, Lucas C.; Burstedde, Carsten SIAM Journal on Scientific Computing, Vol. 34, Issue 3 https://doi.org/10.1137/110845598	journal	January 2012
Adaptive Smolyak Pseudospectral Approximations Conrad, Patrick R.; Marzouk, Youssef M. SIAM Journal on Scientific Computing, Vol. 35, Issue 6 https://doi.org/10.1137/120890715	journal	January 2013
A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems Part I: The Linearized Case, with Application to Global Seismic Inversion Bui-Thanh, Tan; Ghattas, Omar; Martin, James SIAM Journal on Scientific Computing, Vol. 35, Issue 6 https://doi.org/10.1137/12089586X	journal	January 2013
A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems, Part II: Stochastic Newton MCMC with Application to Ice Sheet Flow Inverse Problems Petra, Noemi; Martin, James; Stadler, Georg SIAM Journal on Scientific Computing, Vol. 36, Issue 4 https://doi.org/10.1137/130934805	journal	January 2014
Algorithms for Kullback--Leibler Approximation of Probability Measures in Infinite Dimensions Pinski, F. J.; Simpson, G.; Stuart, A. M. SIAM Journal on Scientific Computing, Vol. 37, Issue 6 https://doi.org/10.1137/14098171X	journal	January 2015
Transport Map Accelerated Markov Chain Monte Carlo Parno, Matthew D.; Marzouk, Youssef M. SIAM/ASA Journal on Uncertainty Quantification, Vol. 6, Issue 2 https://doi.org/10.1137/17M1134640	journal	January 2018
Scalable Matrix-Free Adaptive Product-Convolution Approximation for Locally Translation-Invariant Operators Alger, Nick; Rao, Vishwas; Myers, Aaron SIAM Journal on Scientific Computing, Vol. 41, Issue 4 https://doi.org/10.1137/18M1189324	journal	January 2019
Scalable Optimization-Based Sampling on Function Space Bardsley, Johnathan M.; Cui, Tiangang; Marzouk, Youssef M. SIAM Journal on Scientific Computing, Vol. 42, Issue 2 https://doi.org/10.1137/19M1245220	journal	January 2020
Multilevel Markov Chain Monte Carlo Dodwell, T. J.; Ketelsen, C.; Scheichl, R. SIAM Review, Vol. 61, Issue 3 https://doi.org/10.1137/19M126966X	journal	January 2019
Hierarchical Matrix Approximations of Hessians Arising in Inverse Problems Governed by PDEs Ambartsumyan, Ilona; Boukaram, Wajih; Bui-Thanh, Tan SIAM Journal on Scientific Computing, Vol. 42, Issue 5 https://doi.org/10.1137/19M1270367	journal	January 2020
hIPPYlib: An Extensible Software Framework for Large-Scale Inverse Problems Governed by PDEs: Part I: Deterministic Inversion and Linearized Bayesian Inference Villa, Umberto; Petra, Noemi; Ghattas, Omar ACM Transactions on Mathematical Software, Vol. 47, Issue 2 https://doi.org/10.1145/3428447	journal	April 2021
Bayesian Data Analysis Gelman, Andrew; Carlin, John B.; Stern, Hal S. Chapman and Hall/CRC https://doi.org/10.1201/9780429258480	book	July 2003
Batch means and spectral variance estimators in Markov chain Monte Carlo Flegal, James M.; Jones, Galin L. The Annals of Statistics, Vol. 38, Issue 2 https://doi.org/10.1214/09-AOS735	journal	April 2010
Spectral gaps for a Metropolis–Hastings algorithm in infinite dimensions Hairer, Martin; Stuart, Andrew M.; Vollmer, Sebastian J. The Annals of Applied Probability, Vol. 24, Issue 6 https://doi.org/10.1214/13-AAP982	journal	December 2014
MCMC Methods for Functions: Modifying Old Algorithms to Make Them Faster Cotter, S. L.; Roberts, G. O.; Stuart, A. M. Statistical Science, Vol. 28, Issue 3 https://doi.org/10.1214/13-STS421	journal	August 2013
General state space Markov chains and MCMC algorithms Roberts, Gareth O.; Rosenthal, Jeffrey S. Probability Surveys, Vol. 1, Issue 0 https://doi.org/10.1214/154957804100000024	journal	January 2004
Rank-Normalization, Folding, and Localization: An Improved Rˆ for Assessing Convergence of MCMC (with Discussion) Vehtari, Aki; Gelman, Andrew; Simpson, Daniel Bayesian Analysis, Vol. 16, Issue 2 https://doi.org/10.1214/20-ba1221	journal	June 2021
Inference from Iterative Simulation Using Multiple Sequences Gelman, Andrew; Rubin, Donald B. Statistical Science, Vol. 7, Issue 4 https://doi.org/10.1214/ss/1177011136	journal	November 1992
Laplace's 1774 Memoir on Inverse Probability Stigler, Stephen M. Statistical Science, Vol. 1, Issue 3 https://doi.org/10.1214/ss/1177013620	journal	August 1986
An Adaptive Metropolis Algorithm Haario, Heikki; Saksman, Eero; Tamminen, Johanna Bernoulli, Vol. 7, Issue 2 https://doi.org/10.2307/3318737	journal	April 2001
Mitigating the influence of the boundary on PDE-based covariance operators Daon, Yair; Stadler, Georg Inverse Problems & Imaging, Vol. 12, Issue 5 https://doi.org/10.3934/ipi.2018045	journal	January 2018
Gaussian Processes for Machine Learning Rasmussen, Carl Edward; Williams, Christopher K. I. The MIT Press https://doi.org/10.7551/mitpress/3206.001.0001	book	January 2005

Similar Records

Fast alogorithms for Bayesian uncertainty quantification in large-scale linear inverse problems based on low-rank partial Hessian approximations

Journal Article · Fri Dec 31 23:00:00 EST 2010 · SIAM Journal on Scientific Computing · OSTI ID:1006461

Large-Scale Optimization for Bayesian Inference in Complex Systems (Final Report)

hIPPYlib-MUQ: A Bayesian Inference Software Framework for Integration of Data with Complex Predictive Models under Uncertainty

Citation Formats

References (51)

Similar Records

Related Subjects