Scalable posterior approximations for large-scale Bayesian inverse problems via likelihood-informed parameter and state reduction
Abstract
Two prime bottlenecks to the solution of large-scale Bayesian inverse problems are the scaling of posterior sampling algorithms to high-dimensional parameter spaces and the computational cost of forward model evaluations. However, incomplete or noisy data, the state variation and parameter dependence of the forward model, and correlations in the prior collectively provide useful structure that can be exploited for dimension reduction in this setting—both in the parameter space of the inverse problem and in the state space of the forward model. To this end, we show how to jointly construct low-dimensional subspaces of the parameter space and the state space in order to accelerate the Bayesian solution of the inverse problem. As a byproduct of state dimension reduction, we also show how to identify low-dimensional subspaces of the data in problems with high-dimensional observations. These subspaces enable approximation of the posterior as a product of two factors: (i) a projection of the posterior onto a low-dimensional parameter subspace, wherein the original likelihood is replaced by an approximation involving a reduced model; and (ii) the marginal prior distribution on the high-dimensional complement of the parameter subspace. We introduce and compare several strategies for constructing these subspaces using only a limited numbermore »
- Authors:
-
- Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)
- Publication Date:
- Research Org.:
- Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); Finnish Meteorological Institute
- OSTI Identifier:
- 1548326
- Alternate Identifier(s):
- OSTI ID: 1325283
- Grant/Contract Number:
- SC0009297
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Journal of Computational Physics
- Additional Journal Information:
- Journal Volume: 315; Journal Issue: C; Journal ID: ISSN 0021-9991
- Publisher:
- Elsevier
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; Inverse problems; Bayesian inference; Dimension reduction; Model reduction; Low-rank approximation; Markov chain Monte Carlo
Citation Formats
Cui, Tiangang, Marzouk, Youssef, and Willcox, Karen. Scalable posterior approximations for large-scale Bayesian inverse problems via likelihood-informed parameter and state reduction. United States: N. p., 2016.
Web. doi:10.1016/j.jcp.2016.03.055.
Cui, Tiangang, Marzouk, Youssef, & Willcox, Karen. Scalable posterior approximations for large-scale Bayesian inverse problems via likelihood-informed parameter and state reduction. United States. https://doi.org/10.1016/j.jcp.2016.03.055
Cui, Tiangang, Marzouk, Youssef, and Willcox, Karen. Tue .
"Scalable posterior approximations for large-scale Bayesian inverse problems via likelihood-informed parameter and state reduction". United States. https://doi.org/10.1016/j.jcp.2016.03.055. https://www.osti.gov/servlets/purl/1548326.
@article{osti_1548326,
title = {Scalable posterior approximations for large-scale Bayesian inverse problems via likelihood-informed parameter and state reduction},
author = {Cui, Tiangang and Marzouk, Youssef and Willcox, Karen},
abstractNote = {Two prime bottlenecks to the solution of large-scale Bayesian inverse problems are the scaling of posterior sampling algorithms to high-dimensional parameter spaces and the computational cost of forward model evaluations. However, incomplete or noisy data, the state variation and parameter dependence of the forward model, and correlations in the prior collectively provide useful structure that can be exploited for dimension reduction in this setting—both in the parameter space of the inverse problem and in the state space of the forward model. To this end, we show how to jointly construct low-dimensional subspaces of the parameter space and the state space in order to accelerate the Bayesian solution of the inverse problem. As a byproduct of state dimension reduction, we also show how to identify low-dimensional subspaces of the data in problems with high-dimensional observations. These subspaces enable approximation of the posterior as a product of two factors: (i) a projection of the posterior onto a low-dimensional parameter subspace, wherein the original likelihood is replaced by an approximation involving a reduced model; and (ii) the marginal prior distribution on the high-dimensional complement of the parameter subspace. We introduce and compare several strategies for constructing these subspaces using only a limited number of forward and adjoint model simulations. The resulting posterior approximations can rapidly be characterized using standard sampling techniques, e.g., Markov chain Monte Carlo. Two numerical examples demonstrate the accuracy and efficiency of our approach: inversion of an integral equation in atmospheric remote sensing, where the data dimension is very high; and the inference of a heterogeneous transmissivity field in a groundwater system, which involves a partial differential equation forward model with high dimensional state and parameters.},
doi = {10.1016/j.jcp.2016.03.055},
journal = {Journal of Computational Physics},
number = C,
volume = 315,
place = {United States},
year = {Tue Mar 29 00:00:00 EDT 2016},
month = {Tue Mar 29 00:00:00 EDT 2016}
}
Web of Science
Works referenced in this record:
Equation of State Calculations by Fast Computing Machines
journal, June 1953
- Metropolis, Nicholas; Rosenbluth, Arianna W.; Rosenbluth, Marshall N.
- The Journal of Chemical Physics, Vol. 21, Issue 6
Monte Carlo sampling methods using Markov chains and their applications
journal, April 1970
- Hastings, W. K.
- Biometrika, Vol. 57, Issue 1
Optimal scaling of discrete approximations to Langevin diffusions
journal, February 1998
- Roberts, Gareth O.; Rosenthal, Jeffrey S.
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 60, Issue 1
Optimal scaling for various Metropolis-Hastings algorithms
journal, November 2001
- Roberts, Gareth O.; Rosenthal, Jeffrey S.
- Statistical Science, Vol. 16, Issue 4
Diffusion limits of the random walk Metropolis algorithm in high dimensions
journal, June 2012
- Mattingly, Jonathan C.; Pillai, Natesh S.; Stuart, Andrew M.
- The Annals of Applied Probability, Vol. 22, Issue 3
Optimal scaling and diffusion limits for the Langevin algorithm in high dimensions
journal, December 2012
- Pillai, Natesh S.; Stuart, Andrew M.; Thiéry, Alexandre H.
- The Annals of Applied Probability, Vol. 22, Issue 6
Optimal Low-rank Approximations of Bayesian Linear Inverse Problems
journal, January 2015
- Spantini, Alessio; Solonen, Antti; Cui, Tiangang
- SIAM Journal on Scientific Computing, Vol. 37, Issue 6
Likelihood-informed dimension reduction for nonlinear inverse problems
journal, October 2014
- Cui, T.; Martin, J.; Marzouk, Y. M.
- Inverse Problems, Vol. 30, Issue 11
Data-driven model reduction for the Bayesian solution of inverse problems: DATA-DRIVEN MODEL REDUCTION FOR INVERSE PROBLEMS
journal, August 2014
- Cui, Tiangang; Marzouk, Youssef M.; Willcox, Karen E.
- International Journal for Numerical Methods in Engineering, Vol. 102, Issue 5
Dimension-independent likelihood-informed MCMC
journal, January 2016
- Cui, Tiangang; Law, Kody J. H.; Marzouk, Youssef M.
- Journal of Computational Physics, Vol. 304
Reduced basis technique for collapse analysis of shells
journal, March 1981
- Noor, Ahmed K.; Peters, Jeanne M.; Andersen, C. M.
- AIAA Journal, Vol. 19, Issue 3
Turbulence and the dynamics of coherent structures. I. Coherent structures
journal, January 1987
- Sirovich, Lawrence
- Quarterly of Applied Mathematics, Vol. 45, Issue 3
Markov chain Monte Carlo Using an Approximation
journal, December 2005
- Christen, J. Andrés; Fox, Colin
- Journal of Computational and Graphical Statistics, Vol. 14, Issue 4
Dimensionality reduction and polynomial chaos acceleration of Bayesian inference in inverse problems
journal, April 2009
- Marzouk, Youssef M.; Najm, Habib N.
- Journal of Computational Physics, Vol. 228, Issue 6
The Wiener--Askey Polynomial Chaos for Stochastic Differential Equations
journal, January 2002
- Xiu, Dongbin; Karniadakis, George Em
- SIAM Journal on Scientific Computing, Vol. 24, Issue 2
Electrical impedance tomography imaging with reduced-order model based on proper orthogonal decomposition
journal, April 2013
- Lipponen, Antti; Seppänen, Aku; Kaipio, Jari
- Journal of Electronic Imaging, Vol. 22, Issue 2
An efficient Bayesian inference approach to inverse problems based on an adaptive sparse grid collocation method
journal, February 2009
- Ma, Xiang; Zabaras, Nicholas
- Inverse Problems, Vol. 25, Issue 3
Parameter and State Model Reduction for Large-Scale Statistical Inverse Problems
journal, January 2010
- Lieberman, Chad; Willcox, Karen; Ghattas, Omar
- SIAM Journal on Scientific Computing, Vol. 32, Issue 5
Fast Algorithms for Bayesian Uncertainty Quantification in Large-Scale Linear Inverse Problems Based on Low-Rank Partial Hessian Approximations
journal, January 2011
- Flath, H. P.; Wilcox, L. C.; Akçelik, V.
- SIAM Journal on Scientific Computing, Vol. 33, Issue 1
A Stochastic Newton MCMC Method for Large-Scale Statistical Inverse Problems with Application to Seismic Inversion
journal, January 2012
- Martin, James; Wilcox, Lucas C.; Burstedde, Carsten
- SIAM Journal on Scientific Computing, Vol. 34, Issue 3
A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems Part I: The Linearized Case, with Application to Global Seismic Inversion
journal, January 2013
- Bui-Thanh, Tan; Ghattas, Omar; Martin, James
- SIAM Journal on Scientific Computing, Vol. 35, Issue 6
A Computational Framework for Infinite-Dimensional Bayesian Inverse Problems, Part II: Stochastic Newton MCMC with Application to Ice Sheet Flow Inverse Problems
journal, January 2014
- Petra, Noemi; Martin, James; Stadler, Georg
- SIAM Journal on Scientific Computing, Vol. 36, Issue 4
An ‘empirical interpolation’ method: application to efficient reduced-basis discretization of partial differential equations
journal, November 2004
- Barrault, Maxime; Maday, Yvon; Nguyen, Ngoc Cuong
- Comptes Rendus Mathematique, Vol. 339, Issue 9
Nonlinear Model Reduction via Discrete Empirical Interpolation
journal, January 2010
- Chaturantabut, Saifon; Sorensen, Danny C.
- SIAM Journal on Scientific Computing, Vol. 32, Issue 5
Using Bayesian statistics in the estimation of heat source in radiation
journal, January 2005
- Wang, Jingbo; Zabaras, Nicholas
- International Journal of Heat and Mass Transfer, Vol. 48, Issue 1
Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions
journal, January 2011
- Halko, N.; Martinsson, P. G.; Tropp, J. A.
- SIAM Review, Vol. 53, Issue 2
Randomized algorithms for the low-rank approximation of matrices
journal, December 2007
- Liberty, E.; Woolfe, F.; Martinsson, P. -G.
- Proceedings of the National Academy of Sciences, Vol. 104, Issue 51
An Adaptive Version for the Metropolis Adjusted Langevin Algorithm with a Truncated Drift
journal, June 2006
- Atchadé, Yves F.
- Methodology and Computing in Applied Probability, Vol. 8, Issue 2
A random map implementation of implicit filters
journal, February 2012
- Morzfeld, Matthias; Tu, Xuemin; Atkins, Ethan
- Journal of Computational Physics, Vol. 231, Issue 4
Randomize-Then-Optimize: A Method for Sampling from Posterior Distributions in Nonlinear Inverse Problems
journal, January 2014
- Bardsley, Johnathan M.; Solonen, Antti; Haario, Heikki
- SIAM Journal on Scientific Computing, Vol. 36, Issue 4
Sparse deterministic approximation of Bayesian inverse problems
journal, March 2012
- Schwab, C.; Stuart, A. M.
- Inverse Problems, Vol. 28, Issue 4
Sparse, adaptive Smolyak quadratures for Bayesian inverse problems
journal, May 2013
- Schillings, Claudia; Schwab, Christoph
- Inverse Problems, Vol. 29, Issue 6
Sparse-grid, reduced-basis Bayesian inversion
journal, December 2015
- Chen, Peng; Schwab, Christoph
- Computer Methods in Applied Mechanics and Engineering, Vol. 297
Markov chain Monte Carlo methods for high dimensional inversion in remote sensing
journal, August 2004
- Haario, H.; Laine, M.; Lehtinen, M.
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 66, Issue 3
On the convergence of interior-reflective Newton methods for nonlinear minimization subject to bounds
journal, October 1994
- Coleman, Thomas F.; Li, Yuying
- Mathematical Programming, Vol. 67, Issue 1-3
An Interior Trust Region Approach for Nonlinear Minimization Subject to Bounds
journal, May 1996
- Coleman, Thomas F.; Li, Yuying
- SIAM Journal on Optimization, Vol. 6, Issue 2
An explicit link between Gaussian fields and Gaussian Markov random fields: the stochastic partial differential equation approach: Link between Gaussian Fields and Gaussian Markov Random Fields
journal, August 2011
- Lindgren, Finn; Rue, Håvard; Lindström, Johan
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 73, Issue 4
Optimal scaling for various Metropolis-Hastings algorithms
journal, November 2001
- Roberts, Gareth O.; Rosenthal, Jeffrey S.
- Statistical Science, Vol. 16, Issue 4
Diffusion limits of the random walk Metropolis algorithm in high dimensions
text, January 2010
- Mattingly, Jonathan C.; Pillai, Natesh S.; Stuart, Andrew M.
- arXiv
Works referencing / citing this record:
Bayesian Probabilistic Numerical Methods in Time-Dependent State Estimation for Industrial Hydrocyclone Equipment
journal, February 2019
- Oates, Chris J.; Cockayne, Jon; Aykroyd, Robert G.
- Journal of the American Statistical Association
Adaptivity in Bayesian Inverse Finite Element Problems: Learning and Simultaneous Control of Discretisation and Sampling Errors
journal, February 2019
- Kerfriden, Pierre; Kundu, Abhishek; Claus, Susanne
- Materials, Vol. 12, Issue 4
Bayesian Probabilistic Numerical Methods in Time-Dependent State Estimation for Industrial Hydrocyclone Equipment
preprint, January 2017
- Oates, Chris. J.; Cockayne, Jon; Aykroyd, Robert G.
- arXiv
Rate-optimal refinement strategies for local approximation MCMC
journal, August 2022
- Davis, Andrew D.; Marzouk, Youssef; Smith, Aaron
- Statistics and Computing, Vol. 32, Issue 4
Hessian-based sampling for high-dimensional model reduction
preprint, January 2018
- Chen, Peng; Ghattas, Omar
- arXiv