Kullback--Leibler Approximation for Probability Measures on Infinite Dimensional Spaces
Abstract
In a variety of applications it is important to extract information from a probability measure μ on an infinite dimensional space. Examples include the Bayesian approach to inverse problems and (possibly conditioned) continuous time Markov processes. It may then be of interest to find a measure ν, from within a simple class of measures, which approximates μ. Here, this problem is studied in the case where the Kullback–Leibler divergence is employed to measure the quality of the approximation. A calculus of variations viewpoint is adopted, and the particular case where ν is chosen from the set of Gaussian measures is studied in detail. Basic existence and uniqueness theorems are established, together with properties of minimizing sequences. Furthermore, parameterization of the class of Gaussians through the mean and inverse covariance is introduced, the need for regularization is explained, and a regularized minimization is studied in detail. The calculus of variations framework resulting from this work provides the appropriate underpinning for computational algorithms.
- Authors:
-
- Univ. of Cincinnati, OH (United States). Dept. of Physics
- Drexel Univ., Philadelphia, PA (United States). Dept. of Mathematics
- Warwick Univ., Coventry (United Kingdom). Mathematics Inst.
- Publication Date:
- Research Org.:
- Warwick Univ., Coventry (United Kingdom)
- Sponsoring Org.:
- USDOE Office of Science (SC); European Research Council (ERC); Engineering and Physical Sciences Research Council (EPSRC); US Department of the Navy, Office of Naval Research (ONR); National Science Foundation (NSF)
- OSTI Identifier:
- 1459163
- Grant/Contract Number:
- SC0002085; OISE-0967140
- Resource Type:
- Accepted Manuscript
- Journal Name:
- SIAM Journal of Mathematical Analysis
- Additional Journal Information:
- Journal Volume: 47; Journal Issue: 6; Journal ID: ISSN 0036-1410
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; Kullback–Leibler divergence; relative entropy; Gaussian measures
Citation Formats
Pinski, Frank J., Simpson, G., Stuart, A. M., and Weber, Hendrik. Kullback--Leibler Approximation for Probability Measures on Infinite Dimensional Spaces. United States: N. p., 2015.
Web. doi:10.1137/140962802.
Pinski, Frank J., Simpson, G., Stuart, A. M., & Weber, Hendrik. Kullback--Leibler Approximation for Probability Measures on Infinite Dimensional Spaces. United States. https://doi.org/10.1137/140962802
Pinski, Frank J., Simpson, G., Stuart, A. M., and Weber, Hendrik. Thu .
"Kullback--Leibler Approximation for Probability Measures on Infinite Dimensional Spaces". United States. https://doi.org/10.1137/140962802. https://www.osti.gov/servlets/purl/1459163.
@article{osti_1459163,
title = {Kullback--Leibler Approximation for Probability Measures on Infinite Dimensional Spaces},
author = {Pinski, Frank J. and Simpson, G. and Stuart, A. M. and Weber, Hendrik},
abstractNote = {In a variety of applications it is important to extract information from a probability measure μ on an infinite dimensional space. Examples include the Bayesian approach to inverse problems and (possibly conditioned) continuous time Markov processes. It may then be of interest to find a measure ν, from within a simple class of measures, which approximates μ. Here, this problem is studied in the case where the Kullback–Leibler divergence is employed to measure the quality of the approximation. A calculus of variations viewpoint is adopted, and the particular case where ν is chosen from the set of Gaussian measures is studied in detail. Basic existence and uniqueness theorems are established, together with properties of minimizing sequences. Furthermore, parameterization of the class of Gaussians through the mean and inverse covariance is introduced, the need for regularization is explained, and a regularized minimization is studied in detail. The calculus of variations framework resulting from this work provides the appropriate underpinning for computational algorithms.},
doi = {10.1137/140962802},
journal = {SIAM Journal of Mathematical Analysis},
number = 6,
volume = 47,
place = {United States},
year = {Thu Nov 05 00:00:00 EST 2015},
month = {Thu Nov 05 00:00:00 EST 2015}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 27 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
A numerical method for detecting singular minimizers
journal, March 1987
- Ball, J. M.; Knowles, G.
- Numerische Mathematik, Vol. 51, Issue 2
Bayesian inverse problems for functions and applications to fluid mechanics
journal, October 2009
- Cotter, S. L.; Dashti, M.; Robinson, J. C.
- Inverse Problems, Vol. 25, Issue 11
$I$-Divergence Geometry of Probability Distributions and Minimization Problems
journal, February 1975
- Csiszar, I.
- The Annals of Probability, Vol. 3, Issue 1
MAP estimators and their consistency in Bayesian nonparametric inverse problems
journal, September 2013
- Dashti, M.; Law, K. J. H.; Stuart, A. M.
- Inverse Problems, Vol. 29, Issue 9
Uncertainty Quantification and Weak Approximation of an Elliptic Inverse Problem
journal, January 2011
- Dashti, M.; Stuart, A. M.
- SIAM Journal on Numerical Analysis, Vol. 49, Issue 6
Monge-Kantorovitch Measure Transportation and Monge-Amp�re Equation on Wiener Space
journal, March 2004
- Feyel, D.; �st�nel, A. S.
- Probability Theory and Related Fields, Vol. 128, Issue 3
Quantifying the Predictive Skill in Long-Range Forecasting. Part I: Coarse-Grained Predictions in a Simple Ocean Model
journal, March 2012
- Giannakis, Dimitrios; Majda, Andrew J.
- Journal of Climate, Vol. 25, Issue 6
Information-theoretic tools for parametrized coarse-graining of non-equilibrium extended systems
journal, August 2013
- Katsoulakis, Markos A.; Plecháč, Petr
- The Journal of Chemical Physics, Vol. 139, Issue 7
Coarse-graining schemes and a posteriori error estimates for stochastic lattice systems
journal, May 2007
- Katsoulakis, Markos A.; Plecháč, Petr; Rey-Bellet, Luc
- ESAIM: Mathematical Modelling and Numerical Analysis, Vol. 41, Issue 3
Improving model fidelity and sensitivity for complex systems through empirical information theory
journal, June 2011
- Majda, A. J.; Gershgorin, B.
- Proceedings of the National Academy of Sciences, Vol. 108, Issue 25
A Convexity Principle for Interacting Gases
journal, June 1997
- McCann, Robert J.
- Advances in Mathematics, Vol. 128, Issue 1
Γ-Limit for Transition Paths of Maximal Probability
journal, February 2012
- Pinski, F. J.; Stuart, A. M.; Theil, F.
- Journal of Statistical Physics, Vol. 146, Issue 5
Posterior consistency via precision operators for Bayesian nonparametric drift estimation in SDEs
journal, February 2013
- Pokern, Y.; Stuart, A. M.; van Zanten, J. H.
- Stochastic Processes and their Applications, Vol. 123, Issue 2
Works referencing / citing this record:
A Generalized Relative (α, β)-Entropy: Geometric Properties and Applications to Robust Statistical Inference
journal, May 2018
- Ghosh, Abhik; Basu, Ayanendranath
- Entropy, Vol. 20, Issue 5
Computationally Efficient Variational Approximations for Bayesian Inverse Problems
journal, July 2016
- Tsilifis, Panagiotis; Bilionis, Ilias; Katsounaros, Ioannis
- Journal of Verification, Validation and Uncertainty Quantification, Vol. 1, Issue 3