Non-asymptotic analysis of ensemble Kalman updates: effective dimension and localization

Al-Ghattas, Omar; Sanz-Alonso, Daniel

doi:10.1093/imaiai/iaad043

Title: Non-asymptotic analysis of ensemble Kalman updates: effective dimension and localization

Journal Article · 27 December 2023 · Information and Inference (Online)

DOI: https://doi.org/10.1093/imaiai/iaad043 · OSTI ID:2274698

Al-Ghattas, Omar ^[1]; Sanz-Alonso, Daniel ^[1]

University of Chicago, IL (United States)

Many modern algorithms for inverse problems and data assimilation rely on ensemble Kalman updates to blend prior predictions with observed data. Ensemble Kalman methods often perform well with a small ensemble size, which is essential in applications where generating each particle is costly. This paper develops a non-asymptotic analysis of ensemble Kalman updates, which rigorously explains why a small ensemble size suffices if the prior covariance has moderate effective dimension due to fast spectrum decay or approximate sparsity. Here, we present our theory in a unified framework, comparing everal implementations of ensemble Kalman updates that use perturbed observations, square root filtering and localization. As part of our analysis, we develop new dimension-free covariance estimation bounds for approximately sparse matrices that may be of independent interest.

View Journal Article

Cite

Export

Save

Research Organization:: Univ. of Chicago, IL (United States); University of Chicago, IL (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE; USDOE Office of Science (SC); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0022232

OSTI ID:: 2274698

Journal Information:: Information and Inference (Online), Journal Name: Information and Inference (Online) Journal Issue: 1 Vol. 13; ISSN 2049-8772

Publisher:: Oxford University PressCopyright Statement

Country of Publication:: United States

Language:: English

References (89)

Conditions for successful data assimilation: CONDITIONS FOR DATA ASSIMILATION Chorin, Alexandre J.; Morzfeld, Matthias Journal of Geophysical Research: Atmospheres, Vol. 118, Issue 20 https://doi.org/10.1002/2013JD019838	journal	October 2013
Performance of Ensemble Kalman Filters in Large Dimensions Majda, Andrew J.; Tong, Xin T. Communications on Pure and Applied Mathematics, Vol. 71, Issue 5 https://doi.org/10.1002/cpa.21722	journal	November 2017
Construction of correlation functions in two and three dimensions Gaspari, Gregory; Cohn, Stephen E. Quarterly Journal of the Royal Meteorological Society, Vol. 125, Issue 554 https://doi.org/10.1002/qj.49712555417	journal	January 1999
A localization technique for ensemble Kalman filters Bergemann, Kay; Reich, Sebastian Quarterly Journal of the Royal Meteorological Society, Vol. 136, Issue 648 https://doi.org/10.1002/qj.591	journal	March 2010
A mollified ensemble Kalman filter Bergemann, Kay; Reich, Sebastian Quarterly Journal of the Royal Meteorological Society, Vol. 136, Issue 651 https://doi.org/10.1002/qj.672	journal	July 2010
Nonlinear Data Assimilation Van Leeuwen, Peter Jan; Cheng, Yuan; Reich, Sebastian Frontiers in Applied Dynamical Systems: Reviews and Tutorials https://doi.org/10.1007/978-3-319-18347-3	book	January 2015
Data Assimilation: A Mathematical Introduction Law, Kody; Stuart, Andrew; Zygalakis, Konstantinos Texts in Applied Mathematics https://doi.org/10.1007/978-3-319-20325-6	book	January 2015
Data Assimilation Evensen, Geir https://doi.org/10.1007/978-3-642-03711-5	book	January 2009
Upper and Lower Bounds for Stochastic Processes Talagrand, Michel Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge / A Series of Modern Surveys in Mathematics https://doi.org/10.1007/978-3-642-54075-2	book	January 2014
Performance Analysis of Local Ensemble Kalman Filter Tong, Xin T. Journal of Nonlinear Science, Vol. 28, Issue 4 https://doi.org/10.1007/s00332-018-9453-2	journal	March 2018
Partial estimation of covariance matrices Levina, Elizaveta; Vershynin, Roman Probability Theory and Related Fields, Vol. 153, Issue 3-4 https://doi.org/10.1007/s00440-011-0349-4	journal	March 2011
Sampling strategies and square root analysis schemes for the EnKF Evensen, Geir Ocean Dynamics, Vol. 54, Issue 6 https://doi.org/10.1007/s10236-004-0099-2	journal	December 2004
On the convergence of the ensemble Kalman filter Mandel, Jan; Cobb, Loren; Beezley, Jonathan D. Applications of Mathematics, Vol. 56, Issue 6 https://doi.org/10.1007/s10492-011-0031-2	journal	December 2011
Ensemble Kalman inversion: mean-field limit and convergence analysis Ding, Zhiyan; Li, Qin Statistics and Computing, Vol. 31, Issue 1 https://doi.org/10.1007/s11222-020-09976-0	journal	January 2021
On numerical properties of the ensemble Kalman filter for data assimilation Li, Jia; Xiu, Dongbin Computer Methods in Applied Mechanics and Engineering, Vol. 197, Issue 43-44 https://doi.org/10.1016/j.cma.2008.03.022	journal	August 2008
Estimation of high-dimensional prior and posterior covariance matrices in Kalman filter variants Furrer, Reinhard; Bengtsson, Thomas Journal of Multivariate Analysis, Vol. 98, Issue 2 https://doi.org/10.1016/j.jmva.2006.08.003	journal	February 2007
On the iterated forms of Kalman filters using statistical linearization Ungarala, Sridhar Journal of Process Control, Vol. 22, Issue 5 https://doi.org/10.1016/j.jprocont.2012.03.004	journal	June 2012
Upper bounds on product and multiplier empirical processes Mendelson, Shahar Stochastic Processes and their Applications, Vol. 126, Issue 12 https://doi.org/10.1016/j.spa.2016.04.019	journal	December 2016
Inverse Problems and Data Assimilation Sanz-Alonso, Daniel; Stuart, Andrew; Taeb, Armeen https://doi.org/10.1017/9781009414319	journal	July 2023
High-Dimensional Probability: An Introduction with Applications in Data Science Vershynin, Roman https://doi.org/10.1017/9781108231596	book	September 2018
High-Dimensional Statistics Wainwright, Martin J. https://doi.org/10.1017/9781108627771	journal	February 2019
Probabilistic Forecasting and Bayesian Data Assimilation Reich, Sebastian; Cotter, Colin https://doi.org/10.1017/CBO9781107706804	journal	May 2015
Matrix Analysis Horn, Roger A.; Johnson, Charles R. https://doi.org/10.1017/CBO9781139020411	book	April 2013
Filtering Complex Turbulent Systems Majda, Andrew J.; Harlim, John https://doi.org/10.1017/CBO9781139061308	journal	March 2012
Bayesian Filtering and Smoothing Sarkka, Simo https://doi.org/10.1017/CBO9781139344203	journal	January 2013
Inverse problems: A Bayesian perspective Stuart, A. M. Acta Numerica, Vol. 19 https://doi.org/10.1017/S0962492910000061	journal	May 2010
Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics Evensen, Geir Journal of Geophysical Research, Vol. 99, Issue C5 https://doi.org/10.1029/94JC00572	journal	January 1994
Concrete ensemble Kalman filters with rigorous catastrophic filter divergence Kelly, David; Majda, Andrew J.; Tong, Xin T. Proceedings of the National Academy of Sciences, Vol. 112, Issue 34 https://doi.org/10.1073/pnas.1511063112	journal	August 2015
Understanding the Ensemble Kalman Filter Katzfuss, Matthias; Stroud, Jonathan R.; Wikle, Christopher K. The American Statistician, Vol. 70, Issue 4 https://doi.org/10.1080/00031305.2016.1141709	journal	October 2016
What the collapse of the ensemble Kalman filter tells us about particle filters Morzfeld, Matthias; Hodyss, Daniel; Snyder, Chris Tellus A: Dynamic Meteorology and Oceanography, Vol. 69, Issue 1 https://doi.org/10.1080/16000870.2017.1283809	journal	January 2017
A regularizing Levenberg - Marquardt scheme, with applications to inverse groundwater filtration problems Hanke, Martin Inverse Problems, Vol. 13, Issue 1 https://doi.org/10.1088/0266-5611/13/1/007	journal	February 1997
Ensemble Kalman methods for inverse problems Iglesias, Marco A.; Law, Kody J. H.; Stuart, Andrew M. Inverse Problems, Vol. 29, Issue 4 https://doi.org/10.1088/0266-5611/29/4/045001	journal	March 2013
A regularizing iterative ensemble Kalman method for PDE-constrained inverse problems Iglesias, Marco A. Inverse Problems, Vol. 32, Issue 2 https://doi.org/10.1088/0266-5611/32/2/025002	journal	January 2016
Well-posedness and accuracy of the ensemble Kalman filter in discrete and continuous time Kelly, D. T. B.; Law, K. J. H.; Stuart, A. M. Nonlinearity, Vol. 27, Issue 10 https://doi.org/10.1088/0951-7715/27/10/2579	journal	September 2014
Nonlinear stability and ergodicity of ensemble based Kalman filters Tong, Xin T.; Majda, Andrew J.; Kelly, David Nonlinearity, Vol. 29, Issue 2 https://doi.org/10.1088/0951-7715/29/2/657	journal	January 2016
Well posedness and convergence analysis of the ensemble Kalman inversion Blömker, Dirk; Schillings, Claudia; Wacker, Philipp Inverse Problems, Vol. 35, Issue 8 https://doi.org/10.1088/1361-6420/ab149c	journal	July 2019
Ensemble Kalman inversion: a derivative-free technique for machine learning tasks Kovachki, Nikola B.; Stuart, Andrew M. Inverse Problems, Vol. 35, Issue 9 https://doi.org/10.1088/1361-6420/ab1c3a	journal	August 2019
Localized ensemble Kalman inversion Tong, X. T.; Morzfeld, M. Inverse Problems, Vol. 39, Issue 6 https://doi.org/10.1088/1361-6420/accb08	journal	April 2023
Wasserstein stability estimates for covariance-preconditioned Fokker–Planck equations Carrillo, J. A.; Vaes, U. Nonlinearity, Vol. 34, Issue 4 https://doi.org/10.1088/1361-6544/abbe62	journal	February 2021
Gaussian Measures Bogachev, Vladimir https://doi.org/10.1090/surv/062	book	January 1998
On the spectral norm of Gaussian random matrices van Handel, Ramon Transactions of the American Mathematical Society, Vol. 369, Issue 11 https://doi.org/10.1090/tran/6922	journal	May 2017
The masked sample covariance estimator: an analysis using matrix concentration inequalities Chen, R. Y.; Gittens, A.; Tropp, J. A. Information and Inference, Vol. 1, Issue 1 https://doi.org/10.1093/imaiai/ias001	journal	May 2012
Data Assimilation Asch, Mark; Bocquet, Marc; Nodet, Maëlle https://doi.org/10.1137/1.9781611974546	book	January 2016
Convergence of the Square Root Ensemble Kalman Filter in the Large Ensemble Limit Kwiatkowski, Evan; Mandel, Jan SIAM/ASA Journal on Uncertainty Quantification, Vol. 3, Issue 1 https://doi.org/10.1137/140965363	journal	January 2015
Analysis of the Ensemble and Polynomial Chaos Kalman Filters in Bayesian Inverse Problems Ernst, Oliver G.; Sprungk, Björn; Starkloff, Hans-Jörg SIAM/ASA Journal on Uncertainty Quantification, Vol. 3, Issue 1 https://doi.org/10.1137/140981319	journal	January 2015
Deterministic Mean-Field Ensemble Kalman Filtering Law, Kody J. H.; Tembine, Hamidou; Tempone, Raul SIAM Journal on Scientific Computing, Vol. 38, Issue 3 https://doi.org/10.1137/140984415	journal	January 2016
Analysis of the Ensemble Kalman Filter for Inverse Problems Schillings, Claudia; Stuart, Andrew M. SIAM Journal on Numerical Analysis, Vol. 55, Issue 3 https://doi.org/10.1137/16M105959X	journal	January 2017
Importance Sampling and Necessary Sample Size: An Information Theory Approach Sanz-Alonso, Daniel SIAM/ASA Journal on Uncertainty Quantification, Vol. 6, Issue 2 https://doi.org/10.1137/16M1093549	journal	January 2018
A Strongly Convergent Numerical Scheme from Ensemble Kalman Inversion Blömker, Dirk; Schillings, Claudia; Wacker, Philipp SIAM Journal on Numerical Analysis, Vol. 56, Issue 4 https://doi.org/10.1137/17M1132367	journal	January 2018
Interacting Langevin Diffusions: Gradient Structure and Ensemble Kalman Sampler Garbuno-Inigo, Alfredo; Hoffmann, Franca; Li, Wuchen SIAM Journal on Applied Dynamical Systems, Vol. 19, Issue 1 https://doi.org/10.1137/19M1251655	journal	January 2020
Autodifferentiable Ensemble Kalman Filters Chen, Yuming; Sanz-Alonso, Daniel; Willett, Rebecca SIAM Journal on Mathematics of Data Science, Vol. 4, Issue 2 https://doi.org/10.1137/21M1434477	journal	June 2022
Assimilation of Standard and Targeted Observations within the Unstable Subspace of the Observation–Analysis–Forecast Cycle System Trevisan, Anna; Uboldi, Francesco Journal of the Atmospheric Sciences, Vol. 61, Issue 1 https://doi.org/10.1175/1520-0469(2004)061<0103:AOSATO>2.0.CO;2	journal	January 2004
Methods for Ensemble Prediction Houtekamer, P. L.; Derome, Jacques Monthly Weather Review, Vol. 123, Issue 7 https://doi.org/10.1175/1520-0493(1995)123<2181:MFEP>2.0.CO;2	journal	July 1995
Assimilation of Geosat Altimeter Data for the Agulhas Current Using the Ensemble Kalman Filter with a Quasigeostrophic Model Evensen, Geir; van Leeuwen, Peter Jan Monthly Weather Review, Vol. 124, Issue 1 https://doi.org/10.1175/1520-0493(1996)124<0085:AOGADF>2.0.CO;2	journal	January 1996
Data Assimilation Using an Ensemble Kalman Filter Technique Houtekamer, P. L.; Mitchell, Herschel L. Monthly Weather Review, Vol. 126, Issue 3 https://doi.org/10.1175/1520-0493(1998)126<0796:DAUAEK>2.0.CO;2	journal	March 1998
Analysis Scheme in the Ensemble Kalman Filter Burgers, Gerrit; Jan van Leeuwen, Peter; Evensen, Geir Monthly Weather Review, Vol. 126, Issue 6 https://doi.org/10.1175/1520-0493(1998)126<1719:ASITEK>2.0.CO;2	journal	June 1998
A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation Houtekamer, P. L.; Mitchell, Herschel L. Monthly Weather Review, Vol. 129, Issue 1 https://doi.org/10.1175/1520-0493(2001)129<0123:ASEKFF>2.0.CO;2	journal	January 2001
Adaptive Sampling with the Ensemble Transform Kalman Filter. Part I: Theoretical Aspects Bishop, Craig H.; Etherton, Brian J.; Majumdar, Sharanya J. Monthly Weather Review, Vol. 129, Issue 3 https://doi.org/10.1175/1520-0493(2001)129<0420:ASWTET>2.0.CO;2	journal	March 2001
An Ensemble Adjustment Kalman Filter for Data Assimilation Anderson, Jeffrey L. Monthly Weather Review, Vol. 129, Issue 12 https://doi.org/10.1175/1520-0493(2001)129<2884:AEAKFF>2.0.CO;2	journal	December 2001
Ensemble Square Root Filters* Tippett, Michael K.; Anderson, Jeffrey L.; Bishop, Craig H. Monthly Weather Review, Vol. 131, Issue 7 https://doi.org/10.1175/1520-0493(2003)131<1485:ESRF>2.0.CO;2	journal	July 2003
Implications of Stochastic and Deterministic Filters as Ensemble-Based Data Assimilation Methods in Varying Regimes of Error Growth Lawson, W. Gregory; Hansen, James A. Monthly Weather Review, Vol. 132, Issue 8 https://doi.org/10.1175/1520-0493(2004)132<1966:IOSADF>2.0.CO;2	journal	August 2004
Obstacles to High-Dimensional Particle Filtering Snyder, Chris; Bengtsson, Thomas; Bickel, Peter Monthly Weather Review, Vol. 136, Issue 12 https://doi.org/10.1175/2008MWR2529.1	journal	December 2008
Performance Bounds for Particle Filters Using the Optimal Proposal Snyder, Chris; Bengtsson, Thomas; Morzfeld, Mathias Monthly Weather Review, Vol. 143, Issue 11 https://doi.org/10.1175/MWR-D-15-0144.1	journal	November 2015
Review of the Ensemble Kalman Filter for Atmospheric Data Assimilation Houtekamer, P. L.; Zhang, Fuqing Monthly Weather Review, Vol. 144, Issue 12 https://doi.org/10.1175/MWR-D-15-0440.1	journal	December 2016
The Ensemble Kalman filter: a signal processing perspective Roth, Michael; Hendeby, Gustaf; Fritsche, Carsten EURASIP Journal on Advances in Signal Processing, Vol. 2017, Issue 1 https://doi.org/10.1186/s13634-017-0492-x	journal	August 2017
Regularized estimation of large covariance matrices Bickel, Peter J.; Levina, Elizaveta The Annals of Statistics, Vol. 36, Issue 1 https://doi.org/10.1214/009053607000000758	journal	February 2008
Covariance regularization by thresholding Bickel, Peter J.; Levina, Elizaveta The Annals of Statistics, Vol. 36, Issue 6 https://doi.org/10.1214/08-AOS600	journal	December 2008
Optimal rates of convergence for sparse covariance matrix estimation Cai, T. Tony; Zhou, Harrison H. The Annals of Statistics, Vol. 40, Issue 5 https://doi.org/10.1214/12-AOS998	journal	October 2012
Adaptive covariance matrix estimation through block thresholding Cai, T. Tony; Yuan, Ming The Annals of Statistics, Vol. 40, Issue 4 https://doi.org/10.1214/12-AOS999	journal	August 2012
On the stability and the uniform propagation of chaos properties of Ensemble Kalman–Bucy filters Del Moral, P.; Tugaut, J. The Annals of Applied Probability, Vol. 28, Issue 2 https://doi.org/10.1214/17-AAP1317	journal	April 2018
The sample size required in importance sampling Chatterjee, Sourav; Diaconis, Persi The Annals of Applied Probability, Vol. 28, Issue 2 https://doi.org/10.1214/17-AAP1326	journal	April 2018
Importance Sampling: Intrinsic Dimension and Computational Cost Agapiou, S.; Papaspiliopoulos, O.; Sanz-Alonso, D. Statistical Science, Vol. 32, Issue 3 https://doi.org/10.1214/17-STS611	journal	August 2017
Curse-of-dimensionality revisited: Collapse of the particle filter in very large scale systems Bengtsson, Thomas; Bickel, Peter; Li, Bo Institute of Mathematical Statistics Collections https://doi.org/10.1214/193940307000000518	book	January 2008
Tail bounds via generic chaining Dirksen, Sjoerd Electronic Journal of Probability, Vol. 20, Issue none https://doi.org/10.1214/EJP.v20-3760	journal	January 2015
The impact of ensemble filter definition on the assimilation of temperature profiles in the tropical Pacific Leeuwenburgh, O.; Evensen, G.; Bertino, L. Quarterly Journal of the Royal Meteorological Society, Vol. 131, Issue 613 https://doi.org/10.1256/qj.05.90	journal	October 2005
Ensemble Kalman filter for neural network-based one-shot inversion Guth, Philipp A.; Schillings, Claudia; Weissmann, Simon Optimization and Control for Partial Differential Equations https://doi.org/10.1515/9783110695984-014	book	March 2022
An Introduction to Matrix Concentration Inequalities Tropp, Joel A. https://doi.org/10.1561/9781601988393	book	January 2015
An Iterative Ensemble Kalman Filter for Multiphase Fluid Flow Data Assimilation Gu, Yaqing; Oliver, Dean S. SPE Journal, Vol. 12, Issue 04 https://doi.org/10.2118/108438-PA	journal	November 2007
Concentration inequalities and moment bounds for sample covariance operators Koltchinskii, Vladimir; Lounici, Karim Bernoulli, Vol. 23, Issue 1 https://doi.org/10.3150/15-BEJ730	journal	February 2017
On the Efficiency of Covariance Localisation of the Ensemble Kalman Filter Using Augmented Ensembles Farchi, Alban; Bocquet, Marc Frontiers in Applied Mathematics and Statistics, Vol. 5 https://doi.org/10.3389/fams.2019.00003	journal	February 2019
Bayesian Update with Importance Sampling: Required Sample Size Sanz-Alonso, Daniel; Wang, Zijian Entropy, Vol. 23, Issue 1 https://doi.org/10.3390/e23010022	journal	December 2020
A local ensemble Kalman filter for atmospheric data assimilation Ott, Edward; Hunt, Brian R.; Szunyogh, Istvan Tellus A: Dynamic Meteorology and Oceanography, Vol. 56, Issue 5 https://doi.org/10.3402/tellusa.v56i5.14462	journal	January 2004
Mean field limit of Ensemble Square Root filters - discrete and continuous time Lange, Theresa; Stannat, Wilhelm Foundations of Data Science, Vol. 3, Issue 3 https://doi.org/10.3934/fods.2021003	journal	January 2021
Iterative ensemble Kalman methods: A unified perspective with some new variants Chada, Neil K.; Chen, Yuming; Sanz-Alonso, Daniel Foundations of Data Science, Vol. 3, Issue 3 https://doi.org/10.3934/fods.2021011	journal	January 2021
Hierarchical ensemble Kalman methods with sparsity-promoting generalized gamma hyperpriors Kim, Hwanwoo; Sanz-Alonso, Daniel Foundations of Data Science, Vol. 5, Issue 3 https://doi.org/10.3934/fods.2023003	journal	January 2023
Kinetic methods for inverse problems Herty, Michael; Visconti, Giuseppe Kinetic & Related Models, Vol. 12, Issue 5 https://doi.org/10.3934/krm.2019042	journal	January 2019
Iterative Forms of the Ensemble Kalman Filter C. Reynolds, A.; Zafari, M.; Li, G. ECMOR X - 10th European Conference on the Mathematics of Oil Recovery https://doi.org/10.3997/2214-4609.201402496	conference	January 2006
Catastrophic filter divergence in filtering nonlinear dissipative systems Harlim, John; Majda, Andrew J. Communications in Mathematical Sciences, Vol. 8, Issue 1 https://doi.org/10.4310/CMS.2010.v8.n1.a3	journal	January 2010
A mechanism for catastrophic filter divergence in data assimilation for sparse observation networks Gottwald, G. A.; Majda, A. J. Nonlinear Processes in Geophysics, Vol. 20, Issue 5 https://doi.org/10.5194/npg-20-705-2013	journal	September 2013

Similar Records

Covariance operator estimation: Sparsity, lengthscale, and ensemble Kalman filters

Journal Article · 2025 · Bernoulli · OSTI ID:3376999

Al-Ghattas, Omar; Chen, Jiaheng; Sanz-Alonso, Daniel; +1 more

Hierarchical ensemble Kalman methods with sparsity-promoting generalized gamma hyperpriors

Journal Article · 2023 · Foundations of Data Science · OSTI ID:3377005

Kim, Hwanwoo; Sanz-Alonso, Daniel; Strang, Alexander

Penalized ensemble Kalman filters for high dimensional non-linear systems

Journal Article · 2021 · PLoS ONE · OSTI ID:1771364

Hou, Elizabeth; Lawrence, Earl; Hero, Alfred O.; +1 more

Related Subjects

Mathematics

Title: Non-asymptotic analysis of ensemble Kalman updates: effective dimension and localization

Citation Formats

References (89)

Similar Records

Related Subjects