Sampling, feasibility, and priors in data assimilation
Abstract
Importance sampling algorithms are discussed in detail, with an emphasis on implicit sampling, and applied to data assimilation via particle filters. Implicit sampling makes it possible to use the data to find high-probability samples at relatively low cost, making the assimilation more efficient. A new analysis of the feasibility of data assimilation is presented, showing in detail why feasibility depends on the Frobenius norm of the covariance matrix of the noise and not on the number of variables. A discussion of the convergence of particular particle filters follows. A major open problem in numerical data assimilation is the determination of appropriate priors, a progress report on recent work on this problem is given. The analysis highlights the need for a careful attention both to the data and to the physics in data assimilation problems.
- Authors:
-
- Univ. of Kansas, Lawrence, KS (United States)
- Univ. of Arizona, Tucson, AZ (United States)
- Oregon State Univ., Corvallis, OR (United States)
- Univ. of California, Berkeley, CA (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
- Publication Date:
- Research Org.:
- Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- OSTI Identifier:
- 1439205
- Alternate Identifier(s):
- OSTI ID: 1379537
- Grant/Contract Number:
- AC02-05CH11231
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Discrete and Continuous Dynamical Systems
- Additional Journal Information:
- Journal Volume: 36; Journal Issue: 8; Journal ID: ISSN 1078-0947
- Publisher:
- American Institute of Mathematical Sciences
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
Citation Formats
Tu, Xuemin, Morzfeld, Matthias, Miller, Robert N., Lu, Fei, and Chorin, Alexandre J. Sampling, feasibility, and priors in data assimilation. United States: N. p., 2016.
Web. doi:10.3934/dcds.2016.36.4227.
Tu, Xuemin, Morzfeld, Matthias, Miller, Robert N., Lu, Fei, & Chorin, Alexandre J. Sampling, feasibility, and priors in data assimilation. United States. https://doi.org/10.3934/dcds.2016.36.4227
Tu, Xuemin, Morzfeld, Matthias, Miller, Robert N., Lu, Fei, and Chorin, Alexandre J. Tue .
"Sampling, feasibility, and priors in data assimilation". United States. https://doi.org/10.3934/dcds.2016.36.4227. https://www.osti.gov/servlets/purl/1439205.
@article{osti_1439205,
title = {Sampling, feasibility, and priors in data assimilation},
author = {Tu, Xuemin and Morzfeld, Matthias and Miller, Robert N. and Lu, Fei and Chorin, Alexandre J.},
abstractNote = {Importance sampling algorithms are discussed in detail, with an emphasis on implicit sampling, and applied to data assimilation via particle filters. Implicit sampling makes it possible to use the data to find high-probability samples at relatively low cost, making the assimilation more efficient. A new analysis of the feasibility of data assimilation is presented, showing in detail why feasibility depends on the Frobenius norm of the covariance matrix of the noise and not on the number of variables. A discussion of the convergence of particular particle filters follows. A major open problem in numerical data assimilation is the determination of appropriate priors, a progress report on recent work on this problem is given. The analysis highlights the need for a careful attention both to the data and to the physics in data assimilation problems.},
doi = {10.3934/dcds.2016.36.4227},
journal = {Discrete and Continuous Dynamical Systems},
number = 8,
volume = 36,
place = {United States},
year = {Tue Mar 01 00:00:00 EST 2016},
month = {Tue Mar 01 00:00:00 EST 2016}
}
Works referenced in this record:
An exploration of the equivalent weights particle filter
journal, August 2012
- Ades, M.; van Leeuwen, P. J.
- Quarterly Journal of the Royal Meteorological Society, Vol. 139, Issue 672
A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking
journal, January 2002
- Arulampalam, M. S.; Maskell, S.; Gordon, N.
- IEEE Transactions on Signal Processing, Vol. 50, Issue 2
Implicit Particle Methods and Their Connection with Variational Data Assimilation
journal, June 2013
- Atkins, Ethan; Morzfeld, Matthias; Chorin, Alexandre J.
- Monthly Weather Review, Vol. 141, Issue 6
Linear theory for filtering nonlinear multiscale systems with model error
journal, July 2014
- Berry, Tyrus; Harlim, John
- Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 470, Issue 2167
Beyond Gaussian Statistical Modeling in Geophysical Data Assimilation
journal, August 2010
- Bocquet, Marc; Pires, Carlos A.; Wu, Lin
- Monthly Weather Review, Vol. 138, Issue 8
Estimation of Tropical Sea Level Anomaly by an Improved Kalman Filter
journal, July 1996
- Chan, Ngai Hang; Kadane, Joseph B.; Miller, Robert N.
- Journal of Physical Oceanography, Vol. 26, Issue 7
Estimating the uncertainty in underresolved nonlinear dynamics
journal, October 2013
- Chorin, Alexandre J.; Hald, Ole H.
- Mathematics and Mechanics of Solids, Vol. 19, Issue 1
Discrete approach to stochastic parametrization and dimension reduction in nonlinear dynamics
journal, July 2015
- Chorin, Alexandre J.; Lu, Fei
- Proceedings of the National Academy of Sciences, Vol. 112, Issue 32
Conditions for successful data assimilation: CONDITIONS FOR DATA ASSIMILATION
journal, October 2013
- Chorin, Alexandre J.; Morzfeld, Matthias
- Journal of Geophysical Research: Atmospheres, Vol. 118, Issue 20
Implicit particle filters for data assimilation
journal, January 2010
- Chorin, Alexandre; Morzfeld, Matthias; Tu, Xuemin
- Communications in Applied Mathematics and Computational Science, Vol. 5, Issue 2
Implicit sampling for particle filters
journal, September 2009
- Chorin, A. J.; Tu, X.
- Proceedings of the National Academy of Sciences, Vol. 106, Issue 41
Subgrid-Scale Parameterization with Conditional Markov Chains
journal, August 2008
- Crommelin, Daan; Vanden-Eijnden, Eric
- Journal of the Atmospheric Sciences, Vol. 65, Issue 8
A computational strategy for multiscale systems with applications to Lorenz 96 model
journal, November 2004
- Fatkullin, Ibrahim; Vanden-Eijnden, Eric
- Journal of Computational Physics, Vol. 200, Issue 2
An Introduction to Data Assimilation and Predictability in Geomagnetism
journal, August 2010
- Fournier, Alexandre; Hulot, Gauthier; Jault, Dominique
- Space Science Reviews, Vol. 155, Issue 1-4
Small-Noise Analysis and Symmetrization of Implicit Monte Carlo Samplers
journal, July 2015
- Goodman, Jonathan; Lin, Kevin K.; Morzfeld, Matthias
- Communications on Pure and Applied Mathematics, Vol. 69, Issue 10
Ensemble samplers with affine invariance
journal, January 2010
- Goodman, Jonathan; Weare, Jonathan
- Communications in Applied Mathematics and Computational Science, Vol. 5, Issue 1
A New Approach to Linear Filtering and Prediction Problems
journal, March 1960
- Kalman, R. E.
- Journal of Basic Engineering, Vol. 82, Issue 1
Blind Deconvolution via Sequential Imputations
journal, June 1995
- Liu, Jun S.; Chen, Rong
- Journal of the American Statistical Association, Vol. 90, Issue 430
Deterministic Nonperiodic Flow
journal, March 1963
- Lorenz, Edward N.
- Journal of the Atmospheric Sciences, Vol. 20, Issue 2
Physics constrained nonlinear regression models for time series
journal, November 2012
- Majda, Andrew J.; Harlim, John
- Nonlinearity, Vol. 26, Issue 1
Application of the implicit particle filter to a model of nearshore circulation
journal, April 2014
- Miller, R. N.; Ehret, L. L.
- Journal of Geophysical Research: Oceans, Vol. 119, Issue 4
Advanced Data Assimilation in Strongly Nonlinear Dynamical Systems
journal, April 1994
- Miller, Robert N.; Ghil, Michael; Gauthiez, François
- Journal of the Atmospheric Sciences, Vol. 51, Issue 8
A random map implementation of implicit filters
journal, February 2012
- Morzfeld, Matthias; Tu, Xuemin; Atkins, Ethan
- Journal of Computational Physics, Vol. 231, Issue 4
Parameter estimation by implicit sampling
journal, January 2015
- Morzfeld, Matthias; Tu, Xuemin; Wilkening, Jon
- Communications in Applied Mathematics and Computational Science, Vol. 10, Issue 2
Bayesian inference with optimal maps
journal, October 2012
- El Moselhy, Tarek A.; Marzouk, Youssef M.
- Journal of Computational Physics, Vol. 231, Issue 23
Probabilistic Aspects of Error Growth In Atmospheric Dynamics
journal, April 1992
- Nicolis, C.
- Quarterly Journal of the Royal Meteorological Society, Vol. 118, Issue 505
Parameter estimation for partially observed hypoelliptic diffusions
journal, January 2009
- Pokern, Yvo; Stuart, Andrew M.; Wiberg, Petter
- Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 71, Issue 1
A contrast estimator for completely or partially observed hypoelliptic diffusion
journal, July 2012
- Samson, Adeline; Thieullen, Michèle
- Stochastic Processes and their Applications, Vol. 122, Issue 7
Obstacles to High-Dimensional Particle Filtering
journal, December 2008
- Snyder, Chris; Bengtsson, Thomas; Bickel, Peter
- Monthly Weather Review, Vol. 136, Issue 12
Performance Bounds for Particle Filters Using the Optimal Proposal
journal, November 2015
- Snyder, Chris; Bengtsson, Thomas; Morzfeld, Mathias
- Monthly Weather Review, Vol. 143, Issue 11
Ensemble Square Root Filters*
journal, July 2003
- Tippett, Michael K.; Anderson, Jeffrey L.; Bishop, Craig H.
- Monthly Weather Review, Vol. 131, Issue 7
Particle Filtering in Geophysical Systems
journal, December 2009
- van Leeuwen, Peter Jan
- Monthly Weather Review, Vol. 137, Issue 12
Nonlinear data assimilation in geosciences: an extremely efficient particle filter
journal, October 2010
- van Leeuwen, P. J.
- Quarterly Journal of the Royal Meteorological Society, Vol. 136, Issue 653
Rare Event Simulation of Small Noise Diffusions
journal, September 2012
- Vanden-Eijnden, Eric; Weare, Jonathan
- Communications on Pure and Applied Mathematics, Vol. 65, Issue 12
Data Assimilation in the Low Noise Regime with Application to the Kuroshio
journal, June 2013
- Vanden-Eijnden, Eric; Weare, Jonathan
- Monthly Weather Review, Vol. 141, Issue 6
Implicit Estimation of Ecological Model Parameters
journal, January 2013
- Weir, Brad; Miller, Robert N.; Spitz, Yvette H.
- Bulletin of Mathematical Biology, Vol. 75, Issue 2
Effects of stochastic parametrizations in the Lorenz '96 system
journal, January 2005
- Wilks, Daniel S.
- Quarterly Journal of the Royal Meteorological Society, Vol. 131, Issue 606
Works referencing / citing this record:
Joint state-parameter estimation of a nonlinear stochastic energy balance model from sparse noisy data
journal, January 2019
- Lu, Fei; Weitzel, Nils; Monahan, Adam H.
- Nonlinear Processes in Geophysics, Vol. 26, Issue 3
Joint state-parameter estimation of a nonlinear stochastic energy balance model from sparse noisy data
text, January 2019
- Lu, Fei; Weitzel, Nils; Monahan, Adam H.
- arXiv
Joint state-parameter estimation of a nonlinear stochastic energy balance model from sparse noisy data
journal, January 2019
- Lu, Fei; Weitzel, Nils; Monahan, Adam H.
- Nonlinear Processes in Geophysics, Vol. 26, Issue 3