skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: PyDA: A Hands-On Introduction to Dynamical Data Assimilation with Python

Abstract

Dynamic data assimilation offers a suite of algorithms that merge measurement data with numerical simulations to predict accurate state trajectories. Meteorological centers rely heavily on data assimilation to achieve trustworthy weather forecast. With the advance in measurement systems, as well as the reduction in sensor prices, data assimilation (DA) techniques are applicable to various fields, other than meteorology. However, beginners usually face hardships digesting the core ideas from the available sophisticated resources requiring a steep learning curve. In this tutorial, we lay out the mathematical principles behind DA with easy-to-follow Python module implementations so that this group of newcomers can quickly feel the essence of DA algorithms. We explore a series of common variational, and sequential techniques, and highlight major differences and potential extensions. We demonstrate the presented approaches using an array of fluid flow applications with varying levels of complexity.

Authors:
ORCiD logo [1];  [1]; ORCiD logo [1]
  1. Oklahoma State Univ., Stillwater, OK (United States)
Publication Date:
Research Org.:
Oklahoma State Univ., Stillwater, OK (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1737509
Grant/Contract Number:  
SC0019290
Resource Type:
Accepted Manuscript
Journal Name:
Fluids
Additional Journal Information:
Journal Volume: 5; Journal Issue: 4; Journal ID: ISSN 2311-5521
Publisher:
MDPI
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; data assimilation; variational and sequential methods; Kalman filtering; forward sensitivity; measurements fusion

Citation Formats

Ahmed, Shady E., Pawar, Suraj, and San, Omer. PyDA: A Hands-On Introduction to Dynamical Data Assimilation with Python. United States: N. p., 2020. Web. https://doi.org/10.3390/fluids5040225.
Ahmed, Shady E., Pawar, Suraj, & San, Omer. PyDA: A Hands-On Introduction to Dynamical Data Assimilation with Python. United States. https://doi.org/10.3390/fluids5040225
Ahmed, Shady E., Pawar, Suraj, and San, Omer. Sun . "PyDA: A Hands-On Introduction to Dynamical Data Assimilation with Python". United States. https://doi.org/10.3390/fluids5040225. https://www.osti.gov/servlets/purl/1737509.
@article{osti_1737509,
title = {PyDA: A Hands-On Introduction to Dynamical Data Assimilation with Python},
author = {Ahmed, Shady E. and Pawar, Suraj and San, Omer},
abstractNote = {Dynamic data assimilation offers a suite of algorithms that merge measurement data with numerical simulations to predict accurate state trajectories. Meteorological centers rely heavily on data assimilation to achieve trustworthy weather forecast. With the advance in measurement systems, as well as the reduction in sensor prices, data assimilation (DA) techniques are applicable to various fields, other than meteorology. However, beginners usually face hardships digesting the core ideas from the available sophisticated resources requiring a steep learning curve. In this tutorial, we lay out the mathematical principles behind DA with easy-to-follow Python module implementations so that this group of newcomers can quickly feel the essence of DA algorithms. We explore a series of common variational, and sequential techniques, and highlight major differences and potential extensions. We demonstrate the presented approaches using an array of fluid flow applications with varying levels of complexity.},
doi = {10.3390/fluids5040225},
journal = {Fluids},
number = 4,
volume = 5,
place = {United States},
year = {2020},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

The Maximum Likelihood Ensemble Filter as a non‐differentiable minimization algorithm
journal, April 2008

  • Zupanski, Milija; Navon, I. Michael; Zupanski, Dusanka
  • Quarterly Journal of the Royal Meteorological Society, Vol. 134, Issue 633
  • DOI: 10.1002/qj.251

Optimal sensor placement for variational data assimilation of unsteady flows past a rotationally oscillating cylinder
journal, June 2017

  • Mons, Vincent; Chassaing, Jean-Camille; Sagaut, Pierre
  • Journal of Fluid Mechanics, Vol. 823
  • DOI: 10.1017/jfm.2017.313

Comparison of Hybrid-4DEnVar and Hybrid-4DVar Data Assimilation Methods for Global NWP
journal, January 2015

  • Lorenc, Andrew C.; Bowler, Neill E.; Clayton, Adam M.
  • Monthly Weather Review, Vol. 143, Issue 1
  • DOI: 10.1175/MWR-D-14-00195.1

Why does 4D-Var beat 3D-Var?
journal, October 2005

  • Lorenc, Andrew C.; Rawlins, F.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 131, Issue 613
  • DOI: 10.1256/qj.05.85

Sequential data assimilation with a nonlinear quasi-geostrophic model using Monte Carlo methods to forecast error statistics
journal, January 1994

  • Evensen, Geir
  • Journal of Geophysical Research, Vol. 99, Issue C5
  • DOI: 10.1029/94JC00572

Analysis methods for numerical weather prediction
journal, October 1986

  • Lorenc, A. C.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 112, Issue 474
  • DOI: 10.1002/qj.49711247414

Four-dimensional ensemble-variational data assimilation for global deterministic weather prediction
journal, January 2013

  • Buehner, M.; Morneau, J.; Charette, C.
  • Nonlinear Processes in Geophysics, Vol. 20, Issue 5
  • DOI: 10.5194/npg-20-669-2013

Data Assimilation Using an Ensemble Kalman Filter Technique
journal, March 1998


4D-variational data assimilation with an adjoint air quality model for emission analysis
journal, September 2000


A deterministic formulation of the ensemble Kalman filter: an alternative to ensemble square root filters
journal, January 2008


Sampling the posterior: An approach to non-Gaussian data assimilation
journal, June 2007


The Ensemble Kalman Filter: theoretical formulation and practical implementation
journal, November 2003


A Sequential Ensemble Kalman Filter for Atmospheric Data Assimilation
journal, January 2001


Forward Sensitivity Approach to Dynamic Data Assimilation
journal, January 2010


Four-Gyre Circulation in a Barotropic Model with Double-Gyre Wind Forcing
journal, June 2000


A Hybrid ETKF–3DVAR Data Assimilation Scheme for the WRF Model. Part I: Observing System Simulation Experiment
journal, December 2008

  • Wang, Xuguang; Barker, Dale M.; Snyder, Chris
  • Monthly Weather Review, Vol. 136, Issue 12
  • DOI: 10.1175/2008MWR2444.1

Optimal placement of mobile sensors for data assimilations
journal, February 2012


Analysis Scheme in the Ensemble Kalman Filter
journal, June 1998


Beyond Gaussian Statistical Modeling in Geophysical Data Assimilation
journal, August 2010

  • Bocquet, Marc; Pires, Carlos A.; Wu, Lin
  • Monthly Weather Review, Vol. 138, Issue 8
  • DOI: 10.1175/2010MWR3164.1

DATeS: a highly extensible data assimilation testing suite v1.0
journal, January 2019


State-of-the-art stochastic data assimilation methods for high-dimensional non-Gaussian problems
journal, January 2018

  • Vetra-Carvalho, Sanita; van Leeuwen, Peter Jan; Nerger, Lars
  • Tellus A: Dynamic Meteorology and Oceanography, Vol. 70, Issue 1
  • DOI: 10.1080/16000870.2018.1445364

A Monte Carlo Implementation of the Nonlinear Filtering Problem to Produce Ensemble Assimilations and Forecasts
journal, December 1999


Approximate deconvolution large eddy simulation of a barotropic ocean circulation model
journal, January 2011


The National Meteorological Center's Spectral Statistical-Interpolation Analysis System
journal, August 1992


Diffusion-Induced Chaos in Reaction Systems
journal, January 1978

  • Kuramoto, Yoshiki
  • Progress of Theoretical Physics Supplement, Vol. 64
  • DOI: 10.1143/PTPS.64.346

Dual formulation of four-dimensional variational assimilation
journal, October 1997

  • Courtier, Philippe
  • Quarterly Journal of the Royal Meteorological Society, Vol. 123, Issue 544
  • DOI: 10.1002/qj.49712354414

Cluster Sampling Filters for Non-Gaussian Data Assimilation
journal, May 2018


Extension of 3DVAR to 4DVAR: Implementation of 4DVAR at the Meteorological Service of Canada
journal, June 2007

  • Gauthier, Pierre; Tanguay, Monique; Laroche, Stéphane
  • Monthly Weather Review, Vol. 135, Issue 6
  • DOI: 10.1175/MWR3394.1

The ECMWF operational implementation of four-dimensional variational assimilation. I: Experimental results with simplified physics
journal, April 2000

  • Rabier, F.; Järvinen, H.; Klinker, E.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 126, Issue 564
  • DOI: 10.1002/qj.49712656415

Ensemble Data Assimilation without Perturbed Observations
journal, July 2002


Maximum Likelihood Ensemble Filter: Theoretical Aspects
journal, June 2005


Second-Order Information in Data Assimilation*
journal, March 2002


4DEnVar: link with 4D state formulation of variational assimilation and different possible implementations: 4DEnVar: State Formulation of 4D-Var and Possible Implementations
journal, April 2014

  • Desroziers, Gérald; Camino, Jean-Thomas; Berre, Loïk
  • Quarterly Journal of the Royal Meteorological Society, Vol. 140, Issue 684
  • DOI: 10.1002/qj.2325

Ensemble Square Root Filters*
journal, July 2003


Ensemble Kalman filtering
journal, October 2005

  • Houtekamer, P. L.; Mitchell, Herschel L.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 131, Issue 613
  • DOI: 10.1256/qj.05.135

Particle filters for high‐dimensional geoscience applications: A review
journal, May 2019

  • Leeuwen, Peter Jan; Künsch, Hans R.; Nerger, Lars
  • Quarterly Journal of the Royal Meteorological Society, Vol. 145, Issue 723
  • DOI: 10.1002/qj.3551

A strategy for operational implementation of 4D-Var, using an incremental approach
journal, July 1994

  • Courtier, P.; Thépaut, J. -N.; Hollingsworth, A.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 120, Issue 519
  • DOI: 10.1002/qj.49712051912

Reduced-rank Kalman filters applied to an idealized model of the wind-driven ocean circulation
journal, January 2003