skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Massively parallel solvers for elliptic partial differential equations in numerical weather and climate prediction

Journal Article · · Quarterly Journal of the Royal Meteorological Society
DOI:https://doi.org/10.1002/qj.2327· OSTI ID:1565272

The demand for substantial increases in the spatial resolution of global weather and climate prediction models makes it essential to use numerically efficient and highly scalable algorithms to solve the equations of large-scale atmospheric fluid dynamics. For stability and efficiency reasons, several of the operational forecasting centres, in particular the Met Office and the European Centre for Medium-Range Weather Forecasts (ECMWF) in the UK, use semi-implicit semi-Lagrangian time-stepping in the dynamical core of the model. The additional burden with this method is that a three-dimensional elliptic partial differential equation (PDE) for the pressure correction has to be solved at every model time step and this often constitutes a significant proportion of the time spent in the dynamical core. In global models, this PDE must be solved in a thin spherical shell. To run within tight operational time-scales, the solver has to be parallelized and there seems to be a (perceived) misconception that elliptic solvers do not scale to large processor counts and hence implicit time-stepping cannot be used in very high-resolution global models. After reviewing several methods for solving the elliptic PDE for the pressure correction and their application in atmospheric models, we demonstrate the performance and very good scalability of Krylov subspace solvers and multigrid algorithms for a representative model equation with more than 1010 unknowns on 65 536 cores on the High-End Computing Terascale Resource (HECToR), the UK's national supercomputer. Here, we tested and optimized solvers from two existing numerical libraries (the Distributed and Unified Numerics Environment (DUNE) and Parallel High Performance Preconditioners (hypre)) and implemented both a conjugate gradient solver and a geometric multigrid algorithm based on a tensor-product approach, which exploits the strong vertical anisotropy of the discretized equation. We study both weak and strong scalability and compare the absolute solution times for all methods; in contrast to one-level methods, the multigrid solver is robust with respect to parameter variations.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Organization:
USDOE Office of Science (SC)
Grant/Contract Number:
NE/J005576/1
OSTI ID:
1565272
Journal Information:
Quarterly Journal of the Royal Meteorological Society, Vol. 140, Issue 685; ISSN 0035-9009
Publisher:
Royal Meteorological SocietyCopyright Statement
Country of Publication:
United States
Language:
English
Citation Metrics:
Cited by: 40 works
Citation information provided by
Web of Science

References (42)

UG - A flexible software toolbox for solving partial differential equations journal July 1997
A parallel, load-balanced MHD code on locally-adapted, unstructured grids in 3d journal September 2004
Conservative Finite-Difference Approximations of the Primitive Equations on Quasi-Uniform Spherical Grids journal February 1972
A Multigrid Tutorial, Second Edition book January 2000
Massively parallel implementation of the mesoscale compressible community model journal December 1997
Matrix-free GPU implementation of a preconditioned conjugate gradient solver for anisotropic elliptic PDEs journal April 2013
The Design and Implementation of hypre, a Library of Parallel High Performance Preconditioners book January 2006
Preconditioned Conjugate-Residual Solvers for Helmholtz Equations in Nonhydrostatic Models journal April 1997
A generic grid interface for parallel and adaptive scientific computing. Part II: implementation and tests in DUNE journal June 2008
On the generic parallelisation of iterative solvers for the finite element method journal January 2008
Recent enhancements in MUDPACK, a multigrid software package for elliptic partial differential equations journal May 1991
The Numerical Solution of Parabolic and Elliptic Differential Equations journal March 1955
Towards petascale simulation of atmospheric circulations with soundproof equations journal September 2011
Horizontal grids for global weather and climate prediction models: a review journal November 2011
Algebraic multigrid for discontinuous Galerkin discretizations of heterogeneous elliptic problems: AMG4DG journal February 2012
A Multigrid Solver for the Helmholtz Equation on a Semiregular Grid on the Sphere journal March 1991
Modified multigrid for 3D elliptic equations with cross-derivatives journal June 2001
Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods book January 1994
A stable numerical integration scheme for the primitive meteorological equations journal March 1981
Integration of the Shallow Water Equations on the Sphere Using a Vector Semi-Lagrangian Scheme with a Multigrid Solver journal August 1990
A comparison of parallel multigrid and a fast Fourier transform algorithm for the solution of the Helmholtz equation in numerical weather prediction journal January 1997
A Semi-implicit Send-Lagrangian Fully Compressible Regional Forecast Model journal October 1990
Application of the Multigrid Method and a Flexible Hybrid Coordinate in a Nonhydrostatic Model journal November 2001
Iterative Methods for Sparse Linear Systems book January 2003
A Fast Spherical Harmonics Transform for Global NWP and Climate Models journal October 2013
Scaling Hypre’s Multigrid Solvers to 100,000 Cores book January 2012
A new dynamical core for the Met Office's global and regional modelling of the atmosphere journal April 2005
Review of numerical methods for nonhydrostatic weather prediction models journal January 2003
A generic grid interface for parallel and adaptive scientific computing. Part I: abstract framework journal April 2008
A global shallow-water model using an expanded spherical cube: Gnomonic versus conformal coordinates journal April 1996
mudpack: Multigrid portable fortran software for the efficient solution of linear elliptic partial differential equations journal November 1989
A Semi-Implicit Scheme for grid Point Atmospheric Models of the Primitive Equations journal January 1971
A robust numerical method for the potential vorticity based control variable transform in variational data assimilation
  • Buckeridge, S.; Cullen, M. J. P.; Scheichl, R.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 137, Issue 657 https://doi.org/10.1002/qj.826
journal April 2011
Multigrid Methods for Elliptic Problems: A Review journal May 1986
EULAG, a computational model for multiscale flows journal October 2008
On Forward-in-Time Differencing for Fluids: an Eulerian/Semi-Lagrangian Non-Hydrostatic Model for Stratified Flows journal January 1997
A general method for modeling on irregular grids journal December 2010
Methods of conjugate gradients for solving linear systems journal December 1952
Comparative Test of Direct and Iterative Methods for Solving Helmholtz-Type Equations journal March 1973
Applications of Multigrid Software in the Atmospheric Sciences journal July 1992
Parallel geometric multigrid for global weather prediction journal January 2010
An inherently mass-conserving semi-implicit semi-Lagrangian discretization of the deep-atmosphere global non-hydrostatic equations journal December 2013

Cited By (5)

An efficient exponential time integration method for the numerical solution of the shallow water equations on the sphere journal October 2016
ADI type preconditioners for the steady state inhomogeneous Vlasov equation text January 2016
Vorticity-divergence semi-Lagrangian global atmospheric model SL-AV20: dynamical core journal May 2017
P-CSI v1.0, an accelerated barotropic solver for the high-resolution ocean model component in the Community Earth System Model v2.0 journal July 2016
A mimetic, semi-implicit, forward-in-time, finite volume shallow water model: comparison of hexagonal–icosahedral and cubed-sphere grids journal January 2014

Figures / Tables (14)