skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors

Abstract

We present the acceleration of an IMplicit-EXplicit (IMEX) nonhydrostatic atmospheric model on manycore processors such as graphic processing units (GPUs) and Intel's Many Integrated Core (MIC) architecture. IMEX time integration methods sidestep the constraint imposed by the Courant-Friedrichs-Lewy condition on explicit methods through corrective implicit solves within each time step. In this work, we implement and evaluate the performance of IMEX on manycore processors relative to explicit methods. Using 3D-IMEX at Courant number C = 15, we obtained a speedup of about 4x relative to an explicit time stepping method run with the maximum allowable C = 1. Moreover, the unconditional stability of IMEX with respect to the fast waves means the speedup can increase significantly with the Courant number as long as the accuracy of the resulting solution is acceptable. We show a speedup of 100x at C = 150 using 1D-IMEX to demonstrate this point. Several improvements on the IMEX procedure were necessary in order to outperform our results with explicit methods: (a) reducing the number of degrees of freedom of the IMEX formulation by forming the Schur complement, (b) formulating a horizontally explicit vertically implicit 1D-IMEX scheme that has a lower workload and better scalability than 3D-IMEX,more » (c) using high-order polynomial preconditioners to reduce the condition number of the resulting system, and (d) using a direct solver for the 1D-IMEX method by performing and storing LU factorizations once to obtain a constant cost for any Courant number. Without all of these improvements, explicit time integration methods turned out to be difficult to beat. We discuss in detail the IMEX infrastructure required for formulating and implementing efficient methods on manycore processors. Several parametric studies are conducted to demonstrate the gain from each of the abovementioned improvements. Lastly, we validate our results with standard benchmark problems in numerical weather prediction and evaluate the performance and scalability of the IMEX method using up to 4192 GPUs and 16 Knights Landing processors.« less

Authors:
 [1];  [1];  [2];  [1];  [1];  [3]
  1. Naval Postgraduate School, Monterey, CA (United States)
  2. Argonne National Lab. (ANL), Argonne, IL (United States)
  3. Virginia Polytechnic Inst. and State Univ. (Virginia Tech), Blacksburg, VA (United States)
Publication Date:
Research Org.:
Oak Ridge National Laboratory, Oak Ridge Leadership Computing Facility (OLCF); Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); US Department of the Navy, Office of Naval Research (ONR)
OSTI Identifier:
1498290
Grant/Contract Number:  
AC02-06CH11357
Resource Type:
Accepted Manuscript
Journal Name:
International Journal of High Performance Computing Applications
Additional Journal Information:
Journal Volume: 33; Journal Issue: 2; Journal ID: ISSN 1094-3420
Publisher:
SAGE
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; IMEX; NUMA; GPU; KNL; manycore; HPC; OCCA; atmospheric model; discontinuous Galerkin; continuous Galerkin

Citation Formats

Abdi, Daniel S., Giraldo, Francis X., Constantinescu, Emil M., Carr, III, Lester E., Wilcox, Lucas C., and Warburton, Timothy C. Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors. United States: N. p., 2017. Web. doi:10.1177/1094342017732395.
Abdi, Daniel S., Giraldo, Francis X., Constantinescu, Emil M., Carr, III, Lester E., Wilcox, Lucas C., & Warburton, Timothy C. Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors. United States. doi:10.1177/1094342017732395.
Abdi, Daniel S., Giraldo, Francis X., Constantinescu, Emil M., Carr, III, Lester E., Wilcox, Lucas C., and Warburton, Timothy C. Tue . "Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors". United States. doi:10.1177/1094342017732395. https://www.osti.gov/servlets/purl/1498290.
@article{osti_1498290,
title = {Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors},
author = {Abdi, Daniel S. and Giraldo, Francis X. and Constantinescu, Emil M. and Carr, III, Lester E. and Wilcox, Lucas C. and Warburton, Timothy C.},
abstractNote = {We present the acceleration of an IMplicit-EXplicit (IMEX) nonhydrostatic atmospheric model on manycore processors such as graphic processing units (GPUs) and Intel's Many Integrated Core (MIC) architecture. IMEX time integration methods sidestep the constraint imposed by the Courant-Friedrichs-Lewy condition on explicit methods through corrective implicit solves within each time step. In this work, we implement and evaluate the performance of IMEX on manycore processors relative to explicit methods. Using 3D-IMEX at Courant number C = 15, we obtained a speedup of about 4x relative to an explicit time stepping method run with the maximum allowable C = 1. Moreover, the unconditional stability of IMEX with respect to the fast waves means the speedup can increase significantly with the Courant number as long as the accuracy of the resulting solution is acceptable. We show a speedup of 100x at C = 150 using 1D-IMEX to demonstrate this point. Several improvements on the IMEX procedure were necessary in order to outperform our results with explicit methods: (a) reducing the number of degrees of freedom of the IMEX formulation by forming the Schur complement, (b) formulating a horizontally explicit vertically implicit 1D-IMEX scheme that has a lower workload and better scalability than 3D-IMEX, (c) using high-order polynomial preconditioners to reduce the condition number of the resulting system, and (d) using a direct solver for the 1D-IMEX method by performing and storing LU factorizations once to obtain a constant cost for any Courant number. Without all of these improvements, explicit time integration methods turned out to be difficult to beat. We discuss in detail the IMEX infrastructure required for formulating and implementing efficient methods on manycore processors. Several parametric studies are conducted to demonstrate the gain from each of the abovementioned improvements. Lastly, we validate our results with standard benchmark problems in numerical weather prediction and evaluate the performance and scalability of the IMEX method using up to 4192 GPUs and 16 Knights Landing processors.},
doi = {10.1177/1094342017732395},
journal = {International Journal of High Performance Computing Applications},
number = 2,
volume = 33,
place = {United States},
year = {2017},
month = {10}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 3 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Comparison of Some Schemes for Nonhydrostatic Models with Orography [山を取り扱える非静水圧モデルの各種計算スキームの比較]
journal, January 1988


A High-Order Accurate Discontinuous Finite Element Method for the Numerical Solution of the Compressible Navier–Stokes Equations
journal, March 1997


An Element-Based Spectrally Optimized Approximate Inverse Preconditioner for the Euler Equations
journal, January 2012

  • Carr III, L. E.; Borges, C. F.; Giraldo, F. X.
  • SIAM Journal on Scientific Computing, Vol. 34, Issue 4
  • DOI: 10.1137/11083229X

Improving the Anelastic Approximation
journal, June 1989


A Conservative Discontinuous Galerkin Semi-Implicit Formulation for the Navier–Stokes Equations in Nonhydrostatic Mesoscale Modeling
journal, January 2009

  • Restelli, Marco; Giraldo, Francis X.
  • SIAM Journal on Scientific Computing, Vol. 31, Issue 3
  • DOI: 10.1137/070708470

A Fast and Accurate Numerical Scheme for the Primitive Equations of the Atmosphere
journal, January 1999


A new dynamical framework of nonhydrostatic global model using the icosahedral grid
journal, June 2004


Accuracy Analysis of a Spectral Element Atmospheric Model Using a Fully Implicit Solution Framework
journal, August 2010

  • Evans, Katherine J.; Taylor, Mark A.; Drake, John B.
  • Monthly Weather Review, Vol. 138, Issue 8
  • DOI: 10.1175/2010MWR3288.1

A Semi-Lagrangian and Semi-Implicit Numerical Integration Scheme for Multilevel Atmospheric Models
journal, March 1985


Semi-Implicit Formulations of the Navier–Stokes Equations: Application to Nonhydrostatic Atmospheric Modeling
journal, January 2010

  • Giraldo, F. X.; Restelli, M.; Läuter, M.
  • SIAM Journal on Scientific Computing, Vol. 32, Issue 6
  • DOI: 10.1137/090775889

An element-based displacement preconditioner for linear elasticity problems
journal, December 2006


Additive Runge–Kutta schemes for convection–diffusion–reaction equations
journal, January 2003


A split explicit integration scheme for numerical weather prediction
journal, July 1978

  • Gadd, A. J.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 104, Issue 441
  • DOI: 10.1002/qj.49710444103

A Scalable Fully Implicit Compressible Euler Solver for Mesoscale Nonhydrostatic Simulation of Atmospheric Flows
journal, January 2014

  • Yang, Chao; Cai, Xiao-Chuan
  • SIAM Journal on Scientific Computing, Vol. 36, Issue 5
  • DOI: 10.1137/130919167

Runge–Kutta IMEX schemes for the Horizontally Explicit/Vertically Implicit (HEVI) solution of wave equations
journal, November 2013


Hybrid Eulerian-Lagrangian Semi-Implicit Time-Integrators
journal, October 2006


Implicit-Explicit Formulations of a Three-Dimensional Nonhydrostatic Unified Model of the Atmosphere (NUMA)
journal, January 2013

  • Giraldo, F. X.; Kelly, J. F.; Constantinescu, E. M.
  • SIAM Journal on Scientific Computing, Vol. 35, Issue 5
  • DOI: 10.1137/120876034

The Influence of Gravity Wave Breaking on the General Circulation of the Middle Atmosphere
journal, October 1983


A peta-scalable CPU-GPU algorithm for global atmospheric simulations
journal, August 2013


Horizontally Explicit and Vertically Implicit (HEVI) Time Discretization Scheme for a Discontinuous Galerkin Nonhydrostatic Model
journal, March 2015

  • Bao, Lei; Klöfkorn, Robert; Nair, Ramachandran D.
  • Monthly Weather Review, Vol. 143, Issue 3
  • DOI: 10.1175/MWR-D-14-00083.1

A New Class of Optimal High-Order Strong-Stability-Preserving Time Discretization Methods
journal, January 2002


Efficient construction of unified continuous and discontinuous Galerkin formulations for the 3D Euler equations
journal, September 2016


Petascale solvers for anisotropic PDEs in atmospheric modelling on GPU clusters
journal, December 2015


Accelerating Time Integration for the Shallow Water Equations on the Sphere Using GPUs
journal, January 2015


An Implicit Time Integration Scheme for Baroclinic Models of the Atmosphere
journal, May 1972


A Semi-Implicit Scheme for grid Point Atmospheric Models of the Primitive Equations
journal, January 1971


High-order splitting methods for the incompressible Navier-Stokes equations
journal, December 1991

  • Karniadakis, George Em; Israeli, Moshe; Orszag, Steven A.
  • Journal of Computational Physics, Vol. 97, Issue 2
  • DOI: 10.1016/0021-9991(91)90007-8

Conservative Split-Explicit Time Integration Methods for the Compressible Nonhydrostatic Equations
journal, August 2007

  • Klemp, J. B.; Skamarock, W. C.; Dudhia, J.
  • Monthly Weather Review, Vol. 135, Issue 8
  • DOI: 10.1175/MWR3440.1