skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Performance analysis of fully explicit and fully implicit solvers within a spectral element shallow-water atmosphere model

Abstract

In this study, explicit Runge–Kutta methods and implicit multistep methods utilizing a Newton–Krylov nonlinear solver are evaluated for a range of configurations of the shallow-water dynamical core of the spectral element community atmosphere model to evaluate their computational performance. These configurations are designed to explore the attributes of each method under different but relevant model usage scenarios including varied spectral order within an element, static regional refinement, and scaling to the largest problem sizes. This analysis is performed within the shallow-water dynamical core option of a full climate model code base to enable a wealth of simulations for study, with the aim of informing solver development within the more complete hydrostatic dynamical core used for climate research. The limitations and benefits to using explicit versus implicit methods, with different parameters and settings, are discussed in light of the trade-offs with Message Passing Interface (MPI) communication and memory and their inherent efficiency bottlenecks. Given the performance behavior across the configurations analyzed here, the recommendation for future work using the implicit solvers is conditional based on scale separation and the stiffness of the problem. For the regionally refined configurations, the implicit method has about the same efficiency as the explicit method, withoutmore » considering efficiency gains from a preconditioner. The potential for improvement using a preconditioner is greatest for higher spectral order configurations, where more work is shifted to the linear solver. Finally, initial simulations with OpenACC directives to utilize a Graphics Processing Unit (GPU) when performing function evaluations show improvements locally, and that overall gains are possible with adjustments to data exchanges.« less

Authors:
ORCiD logo [1]; ORCiD logo [1];  [2]; ORCiD logo [1];  [3];  [2]; ORCiD logo [1]
  1. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  2. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
  3. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1490619
Grant/Contract Number:  
AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
International Journal of High Performance Computing Applications
Additional Journal Information:
Journal Volume: 33; Journal Issue: 2; Journal ID: ISSN 1094-3420
Publisher:
SAGE
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Global climate modeling; implicit methods; Newton–Krylov; regional refinement; GPU acceleration

Citation Formats

Evans, Katherine J., Archibald, Richard K., Gardner, David J., Norman, Matthew R., Taylor, Mark A., Woodward, Carol S., and Worley, Patrick H. Performance analysis of fully explicit and fully implicit solvers within a spectral element shallow-water atmosphere model. United States: N. p., 2017. Web. doi:10.1177/1094342017736373.
Evans, Katherine J., Archibald, Richard K., Gardner, David J., Norman, Matthew R., Taylor, Mark A., Woodward, Carol S., & Worley, Patrick H. Performance analysis of fully explicit and fully implicit solvers within a spectral element shallow-water atmosphere model. United States. doi:10.1177/1094342017736373.
Evans, Katherine J., Archibald, Richard K., Gardner, David J., Norman, Matthew R., Taylor, Mark A., Woodward, Carol S., and Worley, Patrick H. Tue . "Performance analysis of fully explicit and fully implicit solvers within a spectral element shallow-water atmosphere model". United States. doi:10.1177/1094342017736373. https://www.osti.gov/servlets/purl/1490619.
@article{osti_1490619,
title = {Performance analysis of fully explicit and fully implicit solvers within a spectral element shallow-water atmosphere model},
author = {Evans, Katherine J. and Archibald, Richard K. and Gardner, David J. and Norman, Matthew R. and Taylor, Mark A. and Woodward, Carol S. and Worley, Patrick H.},
abstractNote = {In this study, explicit Runge–Kutta methods and implicit multistep methods utilizing a Newton–Krylov nonlinear solver are evaluated for a range of configurations of the shallow-water dynamical core of the spectral element community atmosphere model to evaluate their computational performance. These configurations are designed to explore the attributes of each method under different but relevant model usage scenarios including varied spectral order within an element, static regional refinement, and scaling to the largest problem sizes. This analysis is performed within the shallow-water dynamical core option of a full climate model code base to enable a wealth of simulations for study, with the aim of informing solver development within the more complete hydrostatic dynamical core used for climate research. The limitations and benefits to using explicit versus implicit methods, with different parameters and settings, are discussed in light of the trade-offs with Message Passing Interface (MPI) communication and memory and their inherent efficiency bottlenecks. Given the performance behavior across the configurations analyzed here, the recommendation for future work using the implicit solvers is conditional based on scale separation and the stiffness of the problem. For the regionally refined configurations, the implicit method has about the same efficiency as the explicit method, without considering efficiency gains from a preconditioner. The potential for improvement using a preconditioner is greatest for higher spectral order configurations, where more work is shifted to the linear solver. Finally, initial simulations with OpenACC directives to utilize a Graphics Processing Unit (GPU) when performing function evaluations show improvements locally, and that overall gains are possible with adjustments to data exchanges.},
doi = {10.1177/1094342017736373},
journal = {International Journal of High Performance Computing Applications},
number = 2,
volume = 33,
place = {United States},
year = {2017},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

AMIP Simulation with the CAM4 Spectral Element Dynamical Core
journal, February 2013


Spectral Transform Solutions to the Shallow Water Test Set
journal, June 1995

  • Jakob-Chien, Ruediger; Hack, James J.; Williamson, David L.
  • Journal of Computational Physics, Vol. 119, Issue 1
  • DOI: 10.1006/jcph.1995.1125

Algorithmically scalable block preconditioner for fully implicit shallow-water equations in CAM-SE
journal, October 2014

  • Lott, P. Aaron; Woodward, Carol S.; Evans, Katherine J.
  • Computational Geosciences, Vol. 19, Issue 1
  • DOI: 10.1007/s10596-014-9447-6

The Spectral Element Method for the Shallow Water Equations on the Sphere
journal, January 1997

  • Taylor, Mark; Tribbia, Joseph; Iskandarani, Mohamed
  • Journal of Computational Physics, Vol. 130, Issue 1
  • DOI: 10.1006/jcph.1996.5554

Accuracy Analysis of a Spectral Element Atmospheric Model Using a Fully Implicit Solution Framework
journal, August 2010

  • Evans, Katherine J.; Taylor, Mark A.; Drake, John B.
  • Monthly Weather Review, Vol. 138, Issue 8
  • DOI: 10.1175/2010MWR3288.1

Short‐term time step convergence in a climate model
journal, February 2015

  • Wan, Hui; Rasch, Philip J.; Taylor, Mark A.
  • Journal of Advances in Modeling Earth Systems, Vol. 7, Issue 1
  • DOI: 10.1002/2014MS000368

Jacobian-free Newton–Krylov methods: a survey of approaches and applications
journal, January 2004


A new synoptic scale resolving global climate simulation using the Community Earth System Model
journal, December 2014

  • Small, R. Justin; Bacmeister, Julio; Bailey, David
  • Journal of Advances in Modeling Earth Systems, Vol. 6, Issue 4
  • DOI: 10.1002/2014MS000363

Amesos2 and Belos: Direct and Iterative Solvers for Large Sparse Linear Systems
journal, January 2012

  • Bavier, Eric; Hoemmen, Mark; Rajamanickam, Sivasankaran
  • Scientific Programming, Vol. 20, Issue 3
  • DOI: 10.1155/2012/243875

CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model
journal, November 2011

  • Dennis, John M.; Edwards, Jim; Evans, Katherine J.
  • The International Journal of High Performance Computing Applications, Vol. 26, Issue 1
  • DOI: 10.1177/1094342011428142

On balanced approximations for time integration of multiple time scale systems
journal, March 2003


A Spectral Deferred Correction Method Applied to the Shallow Water Equations on a Sphere
journal, October 2013

  • Jia, Jun; Hill, Judith C.; Evans, Katherine J.
  • Monthly Weather Review, Vol. 141, Issue 10
  • DOI: 10.1175/MWR-D-12-00048.1

On the Use of Finite Difference Matrix-vector Products in Newton-krylov Solvers for Implicit Climate Dynamics with Spectral Elements
journal, January 2015


An A Posteriori–A Priori Analysis of Multiscale Operator Splitting
journal, January 2008

  • Estep, D.; Ginting, V.; Ropp, D.
  • SIAM Journal on Numerical Analysis, Vol. 46, Issue 3
  • DOI: 10.1137/07068237X

Effects of time step size on the simulation of tropical climate in NCAR-CAM3
journal, January 2011


GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems
journal, July 1986

  • Saad, Youcef; Schultz, Martin H.
  • SIAM Journal on Scientific and Statistical Computing, Vol. 7, Issue 3
  • DOI: 10.1137/0907058

The Parallel Performance of Standard Parabolic Marching Schemes
journal, March 1991

  • Vandewalle, S.; Van Driessche, R.; Piessens, R.
  • International Journal of High Speed Computing, Vol. 03, Issue 01
  • DOI: 10.1142/S0129053391000024

A Semi-Implicit Version of the MPAS-Atmosphere Dynamical Core
journal, September 2015

  • Sandbach, Steven; Thuburn, John; Vassilev, Danail
  • Monthly Weather Review, Vol. 143, Issue 9
  • DOI: 10.1175/MWR-D-15-0059.1

An initial-value problem for testing numerical models of the global shallow-water equations
journal, January 2004

  • Galewsky, Joseph; Scott, Richard K.; Polvani, Lorenzo M.
  • Tellus A: Dynamic Meteorology and Oceanography, Vol. 56, Issue 5
  • DOI: 10.3402/tellusa.v56i5.14436

Accelerating Time Integration for the Shallow Water Equations on the Sphere Using GPUs
journal, January 2015


The spectral element method (SEM) on variable-resolution grids: evaluating grid sensitivity and resolution-aware numerical viscosity
journal, January 2014

  • Guba, O.; Taylor, M. A.; Ullrich, P. A.
  • Geoscientific Model Development, Vol. 7, Issue 6
  • DOI: 10.5194/gmd-7-2803-2014

An overview of the Trilinos project
journal, September 2005

  • Heroux, Michael A.; Phipps, Eric T.; Salinger, Andrew G.
  • ACM Transactions on Mathematical Software, Vol. 31, Issue 3
  • DOI: 10.1145/1089014.1089021

A global shallow-water model using an expanded spherical cube: Gnomonic versus conformal coordinates
journal, April 1996

  • Rančić, M.; Purser, R. J.; Mesinger, F.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 122, Issue 532
  • DOI: 10.1002/qj.49712253209

The effect of time steps and time-scales on parametrization suites
journal, August 2012

  • Williamson, David L.
  • Quarterly Journal of the Royal Meteorological Society, Vol. 139, Issue 671
  • DOI: 10.1002/qj.1992

A Fully Implicit Domain Decomposition Algorithm for Shallow Water Equations on the Cubed-Sphere
journal, January 2010

  • Yang, Chao; Cao, Jianwen; Cai, Xiao-Chuan
  • SIAM Journal on Scientific Computing, Vol. 32, Issue 1
  • DOI: 10.1137/080727348

A compatible and conservative spectral element method on unstructured grids
journal, August 2010


A multidecadal simulation of Atlantic tropical cyclones using a variable-resolution global atmospheric general circulation model
journal, August 2014

  • Zarzycki, Colin M.; Jablonowski, Christiane
  • Journal of Advances in Modeling Earth Systems, Vol. 6, Issue 3
  • DOI: 10.1002/2014MS000352

A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel
journal, July 2015


Tropical Cyclones in the UPSCALE Ensemble of High-Resolution Global Climate Models
journal, January 2015

  • Roberts, Malcolm J.; Vidale, Pier Luigi; Mizielinski, Matthew S.
  • Journal of Climate, Vol. 28, Issue 2
  • DOI: 10.1175/JCLI-D-14-00131.1

A standard test set for numerical approximations to the shallow water equations in spherical geometry
journal, September 1992