DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Acceleration of the Parameterization of Unified Microphysics Across Scales (PUMAS) on the Graphics Processing Unit (GPU) With Directive-Based Methods

Journal Article · · Journal of Advances in Modeling Earth Systems

Cloud microphysics is one of the most time-consuming components in a climate model. In this study, we port the cloud microphysics parameterization in the Community Atmosphere Model (CAM), known as Parameterization of Unified Microphysics Across Scales (PUMAS), from CPU to GPU to seek a computational speedup. The directive-based methods (OpenACC and OpenMP target offload) are determined as the best fit specifically for our development practices, which enable a single version of source code to run either on the CPU or GPU, and yield a better portability and maintainability. Their performance is first examined in a PUMAS stand-alone kernel and the directive-based methods can outperform a CPU node as long as there is enough computational burden on the GPU. A consistent behavior is observed when we run PUMAS on the GPU in a practical CAM simulation. A 3.6× speedup of the PUMAS execution time, including data movement between CPU and GPU, is achieved at a coarse horizontal resolution (8 NVIDIA V100 GPUs against 36 Intel Skylake CPU cores). This speedup further increases up to 5.4× at a high resolution (24 NVIDIA V100 GPUs against 108 Intel Skylake CPU cores), which highlights the fact that GPU favors larger problem size. This study demonstrates that using GPU in a CAM simulation can save noticeable computational costs even with a small portion of code being GPU-enabled. Therefore, we are encouraged to port more parameterizations to GPU to take advantage of its computational benefit.

Research Organization:
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); National Center for Atmospheric Research (NCAR), Boulder, CO (United States)
Sponsoring Organization:
USDOE; National Science Foundation (NSF)
Grant/Contract Number:
AC05-76RL01830; 531 2004973
OSTI ID:
2217296
Report Number(s):
PNNL-SA-192584
Journal Information:
Journal of Advances in Modeling Earth Systems, Vol. 15, Issue 5; ISSN 1942-2466
Publisher:
American Geophysical Union (AGU)Copyright Statement
Country of Publication:
United States
Language:
English

References (34)

Exploratory High-Resolution Climate Simulations using the Community Atmosphere Model (CAM) journal May 2014
GPU Compute Unified Device Architecture (CUDA)-based Parallelization of the RRTMG Shortwave Rapid Radiative Transfer Model journal February 2016
Simulating Observations of Southern Ocean Clouds and Implications for Climate journal October 2020
Toward Convective-Scale Prediction within the Next Generation Global Prediction System journal July 2019
Using a GPU to Accelerate a Longwave Radiative Transfer Model with Efficient CUDA-Based Methods journal September 2019
Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0 journal January 2018
A New Two-Moment Bulk Stratiform Cloud Microphysics Scheme in the Community Atmosphere Model, Version 3 (CAM3). Part I: Description and Numerical Tests journal August 2008
Nine time steps: ultra-fast statistical consistency testing of the Community Earth System Model (pyCECT v3.0) journal January 2018
Optimizing high-resolution Community Earth System Model on a heterogeneous many-core supercomputing platform journal January 2020
Accelerate the Parameterization of Unified Microphysics Across Scales (PUMAS) on the graphics processing unit (GPU) with directive-based methods dataset January 2022
Impact of Nudging Strategy on the Climate Representativeness and Hindcast Skill of Constrained EAMv1 Simulations journal December 2019
CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model journal November 2011
KGEN: A Python Tool for Automated Fortran Kernel Generation and Verification journal January 2016
GFDL SHiELD: A Unified System for Weather‐to‐Seasonal Prediction journal October 2020
Towards Characterizing the Variability of Statistically Consistent Community Earth System Model Simulations journal January 2016
A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0) journal January 2015
Advanced Two-Moment Bulk Microphysics for Global Models. Part II: Global Model Solutions and Aerosol–Cloud Interactions journal February 2015
Advanced Two-Moment Bulk Microphysics for Global Models. Part I: Off-Line Tests and Comparison with Other Schemes journal February 2015
Two‐Moment Bulk Cloud Microphysics With Prognostic Precipitation in GFDL's Atmosphere Model AM4.0: Configuration and Performance journal June 2021
On the Migration of OpenACC-Based Applications into OpenMP 5+ book January 2022
GPU acceleration of MPAS microphysics WSM6 using OpenACC directives: Performance and verification journal January 2021
HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model journal January 2019
Importance of ice nucleation and precipitation on climate with the Parameterization of Unified Microphysics Across Scales version 1 (PUMASv1) journal March 2023
A New Two-Moment Bulk Stratiform Cloud Microphysics Scheme in the Community Atmosphere Model, Version 3 (CAM3). Part II: Single-Column and Global Results journal August 2008
Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors journal October 2017
The Community Earth System Model Version 2 (CESM2) journal February 2020
Refactoring and Optimizing the Community Atmosphere Model (CAM) on the Sunway TaihuLight Supercomputer conference November 2016
Speeding Up the Computation of WRF Double-Moment 6-Class Microphysics Scheme with GPU journal December 2013
Confronting the Challenge of Modeling Cloud and Precipitation Microphysics journal July 2020
A compatible and conservative spectral element method on unstructured grids journal August 2010
Impact of Cloud Microphysics on the Development of Trailing Stratiform Precipitation in a Simulated Squall Line: Comparison of One- and Two-Moment Schemes journal March 2009
A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel journal July 2015
Assessment of marine boundary layer cloud simulations in the CAM with CLUBB and updated microphysics scheme based on ARM observations from the Azores: MBL CLOUDS IN CAM-CLUBB-MG2 VERSUS ARM OBSERVATIONS journal July 2016
Accelerate the Parameterization of Unified Microphysics Across Scales (PUMAS) on the graphics processing unit (GPU) with directive-based methods dataset January 2022