A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel

Norman, Matthew R.; Larkin, Jeffrey; Vose, Aaron; Evans, Katherine J.

doi:10.1016/j.jocs.2015.04.022

Title: A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel

Abstract

The porting of a key kernel in the tracer advection routines of the Community Atmosphere Model – Spectral Element (CAM-SE) to use Graphics Processing Units (GPUs) using OpenACC is considered in comparison to an existing CUDA FORTRAN port. The development of the OpenACC kernel for GPUs was substantially simpler than that of the CUDA port. Also, OpenACC performance was about 1.5× slower than the optimized CUDA version. Particular focus is given to compiler maturity regarding OpenACC implementation for modern FORTRAN, and it is found that the Cray implementation is currently more mature than the PGI implementation. Still, for the case that ran successfully on PGI, the PGI OpenACC runtime was slightly faster than Cray. The results show encouraging performance for OpenACC implementation compared to CUDA while also exposing some issues that may be necessary before the implementations are suitable for porting all of CAM-SE. Furthermore, most notable are that GPU shared memory should be used by future OpenACC implementations and that derived type support should be expanded.

Authors:

^[1]; Larkin, Jeffrey ^[2]; Vose, Aaron ^[3];

^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Nvidia, Santa Clara, CA (United States)
Cray, Seattle, WA (United States)

Publication Date:: Sat Apr 18 00:00:00 EDT 2015

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1462913

Alternate Identifier(s):: OSTI ID: 1251654

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Computational Science

Additional Journal Information:: Journal Volume: 9; Journal Issue: C; Journal ID: ISSN 1877-7503

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; 54 ENVIRONMENTAL SCIENCES; OpenACC; Climate; CUDA; GPU; HPC

Citation Formats


                    Norman, Matthew R., Larkin, Jeffrey, Vose, Aaron, and Evans, Katherine J. A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel.  United States: N. p., 2015. 
Web.  doi:10.1016/j.jocs.2015.04.022.

Copy to clipboard


                    Norman, Matthew R., Larkin, Jeffrey, Vose, Aaron, & Evans, Katherine J. A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel.  United States.  https://doi.org/10.1016/j.jocs.2015.04.022

Copy to clipboard


                    Norman, Matthew R., Larkin, Jeffrey, Vose, Aaron, and Evans, Katherine J. Sat .  
"A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel".  United States.  https://doi.org/10.1016/j.jocs.2015.04.022.  https://www.osti.gov/servlets/purl/1462913.

Copy to clipboard


                    
@article{osti_1462913,

  title        = {A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel},

  author       = {Norman, Matthew R. and Larkin, Jeffrey and Vose, Aaron and Evans, Katherine J.},

  abstractNote = {The porting of a key kernel in the tracer advection routines of the Community Atmosphere Model – Spectral Element (CAM-SE) to use Graphics Processing Units (GPUs) using OpenACC is considered in comparison to an existing CUDA FORTRAN port. The development of the OpenACC kernel for GPUs was substantially simpler than that of the CUDA port. Also, OpenACC performance was about 1.5× slower than the optimized CUDA version. Particular focus is given to compiler maturity regarding OpenACC implementation for modern FORTRAN, and it is found that the Cray implementation is currently more mature than the PGI implementation. Still, for the case that ran successfully on PGI, the PGI OpenACC runtime was slightly faster than Cray. The results show encouraging performance for OpenACC implementation compared to CUDA while also exposing some issues that may be necessary before the implementations are suitable for porting all of CAM-SE. Furthermore, most notable are that GPU shared memory should be used by future OpenACC implementations and that derived type support should be expanded.},

  doi          = {10.1016/j.jocs.2015.04.022},

  journal      = {Journal of Computational Science},

  number       = C,

  volume       = 9,

  place        = {United States},

  year         = {Sat Apr 18 00:00:00 EDT 2015},

  month        = {Sat Apr 18 00:00:00 EDT 2015}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.jocs.2015.04.022

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 23 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Efficient estimation and ensemble generation in climate modelling
journal, June 2007

Annan, J. D.; Hargreaves, J. C.
Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 365, Issue 1857
DOI: 10.1098/rsta.2007.2067

Atmospheric blocking in a high resolution climate model: influences of mean state, orography and eddy forcing: Atmospheric blocking in a high resolution climate model
journal, January 2013

Berckmans, Julie; Woollings, Tim; Demory, Marie-Estelle
Atmospheric Science Letters, Vol. 14, Issue 1
DOI: 10.1002/asl2.412

Simulated Climate and Climate Change in the GFDL CM2.5 High-Resolution Coupled Climate Model
journal, April 2012

Delworth, Thomas L.; Rosati, Anthony; Anderson, Whit
Journal of Climate, Vol. 25, Issue 8
DOI: 10.1175/JCLI-D-11-00316.1

CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model
journal, November 2011

Dennis, John M.; Edwards, Jim; Evans, Katherine J.
The International Journal of High Performance Computing Applications, Vol. 26, Issue 1
DOI: 10.1177/1094342011428142

High-Resolution Global Climate Simulations with the ECMWF Model in Project Athena: Experimental Design, Model Climate, and Seasonal Forecast Skill
journal, May 2012

Jung, T.; Miller, M. J.; Palmer, T. N.
Journal of Climate, Vol. 25, Issue 9
DOI: 10.1175/JCLI-D-11-00265.1

Simulations of the Atmospheric General Circulation Using a Cloud-Resolving Model as a Superparameterization of Physical Processes
journal, July 2005

Khairoutdinov, Marat; Randall, David; DeMott, Charlotte
Journal of the Atmospheric Sciences, Vol. 62, Issue 7
DOI: 10.1175/JAS3453.1

Emulation to simulate low-resolution atmospheric data
journal, January 2014

Rao, Vishwas; Archibald, Rick; Evans, Katherine J.
International Journal of Computer Mathematics, Vol. 91, Issue 4
DOI: 10.1080/00207160.2013.852663

Effects of vertical resolution and nonorographic gravity wave drag on the simulated climate in the Community Atmosphere Model, version 5
journal, May 2014

Richter, Jadwiga H.; Solomon, Abraham; Bacmeister, Julio T.
Journal of Advances in Modeling Earth Systems, Vol. 6, Issue 2
DOI: 10.1002/2013MS000303

Sensitivity of Simulated Climate to Horizontal and Vertical Resolution in the ECHAM5 Atmosphere Model
journal, August 2006

Roeckner, E.; Brokopf, R.; Esch, M.
Journal of Climate, Vol. 19, Issue 16
DOI: 10.1175/JCLI3824.1

A mass and energy conserving spectral element atmospheric dynamical core on the cubed-sphere grid
journal, July 2007

Taylor, M. A.; Edwards, J.; Thomas, S.
Journal of Physics: Conference Series, Vol. 78
DOI: 10.1088/1742-6596/78/1/012074

Similar Records in DOE PAGES and OSTI.GOV collections:

Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations

Journal Article Gong, Jing ; Markidis, Stefano ; Laure, Erwin ; ... - Journal of Supercomputing

We present a hybrid GPU implementation and performance analysis of Nekbone, which represents one of the core kernels of the incompressible Navier–Stokes solver Nek5000. The implementation is based on OpenACC and CUDA Fortran for local parallelization of the compute-intensive matrix–matrix multiplication part, which significantly minimizes the modification of the existing CPU code while extending the simulation capability of the code to GPU architectures. Our discussion includes the GPU results of OpenACC interoperating with CUDA Fortran and the gather–scatter operations with GPUDirect communication. We demonstrate performance of up to 552 Tflops on 16, 384 GPUs of the OLCF Cray XK7 Titan.
https://doi.org/10.1007/s11227-016-1744-5
Optimization and Portability of a Fusion OpenACC-based FORTRAN HPC Code from NVIDIA to AMD GPUs

Conference Sfiligoi, Igor ; Belli, Emily A. ; Candy, Jeff ; ...

NVIDIA has been the main provider of GPU hardware in HPC systems for over a decade. Most applications that benefit from GPUs have thus been developed and optimized for the NVIDIA software stack. Recent exascale HPC systems are, however, introducing GPUs from other vendors, e.g. with the AMD GPU-based OLCF Frontier system just becoming available. AMD GPUs cannot be directly accessed using the NVIDIA software stack, and require a porting effort by the application developers. This paper provides an overview of our experience porting and optimizing the CGYRO code, a widely-used fusion simulation tool based on FORTRAN with OpenACC-based GPUmore »« less
https://doi.org/10.1145/3569951.3597545

Full Text Available
A feasibility study on porting the community land model onto accelerators using OpenACC

Journal Article Wang, Dali ; Wu, Wei ; Winkler, Frank ; ... - International Journal of Advanced Computer Science and Applications

As environmental models (such as Accelerated Climate Model for Energy (ACME), Parallel Reactive Flow and Transport Model (PFLOTRAN), Arctic Terrestrial Simulator (ATS), etc.) became more and more complicated, we are facing enormous challenges regarding to porting those applications onto hybrid computing architecture. OpenACC appears as a very promising technology, therefore, we have conducted a feasibility analysis on porting the Community Land Model (CLM), a terrestrial ecosystem model within the Community Earth System Models (CESM)). Specifically, we used automatic function testing platform to extract a small computing kernel out of CLM, then we apply this kernel into the actually CLM dataflowmore »« less
https://doi.org/10.14569/IJACSA.2014.051203

Full Text Available
Experiences in porting mini-applications to OpenACC and OpenMP on heterogeneous systems

Journal Article Vergara Larrea, Verónica G. ; Budiardja, Reuben D. ; Gayatri, Rahulkumar ; ... - Concurrency and Computation. Practice and Experience

This article studies mini-applications—Minisweep, GenASiS, GPP, and FF—that use computational methods commonly encountered in HPC. We have ported these applications to develop OpenACC and OpenMP versions, and evaluated their performance on Titan (Cray XK7 with K20x GPUs), Cori (Cray XC40 with Intel KNL), Summit (IBM AC922 with Volta GPUs), and Cori-GPU (Cray CS-Storm 500NX with Intel Skylake and Volta GPUs). Our goals are for these new ports to be useful to both application and compiler developers, to document and describe the lessons learned and the methodology to create optimized OpenMP and OpenACC versions, and to provide a description of possiblemore »« less
Cited by 8
https://doi.org/10.1002/cpe.5780

Full Text Available
OpenACC acceleration of the Nek5000 spectral element code

Journal Article Markidis, Stefano ; Gong, Jing ; Schliephake, Michael ; ... - International Journal of High Performance Computing Applications

We present a case study of porting NekBone, a skeleton version of the Nek5000 code, to a parallel GPU-accelerated system. Nek5000 is a computational fluid dynamics code based on the spectral element method used for the simulation of incompressible flow. The original NekBone Fortran source code has been used as the base and enhanced by OpenACC directives. The profiling of NekBone provided an assessment of the suitability of the code for GPU systems, and indicated possible kernel optimizations. To port NekBone to GPU systems required little effort and a small number of additional lines of code (approximately one OpenACC directivemore »« less
https://doi.org/10.1177/1094342015576846

Similar Records

Title: A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel

Abstract

Citation Formats

Efficient estimation and ensemble generation in climate modelling journal, June 2007

Atmospheric blocking in a high resolution climate model: influences of mean state, orography and eddy forcing: Atmospheric blocking in a high resolution climate model journal, January 2013

Simulated Climate and Climate Change in the GFDL CM2.5 High-Resolution Coupled Climate Model journal, April 2012

CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model journal, November 2011

High-Resolution Global Climate Simulations with the ECMWF Model in Project Athena: Experimental Design, Model Climate, and Seasonal Forecast Skill journal, May 2012

Simulations of the Atmospheric General Circulation Using a Cloud-Resolving Model as a Superparameterization of Physical Processes journal, July 2005

Emulation to simulate low-resolution atmospheric data journal, January 2014

Effects of vertical resolution and nonorographic gravity wave drag on the simulated climate in the Community Atmosphere Model, version 5 journal, May 2014

Sensitivity of Simulated Climate to Horizontal and Vertical Resolution in the ECHAM5 Atmosphere Model journal, August 2006

A mass and energy conserving spectral element atmospheric dynamical core on the cubed-sphere grid journal, July 2007

Efficient estimation and ensemble generation in climate modelling
journal, June 2007

Atmospheric blocking in a high resolution climate model: influences of mean state, orography and eddy forcing: Atmospheric blocking in a high resolution climate model
journal, January 2013

Simulated Climate and Climate Change in the GFDL CM2.5 High-Resolution Coupled Climate Model
journal, April 2012

CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model
journal, November 2011

High-Resolution Global Climate Simulations with the ECMWF Model in Project Athena: Experimental Design, Model Climate, and Seasonal Forecast Skill
journal, May 2012

Simulations of the Atmospheric General Circulation Using a Cloud-Resolving Model as a Superparameterization of Physical Processes
journal, July 2005

Emulation to simulate low-resolution atmospheric data
journal, January 2014

Effects of vertical resolution and nonorographic gravity wave drag on the simulated climate in the Community Atmosphere Model, version 5
journal, May 2014

Sensitivity of Simulated Climate to Horizontal and Vertical Resolution in the ECHAM5 Atmosphere Model
journal, August 2006

A mass and energy conserving spectral element atmospheric dynamical core on the cubed-sphere grid
journal, July 2007