HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model

Bertagna, Luca; Deakin, Michael; Guba, Oksana; Sunderland, Daniel; Bradley, Andrew M.; Tezaur, Irina K.; Taylor, Mark A.; Salinger, Andrew G.

doi:10.5194/gmd-12-1423-2019

HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model

Journal Article · Thu Apr 11 00:00:00 EDT 2019 · Geoscientific Model Development (Online)

DOI:https://doi.org/10.5194/gmd-12-1423-2019· OSTI ID:1529244

^[1]; ^[1]; Guba, Oksana ^[1]; Sunderland, Daniel ^[1]; Bradley, Andrew M. ^[1]; Tezaur, Irina K. ^[1]; ^[1]; Salinger, Andrew G. ^[1]

Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Abstract. We present an architecture-portable and performant implementation of the atmospheric dynamical core (High-Order Methods Modeling Environment, HOMME) of the Energy Exascale Earth System Model (E3SM). The original Fortran implementation is highly performant and scalable on conventional architectures using the Message Passing Interface(MPI) and Open MultiProcessor (OpenMP) programming models.We rewrite the model in C++ and use the Kokkos library to express on-node parallelism in a largely architecture-independent implementation. Kokkos provides an abstraction of a compute node or device,layout-polymorphic multidimensional arrays, and parallel execution constructs. The new implementation achieves the same or better performance on conventional multicore computers and is portable to GPUs. We present performance data for the original and new implementations on multiple platforms, on up to 5400 compute nodes, and study several aspects of the single- and multi-node performance characteristics of the new implementation on conventional CPU (e.g., Intel Xeon), many core CPU (e.g., Intel Xeon Phi Knights Landing),and Nvidia V100 GPU.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: NA0003525; AC02-05CH11231

OSTI ID:: 1529244

Journal Information:: Geoscientific Model Development (Online), Journal Name: Geoscientific Model Development (Online) Journal Issue: 4 Vol. 12; ISSN 1991-9603

Publisher:: European Geosciences UnionCopyright Statement

Country of Publication:: United States

Language:: English

References (28)

Albany: Using Component-Based Design to Develop a Flexible, Generic Multiphysics Analysis code Salinger, Andrew G.; Bartlett, Roscoe A.; Bradley, Andrew M. International Journal for Multiscale Computational Engineering, Vol. 14, Issue 4 https://doi.org/10.1615/intjmultcompeng.2016017040	journal	January 2016
Spectral Methods: Evolution to Complex Geometrics and Applications to Fluid Dynamics Canuto, Claudio; Quarteroni, Alfio; Hussaini, M. Yousuff Scientific Computation https://doi.org/10.1007/978-3-540-30728-0	book	January 2007
Spectral Methods: Algorithms, Analysis and Applications Shen, Jie; Tang, Tao; Wang, Li-Lian Springer Series in Computational Mathematics, Vol. 41 https://doi.org/10.1007/978-3-540-71041-7	book	August 2011
Conservation of Mass and Energy for the Moist Atmospheric Primitive Equations on Unstructured Grids Taylor, Mark A. Numerical Techniques for Global Atmospheric Models https://doi.org/10.1007/978-3-642-11640-7_12	book	January 2011
The Piecewise Parabolic Method (PPM) for gas-dynamical simulations Colella, Phillip; Woodward, Paul R. Journal of Computational Physics, Vol. 54, Issue 1 https://doi.org/10.1016/0021-9991(84)90143-8	journal	April 1984
Optimization-based limiters for the spectral element method Guba, Oksana; Taylor, Mark; St-Cyr, Amik Journal of Computational Physics, Vol. 267 https://doi.org/10.1016/j.jcp.2014.02.029	journal	June 2014
A case study of CUDA FORTRAN and OpenACC for an atmospheric climate kernel Norman, Matthew; Larkin, Jeffrey; Vose, Aaron Journal of Computational Science, Vol. 9 https://doi.org/10.1016/j.jocs.2015.04.022	journal	July 2015
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.07.003	journal	December 2014
Aeras: A Next Generation Global Atmosphere Model Spotz, William F.; Smith, Thomas M.; Demeshko, Irina P. Procedia Computer Science, Vol. 51 https://doi.org/10.1016/j.procs.2015.05.478	journal	January 2015
The Paris Climate Agreement and future sea-level rise from Antarctica DeConto, Robert M.; Pollard, David; Alley, Richard B. Nature, Vol. 593, Issue 7857 https://doi.org/10.1038/s41586-021-03427-0	journal	May 2021
ParalleX An Advanced Parallel Execution Model for Scaling-Impaired Applications Kaiser, Hartmut; Brodowicz, Maciek; Sterling, Thomas 2009 International Conference on Parallel Processing Workshops (ICPPW) https://doi.org/10.1109/ICPPW.2009.14	conference	September 2009
Refactoring and Optimizing the Community Atmosphere Model (CAM) on the Sunway TaihuLight Supercomputer Fu, Haohuan; Liao, Junfeng; Xue, Wei SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.82	conference	November 2016
Vectorizing compilers: a test suite and results Callahan, D.; Dongarra, J.; Levine, D. Proceedings. SUPERCOMPUTING '88 https://doi.org/10.1109/SUPERC.1988.44642	conference	January 1988
Early Experiences with the 360tf ibm blue Gene/L Platform Bhanot, G.; Dennis, J. M.; Edwards, J. International Journal of Computational Methods, Vol. 05, Issue 02 https://doi.org/10.1142/S0219876208001443	journal	June 2008
Performance of the community earth system model Worley, Patrick H.; Mirin, Arthur A.; Craig, Anthony P. Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063457	conference	January 2011
Stella: a python-based domain-specific language for simulations Mohr, David; Stefanovic, Darko Proceedings of the 31st Annual ACM Symposium on Applied Computing - SAC '16 https://doi.org/10.1145/2851613.2851749	conference	January 2016
Redesigning CAM-SE for peta-scale climate modeling performance and ultra-high resolution on Sunway TaihuLight Fu, Haohuan; Liu, Weiguo; Wang, Lanning Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17 https://doi.org/10.1145/3126908.3126909	conference	January 2017
The CLAW DSL: Abstractions for Performance Portable Weather and Climate Models Clement, Valentin; Ferrachat, Sylvaine; Fuhrer, Oliver Proceedings of the Platform for Advanced Scientific Computing Conference on - PASC '18 https://doi.org/10.1145/3218176.3218226	conference	January 2018
The Community Earth System Model: A Framework for Collaborative Research Hurrell, James W.; Holland, M. M.; Gent, P. R. Bulletin of the American Meteorological Society, Vol. 94, Issue 9 https://doi.org/10.1175/BAMS-D-12-00121.1	journal	September 2013
High-Resolution Mesh Convergence Properties and Parallel Efficiency of a Spectral Element Atmospheric Dynamical Core Dennis, John; Fournier, Aimé; Spotz, William F. The International Journal of High Performance Computing Applications, Vol. 19, Issue 3 https://doi.org/10.1177/1094342005056108	journal	August 2005
CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model Dennis, John M.; Edwards, Jim; Evans, Katherine J. The International Journal of High Performance Computing Applications, Vol. 26, Issue 1 https://doi.org/10.1177/1094342011428142	journal	November 2011
Progress towards accelerating HOMME on hybrid multi-core systems Carpenter, I.; Archibald, R. K.; Evans, K. J. The International Journal of High Performance Computing Applications, Vol. 27, Issue 3 https://doi.org/10.1177/1094342012462751	journal	November 2012
Acceleration of the IMplicit–EXplicit nonhydrostatic unified model of the atmosphere on manycore processors Abdi, Daniel S.; Giraldo, Francis X.; Constantinescu, Emil M. The International Journal of High Performance Computing Applications, Vol. 33, Issue 2 https://doi.org/10.1177/1094342017732395	journal	October 2017
Toward performance portability of the Albany finite element analysis code using the Kokkos library Demeshko, Irina; Watkins, Jerry; Tezaur, Irina K. The International Journal of High Performance Computing Applications, Vol. 33, Issue 2 https://doi.org/10.1177/1094342017749957	journal	October 2017
Albany: Using Component-Based Design to Develop a Flexible, Generic Multiphysics Analysis code Salinger, Andrew G.; Bartlett, Roscoe A.; Bradley, Andrew M. International Journal for Multiscale Computational Engineering, Vol. 14, Issue 4 https://doi.org/10.1615/IntJMultCompEng.2016017040	journal	January 2016
Towards Performance Portability in a Compressible CFD Code Howard, Micah; Bradley, Andrew; Bova, Steven W. 23rd AIAA Computational Fluid Dynamics Conference https://doi.org/10.2514/6.2017-4407	conference	June 2017
Near-global climate simulation at 1 km resolution: establishing a performance baseline on 4888 GPUs with COSMO 5.0 Fuhrer, Oliver; Chadha, Tarun; Hoefler, Torsten Geoscientific Model Development, Vol. 11, Issue 4 https://doi.org/10.5194/gmd-11-1665-2018	journal	January 2018
Albany: Using Component-based Design to Develop a Flexible, Generic Multiphysics Analysis Code Salinger, Andrew G.; Bartlett, Roscoe A.; Bradley, Andrew M. Columbia University https://doi.org/10.7916/d8z60pfw	text	January 2016

Cited By (3)

Resilience and fault tolerance in high-performance computing for numerical weather and climate prediction Benacchio, Tommaso; Bonaventura, Luca; Altenbernd, Mirco The International Journal of High Performance Computing Applications, Vol. 35, Issue 4 https://doi.org/10.1177/1094342021990433	journal	February 2021
Efficient IMEX Runge-Kutta methods for nonhydrostatic dynamics Steyer, Andrew; Vogl, Christopher J.; Taylor, Mark arXiv https://doi.org/10.48550/arxiv.1906.07219	preprint	January 2019
Using a GPU to Accelerate a Longwave Radiative Transfer Model with Efficient CUDA-Based Methods Wang, Yuzhu; Zhao, Yuan; Li, Wei Applied Sciences, Vol. 9, Issue 19 https://doi.org/10.3390/app9194039	journal	September 2019

Similar Records

HOMMEXX 1.0: A Performance Portable Atmospheric Dynamical Core for the Energy Exascale Earth System Model

Journal Article · Fri Oct 26 00:00:00 EDT 2018 · Geoscientific Model Development Discussions (Online) · OSTI ID:1497003

Kokkos Version 2.0

Software · Sun Oct 18 20:00:00 EDT 2015 · OSTI ID:code-3666

Related Subjects

58 GEOSCIENCES

HOMMEXX 1.0: a performance-portable atmospheric dynamical core for the Energy Exascale Earth System Model

Citation Formats

References (28)

Cited By (3)

Similar Records

Related Subjects