Asynchronous communication in spectral-element and discontinuous Galerkin methods for atmospheric dynamics – a case study using the High-Order Methods Modeling Environment (HOMME-homme_dg_branch)
Abstract
The scalability of computational applications on current and next-generation supercomputers is increasingly limited by the cost of inter-process communication. We implement non-blocking asynchronous communication in the High-Order Methods Modeling Environment for the time integration of the hydrostatic fluid equations using both the spectral-element and discontinuous Galerkin methods. This allows the overlap of computation with communication, effectively hiding some of the costs of communication. A novel detail about our approach is that it provides some data movement to be performed during the asynchronous communication even in the absence of other computations. This method produces significant performance and scalability gains in large-scale simulations.
- Authors:
-
- National Center for Atmospheric Research, Boulder, CO (United States). Computational Information Systems Lab.
- National Center for Atmospheric Research, Boulder, CO (United States). Computational Information Systems Lab.; International Research Inst. of Stavanger (Norway)
- Publication Date:
- Research Org.:
- National Center for Atmospheric Research (NCAR), Boulder, CO (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Biological and Environmental Research (BER); National Science Foundation (NSF)
- OSTI Identifier:
- 1437146
- Grant/Contract Number:
- SC0006959
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Geoscientific Model Development (Online)
- Additional Journal Information:
- Journal Name: Geoscientific Model Development (Online); Journal Volume: 9; Journal Issue: 8; Journal ID: ISSN 1991-9603
- Publisher:
- European Geosciences Union
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; 58 GEOSCIENCES
Citation Formats
Jamroz, Benjamin F., and Klofkorn, Robert. Asynchronous communication in spectral-element and discontinuous Galerkin methods for atmospheric dynamics – a case study using the High-Order Methods Modeling Environment (HOMME-homme_dg_branch). United States: N. p., 2016.
Web. doi:10.5194/gmd-9-2881-2016.
Jamroz, Benjamin F., & Klofkorn, Robert. Asynchronous communication in spectral-element and discontinuous Galerkin methods for atmospheric dynamics – a case study using the High-Order Methods Modeling Environment (HOMME-homme_dg_branch). United States. https://doi.org/10.5194/gmd-9-2881-2016
Jamroz, Benjamin F., and Klofkorn, Robert. Fri .
"Asynchronous communication in spectral-element and discontinuous Galerkin methods for atmospheric dynamics – a case study using the High-Order Methods Modeling Environment (HOMME-homme_dg_branch)". United States. https://doi.org/10.5194/gmd-9-2881-2016. https://www.osti.gov/servlets/purl/1437146.
@article{osti_1437146,
title = {Asynchronous communication in spectral-element and discontinuous Galerkin methods for atmospheric dynamics – a case study using the High-Order Methods Modeling Environment (HOMME-homme_dg_branch)},
author = {Jamroz, Benjamin F. and Klofkorn, Robert},
abstractNote = {The scalability of computational applications on current and next-generation supercomputers is increasingly limited by the cost of inter-process communication. We implement non-blocking asynchronous communication in the High-Order Methods Modeling Environment for the time integration of the hydrostatic fluid equations using both the spectral-element and discontinuous Galerkin methods. This allows the overlap of computation with communication, effectively hiding some of the costs of communication. A novel detail about our approach is that it provides some data movement to be performed during the asynchronous communication even in the absence of other computations. This method produces significant performance and scalability gains in large-scale simulations.},
doi = {10.5194/gmd-9-2881-2016},
journal = {Geoscientific Model Development (Online)},
number = 8,
volume = 9,
place = {United States},
year = {Fri Aug 26 00:00:00 EDT 2016},
month = {Fri Aug 26 00:00:00 EDT 2016}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 2 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Integrating a scalable and effcient semi-Lagrangian multi-tracer transport scheme in HOMME
journal, January 2012
- Erath, Christoph; Lauritzen, Peter H.; Garcia, Jose H.
- Procedia Computer Science, Vol. 9
A new ensemble-based consistency test for the Community Earth System Model (pyCECT v1.0)
journal, January 2015
- Baker, A. H.; Hammerling, D. M.; Levy, M. N.
- Geoscientific Model Development, Vol. 8, Issue 9
CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model
journal, November 2011
- Dennis, John M.; Edwards, Jim; Evans, Katherine J.
- The International Journal of High Performance Computing Applications, Vol. 26, Issue 1
A conservative multi-tracer transport scheme for spectral-element spherical grids
journal, January 2014
- Erath, Christoph; Nair, Ramachandran D.
- Journal of Computational Physics, Vol. 256
Integrating a scalable and effcient semi-Lagrangian multi-tracer transport scheme in HOMME
journal, January 2012
- Erath, Christoph; Lauritzen, Peter H.; Garcia, Jose H.
- Procedia Computer Science, Vol. 9
The Community Climate System Model Version 4
journal, October 2011
- Gent, Peter R.; Danabasoglu, Gokhan; Donner, Leo J.
- Journal of Climate, Vol. 24, Issue 19
Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers
conference, November 2014
- Heinecke, Alexander; Breuer, Alexander; Rettenberger, Sebastian
- SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
A baroclinic instability test case for atmospheric model dynamical cores
journal, October 2006
- Jablonowski, Christiane; Williamson, David L.
- Quarterly Journal of the Royal Meteorological Society, Vol. 132, Issue 621C
Pracniques: further remarks on reducing truncation errors
journal, January 1965
- Kahan, W.
- Communications of the ACM, Vol. 8, Issue 1
Exaflop/s: The why and the how
journal, February 2011
- Keyes, David E.
- Comptes Rendus Mécanique, Vol. 339, Issue 2-3
Scalable rank-mapping algorithm for an icosahedral grid system on the massive parallel computer with a 3-D torus network
journal, August 2014
- Kodama, Chihiro; Terai, Masaaki; Noda, Akira T.
- Parallel Computing, Vol. 40, Issue 8
Thermosphere extension of the Whole Atmosphere Community Climate Model: WHOLE ATMOSPHERE MODEL
journal, December 2010
- Liu, H. -L.; Foster, B. T.; Hagan, M. E.
- Journal of Geophysical Research: Space Physics, Vol. 115, Issue A12
Computational aspects of a scalable high-order discontinuous Galerkin atmospheric dynamical core
journal, February 2009
- Nair, R. D.; Choi, H. -W.; Tufo, H. M.
- Computers & Fluids, Vol. 38, Issue 2
Diffusion Experiments with a Global Discontinuous Galerkin Shallow-Water Model
journal, October 2009
- Nair, Ramachandran D.
- Monthly Weather Review, Vol. 137, Issue 10
A High-Order Multiscale Global Atmospheric Model (Invited)
conference, June 2016
- Nair, Ram
- 8th AIAA Atmospheric and Space Environments Conference
An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth's mantle
conference, January 2015
- Rudi, Johann; Ghattas, Omar; Malossi, A. Cristiano I.
- Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15
A new synoptic scale resolving global climate simulation using the Community Earth System Model
journal, December 2014
- Small, R. Justin; Bacmeister, Julio; Bailey, David
- Journal of Advances in Modeling Earth Systems, Vol. 6, Issue 4
A compatible and conservative spectral element method on unstructured grids
journal, August 2010
- Taylor, Mark A.; Fournier, Aimé
- Journal of Computational Physics, Vol. 229, Issue 17