Comparing the Performance of Julia on CPUs versus GPUs and Julia-MPI versus Fortran-MPI: a case study with MPAS-Ocean (Version 7.1)

Bishnu, Siddhartha; Strauss, Robert R.; Petersen, Mark R.

doi:10.5194/gmd-16-5539-2023

Comparing the Performance of Julia on CPUs versus GPUs and Julia-MPI versus Fortran-MPI: a case study with MPAS-Ocean (Version 7.1)

Journal Article · Thu Oct 05 00:00:00 EDT 2023 · Geoscientific Model Development (Online)

DOI:https://doi.org/10.5194/gmd-16-5539-2023· OSTI ID:2007667

; ;

Abstract. Some programming languages are easy to develop at the cost of slow execution, while others are fast at runtime but much more difficult to write. Julia is a programming language that aims to be the best of both worlds – a development and production language at the same time. To test Julia's utility in scientific high-performance computing (HPC), we built an unstructured-mesh shallow water model in Julia and compared it against an established Fortran-MPI ocean model, the Model for Prediction Across Scales–Ocean (MPAS-Ocean), as well as a Python shallow water code. Three versions of the Julia shallow water code were created: for single-core CPU, graphics processing unit (GPU), and Message Passing Interface (MPI) CPU clusters. Comparing identical simulations revealed that our first version of the Julia model was 13 times faster than Python using NumPy, where both used an unthreaded single-core CPU. Further Julia optimizations, including static typing and removing implicit memory allocations, provided an additional 10–20× speed-up of the single-core CPU Julia model. The GPU-accelerated Julia code was almost identical in terms of performance to the MPI parallelized code on 64 processes, an unexpected result for such different architectures. Parallelized Julia-MPI performance was identical to Fortran-MPI MPAS-Ocean for low processor counts and ranges from 2× faster to 2× slower for higher processor counts. Our experience is that Julia development is fast and convenient for prototyping but that Julia requires further investment and expertise to be competitive with compiled codes. We provide advice on Julia code optimization for HPC systems.

View Journal Article

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER); USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: 89233218CNA000001; AC02-05CH11231

OSTI ID:: 2007667

Alternate ID(s):: OSTI ID: 2305313

Journal Information:: Geoscientific Model Development (Online), Journal Name: Geoscientific Model Development (Online) Journal Issue: 19 Vol. 16; ISSN 1991-9603

Publisher:: Copernicus GmbHCopyright Statement

Country of Publication:: Germany

Language:: English

References (28)

Portable C++ Code that can Look and Feel Like Fortran Code with Yet Another Kernel Launcher (YAKL) Norman, Matthew; Lyngaas, Isaac; Bagusetty, Abhishek International Journal of Parallel Programming, Vol. 51, Issue 4-5 https://doi.org/10.1007/s10766-022-00739-0	journal	December 2022
A GPU accelerated finite volume coastal ocean model Zhao, Xu-dong; Liang, Shu-xiu; Sun, Zhao-chen Journal of Hydrodynamics, Vol. 29, Issue 4 https://doi.org/10.1016/S1001-6058(16)60780-1	journal	August 2017
Rapid software prototyping for heterogeneous and distributed platforms Besard, Tim; Churavy, Valentin; Edelman, Alan Advances in Engineering Software, Vol. 132 https://doi.org/10.1016/j.advengsoft.2019.02.002	journal	June 2019
Numerical representation of geostrophic modes on arbitrarily structured C-grids Thuburn, J.; Ringler, T. D.; Skamarock, W. C. Journal of Computational Physics, Vol. 228, Issue 22 https://doi.org/10.1016/j.jcp.2009.08.006	journal	December 2009
A unified approach to energy conservation and potential vorticity dynamics for arbitrarily-structured C-grids Ringler, T. D.; Thuburn, J.; Klemp, J. B. Journal of Computational Physics, Vol. 229, Issue 9 https://doi.org/10.1016/j.jcp.2009.12.007	journal	May 2010
MPI for Python Dalcín, Lisandro; Paz, Rodrigo; Storti, Mario Journal of Parallel and Distributed Computing, Vol. 65, Issue 9 https://doi.org/10.1016/j.jpdc.2005.03.010	journal	September 2005
MPI for Python: Performance improvements and MPI-2 extensions Dalcín, Lisandro; Paz, Rodrigo; Storti, Mario Journal of Parallel and Distributed Computing, Vol. 68, Issue 5 https://doi.org/10.1016/j.jpdc.2007.09.005	journal	May 2008
The regional oceanic modeling system (ROMS): a split-explicit, free-surface, topography-following-coordinate oceanic model Shchepetkin, Alexander F.; McWilliams, James C. Ocean Modelling, Vol. 9, Issue 4 https://doi.org/10.1016/j.ocemod.2004.08.002	journal	January 2005
Accelerating a barotropic ocean model using a GPU Bleichrodt, Folkert; Bisseling, Rob H.; Dijkstra, Henk A. Ocean Modelling, Vol. 41 https://doi.org/10.1016/j.ocemod.2011.10.001	journal	January 2012
A multi-resolution approach to global ocean modeling Ringler, Todd; Petersen, Mark; Higdon, Robert L. Ocean Modelling, Vol. 69 https://doi.org/10.1016/j.ocemod.2013.04.010	journal	September 2013
Evaluation of the arbitrary Lagrangian–Eulerian vertical coordinate method in the MPAS-Ocean model Petersen, Mark R.; Jacobsen, Douglas W.; Ringler, Todd D. Ocean Modelling, Vol. 86 https://doi.org/10.1016/j.ocemod.2014.12.004	journal	February 2015
PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation Klöckner, Andreas; Pinto, Nicolas; Lee, Yunsup Parallel Computing, Vol. 38, Issue 3 https://doi.org/10.1016/j.parco.2011.09.001	journal	March 2012
An Evaluation of the Ocean and Sea Ice Climate of E3SM Using MPAS and Interannual CORE‐II Forcing Petersen, Mark R.; Asay‐Davis, Xylar S.; Berres, Anne S. Journal of Advances in Modeling Earth Systems, Vol. 11, Issue 5 https://doi.org/10.1029/2018MS001373	journal	May 2019
The DOE E3SM Coupled Model Version 1: Overview and Evaluation at Standard Resolution Golaz, Jean‐Christophe; Caldwell, Peter M.; Van Roekel, Luke P. Journal of Advances in Modeling Earth Systems, Vol. 11, Issue 7 https://doi.org/10.1029/2018MS001603	journal	July 2019
The DOE E3SM Coupled Model Version 1: Description and Results at High Resolution Caldwell, Peter M.; Mametjanov, Azamat; Tang, Qi Journal of Advances in Modeling Earth Systems, Vol. 11, Issue 12 https://doi.org/10.1029/2019MS001870	journal	December 2019
Fluid Simulations Accelerated With 16 Bits: Approaching 4x Speedup on A64FX by Squeezing ShallowWaters.jl Into Float16 Klöwer, Milan; Hatfield, Sam; Croci, Matteo Journal of Advances in Modeling Earth Systems, Vol. 14, Issue 2 https://doi.org/10.1029/2021MS002684	journal	February 2022
Julia: come for the syntax, stay for the speed Perkel, Jeffrey M. Nature, Vol. 572, Issue 7767 https://doi.org/10.1038/d41586-019-02310-3	journal	July 2019
Statistically significant performance testing of Julia scientific programming language Gevorkyan, M. N.; Demidova, A. V.; Korolkova, A. V. Journal of Physics: Conference Series, Vol. 1205 https://doi.org/10.1088/1742-6596/1205/1/012017	journal	April 2019
Porting LASG/ IAP Climate System Ocean Model to Gpus Using OpenAcc Jiang, Jinrong; Lin, Pengfei; Wang, Joey IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2932443	journal	January 2019
Comparing Julia to Performance Portable Parallel Programming Models for HPC Lin, Wei-Chen; McIntosh-Smith, Simon 2021 International Workshop on Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) https://doi.org/10.1109/PMBS54543.2021.00016	conference	November 2021
Effective Extensible Programming: Unleashing Julia on GPUs Besard, Tim; Foket, Christophe; De Sutter, Bjorn IEEE Transactions on Parallel and Distributed Systems, Vol. 30, Issue 4 https://doi.org/10.1109/TPDS.2018.2872064	journal	April 2019
Kokkos 3: Programming Model Extensions for the Exascale Era Trott, Christian; Lebrun-Grandie, Damien; Arndt, Daniel IEEE Transactions on Parallel and Distributed Systems https://doi.org/10.1109/TPDS.2021.3097283	journal	January 2021
Julia: A Fresh Approach to Numerical Computing Bezanson, Jeff; Edelman, Alan; Karpinski, Stefan SIAM Review, Vol. 59, Issue 1 https://doi.org/10.1137/141000671	journal	January 2017
Numba: a LLVM-based Python JIT compiler Lam, Siu Kwan; Pitrou, Antoine; Seibert, Stanley Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC - LLVM '15 https://doi.org/10.1145/2833157.2833162	conference	January 2015
MPI.jl: Julia bindings for the Message Passing Interface Byrne, Simon; Wilcox, Lucas C.; Churavy, Valentin JuliaCon Proceedings, Vol. 1, Issue 1, p. 68 https://doi.org/10.21105/jcon.00068	journal	July 2021
Oceananigans.jl: Fast and friendly geophysical fluid dynamics on GPUs Ramadhan, Ali; Wagner, Gregory; Hill, Chris Journal of Open Source Software, Vol. 5, Issue 53 https://doi.org/10.21105/joss.02018	journal	September 2020
Scalability and some optimization of the Finite-volumE Sea ice–Ocean Model, Version 2.0 (FESOM2) Koldunov, Nikolay V.; Aizinger, Vadym; Rakowsky, Natalja Geoscientific Model Development, Vol. 12, Issue 9 https://doi.org/10.5194/gmd-12-3991-2019	journal	January 2019
POM.gpu-v1.0: a GPU-based Princeton Ocean Model Xu, S.; Huang, X.; Oey, L. -Y. Geoscientific Model Development, Vol. 8, Issue 9 https://doi.org/10.5194/gmd-8-2815-2015	journal	January 2015

Similar Records

Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes

Conference · Mon May 01 00:00:00 EDT 2023 · OSTI ID:1994693

Creating an MPAS Ocean Shallow Water Core in Julia [Slides]

Technical Report · Sat Nov 06 00:00:00 EDT 2021 · OSTI ID:1829622

Comparing the Performance of Julia on CPUs versus GPUs and Julia-MPI versus Fortran-MPI: a case study with MPAS-Ocean (Version 7.1)

Citation Formats

References (28)

Similar Records

Related Subjects