An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface

Kang, Hyun-Gyu; Tuminaro, Raymond S.; Prokopenko, Andrey; Johnson, Seth R.; Salinger, Andrew G.; Evans, Katherine J.

doi:10.1177/10943420231205601

An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface

Journal Article · Fri Nov 17 23:00:00 EST 2023 · International Journal of High Performance Computing Applications

DOI:https://doi.org/10.1177/10943420231205601· OSTI ID:2251644

^[1]; Tuminaro, Raymond S. ^[2]; ^[1]; ^[1]; ^[3]; ^[1]

Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sandia National Lab. (SNL-CA), Livermore, CA (United States)
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Here, we demonstrate use of a modern Fortran solver interface to manage solver algorithms for an implicit barotropic mode solver in the Model for Predictions Across Scales-Ocean (MPAS-O). ForTrilinos, a Fortran interface to Trilinos that contains a large collection of solver capabilities written in C++, has been implemented in MPAS-O to provide access to a suite of linear solver options. By virtue of the simplified wrapper and interface generator (SWIG) automation tool that generates modern Fortran interfaces to C++ code, we were able to implement the Fortran solver interface in MPAS-O using a familiar Fortran coding style while minimizing performance degradation. The ForTrilinos solver interface is written within MPAS-O’s time stepping modules as a subroutine in conjunction with MPAS-O code. Applied to an idealized ocean and a high-resolution realistic ocean test case, parallel performance of ForTrilinos solvers is examined. It is found that parallel scalability of the ForTrilinos solvers is highly dependent on the number of global synchronization points per solver iteration in each iterative solver algorithm. ForTrilinos solvers perform best compared to the Fortran hand-crafted (FHC) solver when the amount of work per processor is large enough. However, parallel scalability is better with the FHC solver and so when the work per core is modest FHC outperforms ForTrilinos. The intercomparison between the ForTrilinos and FHC solvers reveals that this performance hit in the ForTrilinos solver mostly comes from the global synchronization process, while suggesting that the matrix-vector multiplication process in the FHC solver needs to be optimized for better performance.

View Accepted Manuscript (DOE)

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Los Alamos National Laboratory (LANL), Los Alamos, NM (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER); USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 2251644

Journal Information:: International Journal of High Performance Computing Applications, Journal Name: International Journal of High Performance Computing Applications Journal Issue: 3 Vol. 38; ISSN 1094-3420

Publisher:: SAGECopyright Statement

Country of Publication:: United States

Language:: English

References (49)

Extreme scalability challenges in micro‐finite element simulations of human bone Bekas, C.; Curioni, A.; Arbenz, P. Concurrency and Computation: Practice and Experience, Vol. 22, Issue 16 https://doi.org/10.1002/cpe.1591	journal	April 2010
A Scalable and Adaptable Solution Framework within Components of the Community Climate System Model Evans, Katherine J.; Rouson, Damian W. I.; Salinger, Andrew G. Lecture Notes in Computer Science https://doi.org/10.1007/978-3-642-01973-9_37	book	January 2009
The role of horizontal resolution in simulating drivers of the global hydrological cycle Demory, Marie-Estelle; Vidale, Pier Luigi; Roberts, Malcolm J. Climate Dynamics, Vol. 42, Issue 7-8 https://doi.org/10.1007/s00382-013-1924-4	journal	September 2013
Enhancement of the southward return flow of the Atlantic Meridional Overturning Circulation by data assimilation and its influence in an assimilative ocean simulation forced by CORE-II atmospheric forcing Fujii, Yosuke; Tsujino, Hiroyuki; Toyoda, Takahiro Climate Dynamics, Vol. 49, Issue 3 https://doi.org/10.1007/s00382-015-2780-1	journal	August 2015
Parallel matrix transpose algorithms on distributed memory concurrent computers Choi, Jaeyoung; Dongarra, Jack J.; Walker, David W. Parallel Computing, Vol. 21, Issue 9 https://doi.org/10.1016/0167-8191(95)00016-H	journal	September 1995
Preconditioning techniques for nonsymmetric and indefinite linear systems Saad, Youcef Journal of Computational and Applied Mathematics, Vol. 24, Issue 1-2 https://doi.org/10.1016/0377-0427(88)90345-7	journal	November 1988
The regional oceanic modeling system (ROMS): a split-explicit, free-surface, topography-following-coordinate oceanic model Shchepetkin, Alexander F.; McWilliams, James C. Ocean Modelling, Vol. 9, Issue 4 https://doi.org/10.1016/j.ocemod.2004.08.002	journal	January 2005
Spurious dianeutral mixing and the role of momentum closure Ilıcak, Mehmet; Adcroft, Alistair J.; Griffies, Stephen M. Ocean Modelling, Vol. 45-46 https://doi.org/10.1016/j.ocemod.2011.10.003	journal	January 2012
A multi-resolution approach to global ocean modeling Ringler, Todd; Petersen, Mark; Higdon, Robert L. Ocean Modelling, Vol. 69 https://doi.org/10.1016/j.ocemod.2013.04.010	journal	September 2013
The importance of eddy permitting model resolution for simulation of the heat budget of tropical instability waves Graham, Tim Ocean Modelling, Vol. 79 https://doi.org/10.1016/j.ocemod.2014.04.005	journal	July 2014
Evaluation of the arbitrary Lagrangian–Eulerian vertical coordinate method in the MPAS-Ocean model Petersen, Mark R.; Jacobsen, Douglas W.; Ringler, Todd D. Ocean Modelling, Vol. 86 https://doi.org/10.1016/j.ocemod.2014.12.004	journal	February 2015
An assessment of Antarctic Circumpolar Current and Southern Ocean meridional overturning circulation during 1958–2007 in a suite of interannual CORE-II simulations Farneti, Riccardo; Downes, Stephanie M.; Griffies, Stephen M. Ocean Modelling, Vol. 93 https://doi.org/10.1016/j.ocemod.2015.07.009	journal	September 2015
The communication-hiding pipelined BiCGstab method for the parallel solution of large unsymmetric linear systems Cools, S.; Vanroose, W. Parallel Computing, Vol. 65 https://doi.org/10.1016/j.parco.2017.04.005	journal	July 2017
Why Restricted Additive Schwarz Converges Faster than Additive Schwarz Efstathiou, Evridiki; Gander, Martin J. BIT Numerical Mathematics, Vol. 43, Issue 5 https://doi.org/10.1023/B:BITN.0000014563.33622.1d	journal	January 2003
An Evaluation of the Ocean and Sea Ice Climate of E3SM Using MPAS and Interannual CORE‐II Forcing Petersen, Mark R.; Asay‐Davis, Xylar S.; Berres, Anne S. Journal of Advances in Modeling Earth Systems, Vol. 11, Issue 5 https://doi.org/10.1029/2018MS001373	journal	May 2019
The DOE E3SM Coupled Model Version 1: Overview and Evaluation at Standard Resolution Golaz, Jean‐Christophe; Caldwell, Peter M.; Van Roekel, Luke P. Journal of Advances in Modeling Earth Systems, Vol. 11, Issue 7 https://doi.org/10.1029/2018MS001603	journal	July 2019
The DOE E3SM Coupled Model Version 1: Description and Results at High Resolution Caldwell, Peter M.; Mametjanov, Azamat; Tang, Qi Journal of Advances in Modeling Earth Systems, Vol. 11, Issue 12 https://doi.org/10.1029/2019MS001870	journal	December 2019
Characterizing Tropical Cyclones in the Energy Exascale Earth System Model Version 1 Balaguru, Karthik; Leung, L. Ruby; Van Roekel, Luke P. Journal of Advances in Modeling Earth Systems, Vol. 12, Issue 8 https://doi.org/10.1029/2019MS002024	journal	August 2020
A Scalable Semi‐Implicit Barotropic Mode Solver for the MPAS‐Ocean Kang, Hyun‐Gyu; Evans, Katherine J.; Petersen, Mark R. Journal of Advances in Modeling Earth Systems, Vol. 13, Issue 4 https://doi.org/10.1029/2020MS002238	journal	April 2021
Separating Physics and Dynamics Grids for Improved Computational Efficiency in Spectral Element Earth System Models Hannah, Walter M.; Bradley, Andrew M.; Guba, Oksana Journal of Advances in Modeling Earth Systems, Vol. 13, Issue 7 https://doi.org/10.1029/2020MS002419	journal	June 2021
Global Dust Cycle and Direct Radiative Effect in E3SM Version 1: Impact of Increasing Model Resolution Feng, Y.; Wang, H.; Rasch, P. J. Journal of Advances in Modeling Earth Systems, Vol. 14, Issue 7 https://doi.org/10.1029/2021MS002909	journal	July 2022
The DOE E3SM Model Version 2: Overview of the Physical Model and Initial Model Evaluation Golaz, Jean‐Christophe; Van Roekel, Luke P.; Zheng, Xue Journal of Advances in Modeling Earth Systems, Vol. 14, Issue 12 https://doi.org/10.1029/2022MS003156	journal	December 2022
Implicit free-surface method for the Bryan-Cox-Semtner ocean model Dukowicz, John K.; Smith, Richard D. Journal of Geophysical Research, Vol. 99, Issue C4 https://doi.org/10.1029/93JC03455	journal	January 1994
A minimum overlap restricted additive Schwarz preconditioner and applications in 3D flow simulations Cai, Xiao-Chuan; Farhat, Charbel; Sarkis, Marcus https://doi.org/10.1090/conm/218/03046	book	January 1998
Comparison of parallel solvers for Moving Particle Semi-Implicit method Duan, Guangtao; Chen, Bin Engineering Computations, Vol. 32, Issue 3 https://doi.org/10.1108/EC-02-2014-0029	journal	May 2015
The improved BiCGStab method for large and sparse unsymmetric linear systems on parallel distributed memory architectures Yang, L. T.; Brent, R. P. Fifth International Conference on Algorithms and Architectures for Parallel Processing, 2002. Proceedings. https://doi.org/10.1109/ICAPP.2002.1173595	conference	October 2002
Towards Extreme-Scale Simulations with Next-Generation Trilinos: A Low Mach Fluid Application Case Study Lin, Paul; Bettencourt, Matthew; Domino, Stefan 2014 IEEE International Parallel & Distributed Processing Symposium Workshops https://doi.org/10.1109/IPDPSW.2014.166	conference	May 2014
Automated Fortran–C++ Bindings for Large-Scale Scientific Applications Johnson, Seth R.; Prokopenko, Andrey; Evans, Katherine J. Computing in Science & Engineering, Vol. 22, Issue 5 https://doi.org/10.1109/MCSE.2019.2924204	journal	September 2020
A Matrix Dependent/Algebraic Multigrid Approach for Extruded Meshes with Applications to Ice Sheet Modeling Tuminaro, R.; Perego, M.; Tezaur, I. SIAM Journal on Scientific Computing, Vol. 38, Issue 5 https://doi.org/10.1137/15M1040839	journal	January 2016
A Parallel Implementation of a Two-Level Overlapping Schwarz Method with Energy-Minimizing Coarse Space Based on Trilinos Heinlein, Alexander; Klawonn, Axel; Rheinbach, Oliver SIAM Journal on Scientific Computing, Vol. 38, Issue 6 https://doi.org/10.1137/16M1062843	journal	January 2016
Communication-Efficient Property Preservation in Tracer Transport Bradley, Andrew M.; Bosler, Peter A.; Guba, Oksana SIAM Journal on Scientific Computing, Vol. 41, Issue 3 https://doi.org/10.1137/18M1165414	journal	January 2019
Conservative Multimoment Transport along Characteristics for Discontinuous Galerkin Methods Bosler, Peter A.; Bradley, Andrew M.; Taylor, Mark A. SIAM Journal on Scientific Computing, Vol. 41, Issue 4 https://doi.org/10.1137/18M1165943	journal	January 2019
A Restricted Additive Schwarz Preconditioner for General Sparse Linear Systems Cai, Xiao-Chuan; Sarkis, Marcus SIAM Journal on Scientific Computing, Vol. 21, Issue 2 https://doi.org/10.1137/S106482759732678X	journal	January 1999
Partitioning Rectangular and Structurally Unsymmetric Sparse Matrices for Parallel Processing Hendrickson, Bruce; Kolda, Tamara G. SIAM Journal on Scientific Computing, Vol. 21, Issue 6 https://doi.org/10.1137/S1064827598341475	journal	January 2000
An overview of the Trilinos project Heroux, Michael A.; Phipps, Eric T.; Salinger, Andrew G. ACM Transactions on Mathematical Software, Vol. 31, Issue 3 https://doi.org/10.1145/1089014.1089021	journal	September 2005
Exploring Capabilities within ForTrilinos by Solving the 3D Burgers Equation Morris, Karla; Rouson, Damian W. I.; Lemaster, M. Nicole Scientific Programming, Vol. 20, Issue 3 https://doi.org/10.1155/2012/378791	journal	January 2012
Accuracy Analysis of a Spectral Element Atmospheric Model Using a Fully Implicit Solution Framework Evans, Katherine J.; Taylor, Mark A.; Drake, John B. Monthly Weather Review, Vol. 138, Issue 8 https://doi.org/10.1175/2010MWR3288.1	journal	August 2010
Simulation of ENSO Teleconnections to Precipitation Extremes over the United States in the High-Resolution Version of E3SM Mahajan, Salil; Tang, Qi; Keen, Noel D. Journal of Climate, Vol. 35, Issue 11 https://doi.org/10.1175/JCLI-D-20-1011.1	journal	June 2022
Three-Dimensional Baroclinic Eddies in the Ocean: Evolution, Propagation, Overall Structures, and Angular Models Liao, Guanghong; Xu, Xiaohua; Dong, Changming Journal of Physical Oceanography, Vol. 49, Issue 10 https://doi.org/10.1175/JPO-D-18-0237.1	journal	October 2019
A modern solver interface to manage solution algorithms in the Community Earth System Model Evans, Katherine J.; Salinger, Andrew G.; Worley, Patrick H. The International Journal of High Performance Computing Applications, Vol. 26, Issue 1 https://doi.org/10.1177/1094342011435159	journal	February 2012
Ifpack2 User's Guide 1.0 Prokopenko, Andrey; Siefert, Christopher; Hu, Jonathan J. https://doi.org/10.2172/1259544	report	May 2016
ForTrilinos Design Document Young, Mitchell T.; Johnson, Seth R.; Prokopenko, Andrey V. https://doi.org/10.2172/1424434	report	August 2017
MPAS-Ocean Model User's Guide Version 6.0 Petersen, Mark Roger; Asay-Davis, Xylar Storm; Jacobsen, Douglas William https://doi.org/10.2172/1434462	report	April 2018
MueLu User's Guide Berger-Vergiat, Luc; Glusa, Christian Alexander; Hu, Jonathan J. https://doi.org/10.2172/1491860	report	January 2019
Comparative study of conjugate gradient algorithms performance on the example of steady-state axisymmetric heat transfer problem Ocłoń, Paweł; Łopata, Stanisław; Nowak, Marzena Archives of Thermodynamics, Vol. 34, Issue 3 https://doi.org/10.2478/aoter-2013-0013	journal	September 2013
Amesos2 and Belos: Direct and Iterative Solvers for Large Sparse Linear Systems Bavier, Eric; Hoemmen, Mark; Rajamanickam, Sivasankaran Scientific Programming https://doi.org/10.3233/SPR-2012-0352	journal	January 2012
Regionally refined test bed in E3SM atmosphere model version 1 (EAMv1) and applications for high-resolution modeling Tang, Qi; Klein, Stephen A.; Xie, Shaocheng Geoscientific Model Development, Vol. 12, Issue 7 https://doi.org/10.5194/gmd-12-2679-2019	journal	January 2019
Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization Eyring, Veronika; Bony, Sandrine; Meehl, Gerald A. Geoscientific Model Development, Vol. 9, Issue 5 https://doi.org/10.5194/gmd-9-1937-2016	journal	January 2016
Southern Ocean polynyas and dense water formation in a high-resolution, coupled Earth System Model Jeong, Hyein; Turner, Adrian K.; Roberts, Andrew F. The Cryosphere https://doi.org/10.5194/tc-2022-133	preprint	August 2022

Similar Records

Five years of ForTrilinos ECP

Technical Report · Mon Aug 01 00:00:00 EDT 2022 · OSTI ID:1909128

A Scalable Semi-Implicit Barotropic Mode Solver for the MPAS-Ocean

Journal Article · Wed Apr 07 00:00:00 EDT 2021 · Journal of Advances in Modeling Earth Systems · OSTI ID:1782059

ForTrilinos

Software · Sat Dec 30 19:00:00 EST 2017 · OSTI ID:code-37137

Related Subjects

97 MATHEMATICS AND COMPUTING

An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface

Citation Formats

References (49)

Similar Records

Related Subjects