NekRS, a GPU-accelerated spectral element Navier–Stokes solver

Fischer, Paul; Kerkemeier, Stefan; Min, Misun; Lan, Yu-Hsiang; Phillips, Malachi; Rathnayake, Thilina; Merzari, Elia; Tomboulides, Ananias; Karakus, Ali; Chalmers, Noel; Warburton, Tim

doi:10.1016/j.parco.2022.102982

NekRS, a GPU-accelerated spectral element Navier–Stokes solver

Journal Article · Tue Oct 18 00:00:00 EDT 2022 · Parallel Computing

DOI:https://doi.org/10.1016/j.parco.2022.102982· OSTI ID:2429555

Fischer, Paul ^[1]; Kerkemeier, Stefan ^[2]; ^[2]; Lan, Yu-Hsiang ^[1]; ^[3]; Rathnayake, Thilina ^[3]; Merzari, Elia ^[4]; Tomboulides, Ananias ^[5]; Karakus, Ali ^[6]; ^[7]; Warburton, Tim ^[8]

Argonne National Laboratory (ANL), Argonne, IL (United States); Univ. of Illinois at Urbana-Champaign, IL (United States)
Argonne National Laboratory (ANL), Argonne, IL (United States)
Univ. of Illinois at Urbana-Champaign, IL (United States)
Pennsylvania State Univ., University Park, PA (United States); Argonne National Laboratory (ANL), Argonne, IL (United States)
Aristotle University of Thessaloniki (Greece); Argonne National Laboratory (ANL), Argonne, IL (United States)
Middle East Technical University, Ankara (Turkey)
Advanced Micro Devices Inc., Austin, TX (United States). AMD Research
Virginia Polytechnic Inst. and State Univ. (Virginia Tech), Blacksburg, VA (United States)

The development of NekRS, a GPU-oriented thermal-fluids simulation code based on the spectral element method (SEM) is described. For performance portability, the code is based on the open concurrent compute abstraction and leverages scalable developments in the SEM code Nek5000 and in libParanumal, which is a library of high-performance kernels for high-order discretizations and PDE-based miniapps. Critical performance sections of the Navier–Stokes time advancement are addressed. Performance results on several platforms are presented here, including scaling to 27,648 V100s on OLCF Summit, for calculations of up to 60B gridpoints.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science

Grant/Contract Number:: AC02-06CH11357; AC05-00OR22725

OSTI ID:: 2429555

Journal Information:: Parallel Computing, Journal Name: Parallel Computing Vol. 114; ISSN 0167-8191

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (36)

Efficiency of high-performance discontinuous Galerkin spectral element methods for under-resolved turbulent incompressible flows: High-performance discontinuous Galerkin for turbulent flows Fehn, Niklas; Wall, Wolfgang A.; Kronbichler, Martin International Journal for Numerical Methods in Fluids, Vol. 88, Issue 1 https://doi.org/10.1002/fld.4511	journal	May 2018
A Legendre spectral element method for the Stefan problem Rønquist, Einar M.; Patera, Anthony T. International Journal for Numerical Methods in Engineering, Vol. 24, Issue 12 https://doi.org/10.1002/nme.1620241204	journal	December 1987
Fast Parallel Direct Solvers for Coarse Grid Problems Tufo, H. M.; Fischer, P. F. Journal of Parallel and Distributed Computing, Vol. 61, Issue 2 https://doi.org/10.1006/jpdc.2000.1676	journal	February 2001
Efficient removal of boundary-divergence errors in time-splitting methods Tomboulides, A. G.; Israeli, M.; Karniadakis, G. E. Journal of Scientific Computing, Vol. 4, Issue 3 https://doi.org/10.1007/BF01061059	journal	September 1989
Boundary conditions for incompressible flows Orszag, Steven A.; Israeli, Moshe; Deville, Michel O. Journal of Scientific Computing, Vol. 1, Issue 1 https://doi.org/10.1007/BF01061454	journal	January 1986
An Operator-integration-factor splitting method for time-dependent problems: Application to incompressible fluid flow Maday, Y.; Patera, Anthony T.; R�nquist, Einar M. Journal of Scientific Computing, Vol. 5, Issue 4 https://doi.org/10.1007/BF01063118	journal	December 1990
Hybrid Multigrid/Schwarz Algorithms for the Spectral Element Method Lottes, James W.; Fischer, Paul F. Journal of Scientific Computing, Vol. 24, Issue 1 https://doi.org/10.1007/s10915-004-4787-3	journal	July 2005
Stabilization of the Spectral Element Method in Convection Dominated Flows by Recovery of Skew-Symmetry Malm, Johan; Schlatter, Philipp; Fischer, Paul F. Journal of Scientific Computing, Vol. 57, Issue 2 https://doi.org/10.1007/s10915-013-9704-1	journal	March 2013
A Characteristic-Based Spectral Element Method for Moving-Domain Problems Patel, Saumil; Fischer, Paul; Min, Misun Journal of Scientific Computing, Vol. 79, Issue 1 https://doi.org/10.1007/s10915-018-0876-6	journal	December 2018
Spectral methods for problems in complex geometries Orszag, Steven A. Journal of Computational Physics, Vol. 37, Issue 1 https://doi.org/10.1016/0021-9991(80)90005-4	journal	August 1980
A spectral element method for fluid dynamics: Laminar flow in a channel expansion Patera, Anthony T. Journal of Computational Physics, Vol. 54, Issue 3 https://doi.org/10.1016/0021-9991(84)90128-1	journal	June 1984
Optimal error analysis of spectral methods with emphasis on non-constant coefficients and deformed geometries Maday, Yvon; Rønquist, Einar M. Computer Methods in Applied Mechanics and Engineering, Vol. 80, Issue 1-3 https://doi.org/10.1016/0045-7825(90)90016-F	journal	June 1990
Parallel multigrid smoothing: polynomial versus Gauss–Seidel Adams, Mark; Brezina, Marian; Hu, Jonathan Journal of Computational Physics, Vol. 188, Issue 2 https://doi.org/10.1016/S0021-9991(03)00194-3	journal	July 2003
Projection techniques for iterative solution of with successive right-hand sides Fischer, Paul F. Computer Methods in Applied Mechanics and Engineering, Vol. 163, Issue 1-4 https://doi.org/10.1016/S0045-7825(98)00012-7	journal	September 1998
A GPU accelerated aggregation algebraic multigrid method Gandham, Rajesh; Esler, Kenneth; Zhang, Yongpeng Computers & Mathematics with Applications, Vol. 68, Issue 10 https://doi.org/10.1016/j.camwa.2014.08.022	journal	November 2014
An overview of projection methods for incompressible flows Guermond, J. L.; Minev, P.; Shen, Jie Computer Methods in Applied Mechanics and Engineering, Vol. 195, Issue 44-47 https://doi.org/10.1016/j.cma.2005.10.010	journal	September 2006
Invariant analysis of the Reynolds stress tensor for a nuclear fuel assembly with spacer grid and split type vanes Busco, Giacomo; Merzari, Elia; Hassan, Yassin A. International Journal of Heat and Fluid Flow, Vol. 77 https://doi.org/10.1016/j.ijheatfluidflow.2019.04.006	journal	June 2019
Dispersive behaviour of high order finite element schemes for the one-way wave equation Ainsworth, Mark Journal of Computational Physics, Vol. 259 https://doi.org/10.1016/j.jcp.2013.11.003	journal	February 2014
GPU accelerated spectral finite elements on all-hex meshes Remacle, J. -F.; Gandham, R.; Warburton, T. Journal of Computational Physics, Vol. 324 https://doi.org/10.1016/j.jcp.2016.08.005	journal	November 2016
A GPU accelerated discontinuous Galerkin incompressible flow solver Karakus, A.; Chalmers, N.; Świrydowicz, K. Journal of Computational Physics, Vol. 390 https://doi.org/10.1016/j.jcp.2019.04.010	journal	August 2019
Scaling to the stars – a linearly scaling elliptic solver for p-multigrid Huismann, Immo; Stiller, Jörg; Fröhlich, Jochen Journal of Computational Physics, Vol. 398 https://doi.org/10.1016/j.jcp.2019.108868	journal	December 2019
OpenACC acceleration for the P N – P N - 2 algorithm in Nek5000 Otero, Evelyn; Gong, Jing; Min, Misun Journal of Parallel and Distributed Computing, Vol. 132 https://doi.org/10.1016/j.jpdc.2019.05.010	journal	October 2019
Numerical Simulation of Low Mach Number Reactive Flows Tomboulides, A. G.; Lee, J. C. Y.; Orszag, S. A. Journal of Scientific Computing, Vol. 12, Issue 2, p. 139-167 https://doi.org/10.1023/A:1025669715376	journal	January 1997
Petascale algorithms for reactor hydrodynamics Fischer, P.; Lottes, J.; Pointer, D. Journal of Physics: Conference Series, Vol. 125 https://doi.org/10.1088/1742-6596/125/1/012076	journal	July 2008
Partitioning Sparse Matrices with Eigenvectors of Graphs Pothen, Alex; Simon, Horst D.; Liou, Kang-Pu SIAM Journal on Matrix Analysis and Applications, Vol. 11, Issue 3 https://doi.org/10.1137/0611030	journal	July 1990
GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems Saad, Youcef; Schultz, Martin H. SIAM Journal on Scientific and Statistical Computing, Vol. 7, Issue 3 https://doi.org/10.1137/0907058	journal	July 1986
Finite-Element Preconditioning of G-NI Spectral Methods Canuto, Claudio; Gervasio, Paola; Quarteroni, Alfio SIAM Journal on Scientific Computing, Vol. 31, Issue 6 https://doi.org/10.1137/090746367	journal	January 2010
Tuning Spectral Element Preconditioners for Parallel Scalability on GPUs Phillips, Malachi; Kerkemeier, Stefan; Fischer, Paul Proceedings of the 2022 SIAM Conference on Parallel Processing for Scientific Computing (PP) https://doi.org/10.1137/1.9781611977141.4	book	January 2022
Scalable Low-Order Finite Element Preconditioners for High-Order Spectral Element Poisson Solvers Bello-Maldonado, Pedro D.; Fischer, Paul F. SIAM Journal on Scientific Computing, Vol. 41, Issue 5 https://doi.org/10.1137/18M1194997	journal	January 2019
A Restricted Additive Schwarz Preconditioner for General Sparse Linear Systems Cai, Xiao-Chuan; Sarkis, Marcus SIAM Journal on Scientific Computing, Vol. 21, Issue 2 https://doi.org/10.1137/S106482759732678X	journal	January 1999
Flexible Conjugate Gradients Notay, Yvan SIAM Journal on Scientific Computing, Vol. 22, Issue 4 https://doi.org/10.1137/S1064827599362314	journal	January 2000
Multigrid for Matrix-Free High-Order Finite Element Computations on Graphics Processors Kronbichler, Martin; Ljungkvist, Karl ACM Transactions on Parallel Computing, Vol. 6, Issue 1 https://doi.org/10.1145/3322813	journal	May 2019
An MPI/OpenACC implementation of a high-order electromagnetics solver with GPUDirect communication Otten, Matthew; Gong, Jing; Mametjanov, Azamat The International Journal of High Performance Computing Applications, Vol. 30, Issue 3 https://doi.org/10.1177/1094342015626584	journal	July 2016
Acceleration of tensor-product operations for high-order finite element methods Świrydowicz, Kasia; Chalmers, Noel; Karakus, Ali The International Journal of High Performance Computing Applications, Vol. 33, Issue 4 https://doi.org/10.1177/1094342018816368	journal	September 2018
Scaling Limits for PDE-Based Simulation (Invited) Fischer, Paul F. 22nd AIAA Computational Fluid Dynamics Conference https://doi.org/10.2514/6.2015-3049	conference	June 2015
Comparison of accurate methods for the integration of hyperbolic equations Kreiss, Heinz-Otto; Oliger, Joseph Tellus, Vol. 24, Issue 3 https://doi.org/10.3402/tellusa.v24i3.10634	journal	January 1972

Similar Records

Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations

Journal Article · Mon Jul 18 00:00:00 EDT 2016 · Journal of Supercomputing · OSTI ID:1565549

Nek5000/RS Performance on Advanced GPU Architectures

Technical Report · Fri Sep 30 00:00:00 EDT 2022 · OSTI ID:1894022

High performance sparse multifrontal solvers on modern GPUs

Journal Article · Fri Feb 04 19:00:00 EST 2022 · Parallel Computing · OSTI ID:1960514

Related Subjects

97 MATHEMATICS AND COMPUTING
Exascale applications
GPU
Incompressible Navier-Stokes
Nek5000
NekRS
OCCA
Performance
Scalability
Spectral element method
libParanumal

NekRS, a GPU-accelerated spectral element Navier–Stokes solver

Citation Formats

References (36)

Similar Records

Related Subjects