Energy Exascale Computational Fluid Dynamics Simulations With the Spectral Element Method

Merzari, Elia; Coppo Leite, Victor; Fang, Jun; Shaver, Dillon; Min, Misun; Kerkemeier, Stefan; Fischer, Paul; Tomboulides, Ananias

doi:10.1115/1.4064659

Energy Exascale Computational Fluid Dynamics Simulations With the Spectral Element Method

Journal Article · Thu Feb 22 00:00:00 EST 2024 · Journal of Fluids Engineering

DOI:https://doi.org/10.1115/1.4064659· OSTI ID:2572333

Merzari, Elia ^[1]; Coppo Leite, Victor ^[1]; ^[2]; Shaver, Dillon ^[2]; Min, Misun ^[2]; Kerkemeier, Stefan ^[2]; Fischer, Paul ^[3]; Tomboulides, Ananias ^[4]

Pennsylvania State Univ., University Park, PA (United States)
Argonne National Laboratory (ANL), Argonne, IL (United States)
Univ. of Illinois at Urbana-Champaign, IL (United States)
Aristotle University of Thessaloniki (Greece)

Development and application of the open-source GPU-based fluid-thermal simulation code, NekRS, are described. Time advancement is based on an efficient kth-order accurate timesplit formulation coupled with scalable iterative solvers. Spatial discretization is based on the high-order spectral element method (SEM), which affords the use of fast, low-memory, matrix-free operator evaluation. Further, recent developments include support for nonconforming meshes using overset grids and for GPU-based Lagrangian particle tracking. Results of large-eddy simulations of atmospheric boundary layers for wind-energy applications as well as extensive nuclear energy applications are presented.

View Accepted Manuscript (DOE)

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States). Argonne Leadership Computing Facility (ALCF); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-06CH11357; AC05-00OR22725

OSTI ID:: 2572333

Journal Information:: Journal of Fluids Engineering, Journal Name: Journal of Fluids Engineering Journal Issue: 4 Vol. 146; ISSN 0098-2202; ISSN 1528-901X

Publisher:: ASMECopyright Statement

Country of Publication:: United States

Language:: English

References (40)

Efficiency of high-performance discontinuous Galerkin spectral element methods for under-resolved turbulent incompressible flows: High-performance discontinuous Galerkin for turbulent flows Fehn, Niklas; Wall, Wolfgang A.; Kronbichler, Martin International Journal for Numerical Methods in Fluids, Vol. 88, Issue 1 https://doi.org/10.1002/fld.4511	journal	May 2018
Gmsh: A 3-D finite element mesh generator with built-in pre- and post-processing facilities Geuzaine, Christophe; Remacle, Jean-François International Journal for Numerical Methods in Engineering, Vol. 79, Issue 11 https://doi.org/10.1002/nme.2579	journal	September 2009
An Overlapping Schwarz Method for Spectral Element Solution of the Incompressible Navier–Stokes Equations Fischer, Paul F. Journal of Computational Physics, Vol. 133, Issue 1 https://doi.org/10.1006/jcph.1997.5651	journal	May 1997
A subgrid-scale model for large-eddy simulation of planetary boundary-layer flows Sullivan, Peter P.; McWilliams, James C.; Moeng, Chin-Hoh Boundary-Layer Meteorology, Vol. 71, Issue 3 https://doi.org/10.1007/BF00713741	journal	November 1994
Efficient removal of boundary-divergence errors in time-splitting methods Tomboulides, A. G.; Israeli, M.; Karniadakis, G. E. Journal of Scientific Computing, Vol. 4, Issue 3 https://doi.org/10.1007/BF01061059	journal	September 1989
Boundary conditions for incompressible flows Orszag, Steven A.; Israeli, Moshe; Deville, Michel O. Journal of Scientific Computing, Vol. 1, Issue 1 https://doi.org/10.1007/BF01061454	journal	January 1986
An Operator-integration-factor splitting method for time-dependent problems: Application to incompressible fluid flow Maday, Y.; Patera, Anthony T.; R�nquist, Einar M. Journal of Scientific Computing, Vol. 5, Issue 4 https://doi.org/10.1007/BF01063118	journal	December 1990
On the transport-diffusion algorithm and its applications to the Navier-Stokes equations Pironneau, O. Numerische Mathematik, Vol. 38, Issue 3 https://doi.org/10.1007/BF01396435	journal	October 1982
An Intercomparison of Large-Eddy Simulations of the Stable Boundary Layer Beare, Robert J.; Macvean, Malcolm K.; Holtslag, Albert A. M. Boundary-Layer Meteorology, Vol. 118, Issue 2 https://doi.org/10.1007/s10546-004-2820-6	journal	February 2006
Hybrid Multigrid/Schwarz Algorithms for the Spectral Element Method Lottes, James W.; Fischer, Paul F. Journal of Scientific Computing, Vol. 24, Issue 1 https://doi.org/10.1007/s10915-004-4787-3	journal	July 2005
Stabilization of the Spectral Element Method in Convection Dominated Flows by Recovery of Skew-Symmetry Malm, Johan; Schlatter, Philipp; Fischer, Paul F. Journal of Scientific Computing, Vol. 57, Issue 2 https://doi.org/10.1007/s10915-013-9704-1	journal	March 2013
A Characteristic-Based Spectral Element Method for Moving-Domain Problems Patel, Saumil; Fischer, Paul; Min, Misun Journal of Scientific Computing, Vol. 79, Issue 1 https://doi.org/10.1007/s10915-018-0876-6	journal	December 2018
Subgrid scale model for finite difference simulations of turbulent flows in plane channels and annuli Schumann, U. Journal of Computational Physics, Vol. 18, Issue 4 https://doi.org/10.1016/0021-9991(75)90093-5	journal	August 1975
Spectral methods for problems in complex geometries Orszag, Steven A. Journal of Computational Physics, Vol. 37, Issue 1 https://doi.org/10.1016/0021-9991(80)90005-4	journal	August 1980
A spectral element method for fluid dynamics: Laminar flow in a channel expansion Patera, Anthony T. Journal of Computational Physics, Vol. 54, Issue 3 https://doi.org/10.1016/0021-9991(84)90128-1	journal	June 1984
OpenMC: A state-of-the-art Monte Carlo code for research and development Romano, Paul K.; Horelik, Nicholas E.; Herman, Bryan R. Annals of Nuclear Energy, Vol. 82 https://doi.org/10.1016/j.anucene.2014.07.048	journal	August 2015
Nonconforming Schwarz-spectral element methods for incompressible flow Mittal, Ketan; Dutta, Som; Fischer, Paul Computers & Fluids, Vol. 191 https://doi.org/10.1016/j.compfluid.2019.104237	journal	September 2019
Time-resolved particle image velocimetry measurements of a single impinging jet in the upper plenum of a scaled facility of high temperature gas-cooled reactors Alwafi, Anas; Nguyen, Thien; Hassan, Yassin International Journal of Heat and Fluid Flow, Vol. 76 https://doi.org/10.1016/j.ijheatfluidflow.2019.02.003	journal	April 2019
A spectrally accurate method for overlapping grid solution of incompressible Navier–Stokes equations Merrill, Brandon E.; Peet, Yulia T.; Fischer, Paul F. Journal of Computational Physics, Vol. 307 https://doi.org/10.1016/j.jcp.2015.11.057	journal	February 2016
Implementation, capabilities, and benchmarking of Shift, a massively parallel Monte Carlo radiation transport code Pandya, Tara M.; Johnson, Seth R.; Evans, Thomas M. Journal of Computational Physics, Vol. 308 https://doi.org/10.1016/j.jcp.2015.12.037	journal	March 2016
Moving overlapping grid methodology of spectral accuracy for incompressible flow solutions around rigid bodies in motion Merrill, Brandon E.; Peet, Yulia T. Journal of Computational Physics, Vol. 390 https://doi.org/10.1016/j.jcp.2019.01.048	journal	August 2019
Spectral element applications in complex nuclear reactor geometries: Tet-to-hex meshing Yuan, Haomin; Yildiz, Mustafa A.; Merzari, Elia Nuclear Engineering and Design, Vol. 357 https://doi.org/10.1016/j.nucengdes.2019.110422	journal	February 2020
Feasibility of full-core pin resolved CFD simulations of small modular reactor with momentum sources Fang, Jun; Shaver, Dillon R.; Tomboulides, Ananias Nuclear Engineering and Design, Vol. 378 https://doi.org/10.1016/j.nucengdes.2021.111143	journal	July 2021
An improved pressure drop correlation for modeling localized effects in a pebble bed reactor Reger, David; Merzari, Elia; Balestra, Paolo Nuclear Engineering and Design, Vol. 403 https://doi.org/10.1016/j.nucengdes.2022.112123	journal	March 2023
Demonstration of RANS models with wall functions in the spectral element code Nek5000 Shaver, Dillon R.; Tomboulides, Ananias; Obabko, Aleksandr Nuclear Engineering and Design, Vol. 408 https://doi.org/10.1016/j.nucengdes.2023.112302	journal	July 2023
NekRS, a GPU-accelerated spectral element Navier–Stokes solver Fischer, Paul; Kerkemeier, Stefan; Min, Misun Parallel Computing, Vol. 114 https://doi.org/10.1016/j.parco.2022.102982	journal	December 2022
Transition to turbulence in plane Poiseuille and plane Couette flow Orszag, Steven A.; Kells, Lawrence C. Journal of Fluid Mechanics, Vol. 96, Issue 1 https://doi.org/10.1017/S0022112080002066	journal	January 1980
High-pass filtered eddy-viscosity models for large-eddy simulations of transitional and turbulent flow Stolz, S.; Schlatter, P.; Kleiser, L. Physics of Fluids, Vol. 17, Issue 6 https://doi.org/10.1063/1.1923048	journal	June 2005
Cardinal: A Lower-Length-Scale Multiphysics Simulator for Pebble-Bed Reactors Merzari, Elia; Yuan, Haomin; Min, Misun Nuclear Technology, Vol. 207, Issue 7 https://doi.org/10.1080/00295450.2020.1824471	journal	January 2021
Direct Numerical Simulation of the Flow Through a Randomly Packed Pebble Bed Yildiz, Mustafa Alper; Botha, Gerrit; Yuan, Haomin Journal of Fluids Engineering, Vol. 142, Issue 4 https://doi.org/10.1115/1.4045439	journal	January 2020
Finite-Element Preconditioning of G-NI Spectral Methods Canuto, Claudio; Gervasio, Paola; Quarteroni, Alfio SIAM Journal on Scientific Computing, Vol. 31, Issue 6 https://doi.org/10.1137/090746367	journal	January 2010
Tuning Spectral Element Preconditioners for Parallel Scalability on GPUs Phillips, Malachi; Kerkemeier, Stefan; Fischer, Paul Proceedings of the 2022 SIAM Conference on Parallel Processing for Scientific Computing (PP) https://doi.org/10.1137/1.9781611977141.4	book	January 2022
Stability Analysis of Interface Temporal Discretization in Grid Overlapping Methods Peet, Yulia T.; Fischer, Paul F. SIAM Journal on Numerical Analysis, Vol. 50, Issue 6 https://doi.org/10.1137/110831234	journal	January 2012
Scalable Low-Order Finite Element Preconditioners for High-Order Spectral Element Poisson Solvers Bello-Maldonado, Pedro D.; Fischer, Paul F. SIAM Journal on Scientific Computing, Vol. 41, Issue 5 https://doi.org/10.1137/18M1194997	journal	January 2019
Efficient Low-Order Refined Preconditioners for High-Order Matrix-Free Continuous and Discontinuous Galerkin Methods Pazner, Will SIAM Journal on Scientific Computing, Vol. 42, Issue 5 https://doi.org/10.1137/19M1282052	journal	January 2020
Exascale Multiphysics Nuclear Reactor Simulations for Advanced Designs Merzari, Elia; Hamilton, Steven; Evans, Thomas Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3581784.3627038	conference	November 2023
General Circulation Experiments with the Primitive Equations: i. the Basic Experiment* Smagorinsky, J. Monthly Weather Review, Vol. 91, Issue 3 https://doi.org/10.1175/1520-0493(1963)091<0099:GCEWTP>2.3.CO;2	journal	March 1963
Numerical Methods in Coupled Monte Carlo and Thermal-Hydraulic Calculations Gill, Daniel F.; Griesheimer, David P.; Aumiller, David L. Nuclear Science and Engineering, Vol. 185, Issue 1 https://doi.org/10.13182/NSE16-3	journal	January 2017
On the implementation of the κ-ε turbulence model in incompressible flow solvers based on a finite element discretisation Kuzmin, D.; Mierka, O.; Turek, S. International Journal of Computing Science and Mathematics, Vol. 1, Issue 2/3/4 https://doi.org/10.1504/IJCSM.2007.016531	journal	January 2007
A methodology for the design and testing of atmospheric boundary layer models for wind energy applications Sanz Rodrigo, Javier; Churchfield, Matthew; Kosovic, Branko Wind Energy Science, Vol. 2, Issue 1 https://doi.org/10.5194/wes-2-35-2017	journal	January 2017

Similar Records

NekRS, a GPU-accelerated spectral element Navier–Stokes solver

Journal Article · Mon Oct 17 20:00:00 EDT 2022 · Parallel Computing · OSTI ID:2429555

Towards exascale for wind energy simulations

Journal Article · Thu May 23 20:00:00 EDT 2024 · International Journal of High Performance Computing Applications · OSTI ID:2361165

Related Subjects

42 ENGINEERING

Energy Exascale Computational Fluid Dynamics Simulations With the Spectral Element Method

Citation Formats

References (40)

Similar Records

Related Subjects