PeleC: An adaptive mesh refinement solver for compressible reacting flows

Henry de Frahan, Marc T.; Rood, Jon S.; Day, Marc S.; Sitaraman, Hariswaran; Yellapantula, Shashank; Perry, Bruce A.; Grout, Ray W.; Almgren, Ann; Zhang, Weiqun; Bell, John B.; Chen, Jacqueline H.

doi:10.1177/10943420221121151

PeleC: An adaptive mesh refinement solver for compressible reacting flows

Journal Article · Tue Sep 06 04:00:00 EDT 2022 · International Journal of High Performance Computing Applications

DOI:https://doi.org/10.1177/10943420221121151· OSTI ID:1885604

^[1]; ^[1]; Day, Marc S. ^[1]; Sitaraman, Hariswaran ^[1]; Yellapantula, Shashank ^[1]; ^[1]; Grout, Ray W. ^[1]; ^[2]; ^[2]; ^[2]; Chen, Jacqueline H. ^[3]

Computational Science Center, National Renewable Energy Laboratory, Golden, CO, USA
Center for Computational Sciences and Engineering, Lawrence Berkeley National Laboratory, Berkeley, CA, USA
Combustion Research Facility, Sandia National Laboratories, Livermore, CA, USA

Reacting flow simulations for combustion applications require extensive computing capabilities. Leveraging the AMReX library, the Pele suite of combustion simulation tools targets the largest supercomputers available and future exascale machines. We introduce PeleC, the compressible solver in the Pele suite, and detail its capabilities, including complex geometry representation, chemistry integration, and discretization. We present a comparison of development efforts using both OpenACC and AMReX’s C++ performance portability framework for execution on multiple GPU architectures. We discuss relevant details that have allowed PeleC to achieve high performance and scalability. PeleC’s performance characteristics are measured through relevant simulations on multiple supercomputers. The success of PeleC’s design for exascale is exhibited through demonstration of a 160 billion cell simulation and weak scaling onto 100% of Summit, an NVIDIA-based GPU supercomputer at Oak Ridge National Laboratory. Our results provide confidence that PeleC will enable future combustion science simulations with unprecedented fidelity.

View Journal Article

Sponsoring Organization:: USDOE

OSTI ID:: 1885604

Alternate ID(s):: OSTI ID: 1889664
OSTI ID: 1902911

Journal Information:: International Journal of High Performance Computing Applications, Journal Name: International Journal of High Performance Computing Applications Journal Issue: 2 Vol. 37; ISSN 1094-3420

Publisher:: SAGE PublicationsCopyright Statement

Country of Publication:: United States

Language:: English

References (32)

Discrete filters for large eddy simulation Sagaut, P.; Grohens, R. International Journal for Numerical Methods in Fluids, Vol. 31, Issue 8 https://doi.org/10.1002/(SICI)1097-0363(19991230)31:8<1195::AID-FLD914>3.0.CO;2-H	journal	December 1999
A performance spectrum for parallel computational frameworks that solve PDEs: A performance spectrum for parallel computational frameworks that solve PDEs Chang, J.; Nakshatrala, K. B.; Knepley, M. G. Concurrency and Computation: Practice and Experience, Vol. 30, Issue 11 https://doi.org/10.1002/cpe.4401	journal	December 2017
An Adaptive Cartesian Grid Method for Unsteady Compressible Flow in Irregular Regions Pember, Richard B.; Bell, John B.; Colella, Phillip Journal of Computational Physics, Vol. 120, Issue 2 https://doi.org/10.1006/jcph.1995.1165	journal	September 1995
A Conservative Adaptive Projection Method for the Variable Density Incompressible Navier–Stokes Equations Almgren, Ann S.; Bell, John B.; Colella, Phillip Journal of Computational Physics, Vol. 142, Issue 1 https://doi.org/10.1006/jcph.1998.5890	journal	May 1998
Equilibrium constants from a modified Redlich-Kwong equation of state Soave, Giorgio Chemical Engineering Science, Vol. 27, Issue 6 https://doi.org/10.1016/0009-2509(72)80096-4	journal	June 1972
The Piecewise Parabolic Method (PPM) for gas-dynamical simulations Colella, Phillip; Woodward, Paul R. Journal of Computational Physics, Vol. 54, Issue 1 https://doi.org/10.1016/0021-9991(84)90143-8	journal	April 1984
Low-storage, explicit Runge–Kutta schemes for the compressible Navier–Stokes equations Kennedy, Christopher A.; Carpenter, Mark H.; Lewis, R. Michael Applied Numerical Mathematics, Vol. 35, Issue 3 https://doi.org/10.1016/S0168-9274(99)00141-5	journal	November 2000
Adaptive mesh based combustion simulations of direct fuel injection effects in a supersonic cavity flame-holder Sitaraman, Hariswaran; Yellapantula, Shashank; Henry de Frahan, Marc T. Combustion and Flame, Vol. 232 https://doi.org/10.1016/j.combustflame.2021.111531	journal	October 2021
A state redistribution algorithm for finite volume schemes on cut cell meshes Berger, Marsha; Giuliani, Andrew Journal of Computational Physics, Vol. 428 https://doi.org/10.1016/j.jcp.2020.109820	journal	March 2021
A weighted state redistribution algorithm for embedded boundary grids Giuliani, A.; Almgren, A. S.; Bell, J. B. Journal of Computational Physics, Vol. 464 https://doi.org/10.1016/j.jcp.2022.111305	journal	September 2022
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.07.003	journal	December 2014
Numerical simulation of a laboratory-scale turbulent slot flame Bell, John B.; Day, Marcus S.; Grcar, Joseph F. Proceedings of the Combustion Institute, Vol. 31, Issue 1 https://doi.org/10.1016/j.proci.2006.07.186	journal	January 2007
A deferred correction coupling strategy for low Mach number flow with complex chemistry Nonaka, A.; Bell, J. B.; Day, M. S. Combustion Theory and Modelling, Vol. 16, Issue 6 https://doi.org/10.1080/13647830.2012.701019	journal	December 2012
High-order algorithms for compressible reacting flow with complex chemistry Emmett, Matthew; Zhang, Weiqun; Bell, John B. Combustion Theory and Modelling, Vol. 18, Issue 3 https://doi.org/10.1080/13647830.2014.919410	journal	May 2014
A conservative, thermodynamically consistent numerical approach for low Mach number combustion. Part I: Single-level integration Nonaka, Andrew; Day, Marcus S.; Bell, John B. Combustion Theory and Modelling, Vol. 22, Issue 1 https://doi.org/10.1080/13647830.2017.1390610	journal	August 2017
FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes Fryxell, B.; Olson, K.; Ricker, P. The Astrophysical Journal Supplement Series, Vol. 131, Issue 1 https://doi.org/10.1086/317361	journal	November 2000
Castro: a new Compressible Astrophysical Solver. i. Hydrodynamics and Self-Gravity Almgren, A. S.; Beckner, V. E.; Bell, J. B. The Astrophysical Journal, Vol. 715, Issue 2 https://doi.org/10.1088/0004-637X/715/2/1221	journal	May 2010
Numerical simulation of laminar reacting flows with complex chemistry Day, M. S.; Bell, J. B. Combustion Theory and Modelling, Vol. 4, Issue 4 https://doi.org/10.1088/1364-7830/4/4/309	journal	December 2000
Visualizations of direct fuel injection effects in a supersonic cavity flameholder Sitaraman, Hariswaran; Brunhart-Lupo, Nicholas; de Frahan, Marc Henry Physical Review Fluids, Vol. 6, Issue 11 https://doi.org/10.1103/PhysRevFluids.6.110504	journal	November 2021
RAJA: Portable Performance for Large-Scale Scientific Applications Beckingsale, David A.; Scogland, Thomas RW; Burmark, Jason 2019 IEEE/ACM International Workshop on Performance, Portability and Productivity in HPC (P3HPC) https://doi.org/10.1109/P3HPC49587.2019.00012	conference	November 2019
Preparing Nuclear Astrophysics for Exascale Katz, Max P.; Almgren, Ann; Sazo, Maria Barrios SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00095	conference	November 2020
Code Verification by the Method of Manufactured Solutions Roache, Patrick J. Journal of Fluids Engineering, Vol. 124, Issue 1 https://doi.org/10.1115/1.1436090	journal	November 2001
BoxLib with Tiling: An Adaptive Mesh Refinement Software Framework Zhang, Weiqun; Almgren, Ann; Day, Marcus SIAM Journal on Scientific Computing, Vol. 38, Issue 5 https://doi.org/10.1137/15M102616X	journal	January 2016
Comparative Study of Finite Element Methods Using the Time-Accuracy-Size(TAS) Spectrum Analysis Chang, Justin; Fabien, Maurice S.; Knepley, Matthew G. SIAM Journal on Scientific Computing, Vol. 40, Issue 6 https://doi.org/10.1137/18M1172260	journal	January 2018
SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers Hindmarsh, Alan C.; Brown, Peter N.; Grant, Keith E. ACM Transactions on Mathematical Software, Vol. 31, Issue 3 https://doi.org/10.1145/1089014.1089020	journal	September 2005
Large scale parallel structured AMR calculations using the SAMRAI framework Wissink, Andrew M.; Hornung, Richard D.; Kohn, Scott R. Proceedings of the 2001 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '01 https://doi.org/10.1145/582034.582040	conference	January 2001
AMReX: Block-structured adaptive mesh refinement for multiphysics applications Zhang, Weiqun; Myers, Andrew; Gott, Kevin The International Journal of High Performance Computing Applications https://doi.org/10.1177/10943420211022811	journal	June 2021
AMReX: a framework for block-structured adaptive mesh refinement Zhang, Weiqun; Almgren, Ann; Beckner, Vince Journal of Open Source Software, Vol. 4, Issue 37 https://doi.org/10.21105/joss.01370	journal	May 2019
Investigation of finite-volume methods to capture shocks and turbulence spectra in compressible flows Motheau, Emmanuel; Wakefield, John Communications in Applied Mathematics and Computational Science, Vol. 15, Issue 1 https://doi.org/10.2140/camcos.2020.15.1_	journal	June 2020
Chemkin-II: A Fortran chemical kinetics package for the analysis of gas-phase chemical kinetics Kee, R.; Rupley, F.; Miller, J. https://doi.org/10.2172/5681118	report	September 1989
Optimizing performance of combustion chemistry solvers on Intel’s Many Integrated Core (MIC) architectures Sitaraman, Hariswaran; Grout, Ray 23rd AIAA Computational Fluid Dynamics Conference https://doi.org/10.2514/6.2017-4410	conference	June 2017
Comparative Performance Evaluation of Modern Heterogeneous High-Performance Computing Systems CPUs Sorokin, Aleksei; Malkovsky, Sergey; Tsoy, Georgiy Electronics, Vol. 9, Issue 6 https://doi.org/10.3390/electronics9061035	journal	June 2020

Similar Records

Enabling Combustion Science Simulations for Future Exascale Machines

Conference · Tue Nov 23 23:00:00 EST 2021 · OSTI ID:1833338

PeleC: An adaptive mesh refinement solver for compressible reacting flows

Citation Formats

References (32)

Similar Records

Related Subjects