OpenACC acceleration of an unstructured CFD solver based on a reconstructed discontinuous Galerkin method for compressible flows

Xia, Yidong; Lou, Jialin; Luo, Hong; Edwards, Jack; Mueller, Frank

doi:10.1002/fld.4009

OpenACC acceleration of an unstructured CFD solver based on a reconstructed discontinuous Galerkin method for compressible flows

Journal Article · Mon Feb 09 00:00:00 EST 2015 · International Journal for Numerical Methods in Fluids

DOI:https://doi.org/10.1002/fld.4009· OSTI ID:1249553

Xia, Yidong ^[1]; Lou, Jialin ^[2]; Luo, Hong ^[2]; Edwards, Jack ^[2]; Mueller, Frank ^[2]

Idaho National Lab. (INL), Idaho Falls, ID (United States)
North Carolina State Univ., Raleigh, NC (United States)

Here, an OpenACC directive-based graphics processing unit (GPU) parallel scheme is presented for solving the compressible Navier–Stokes equations on 3D hybrid unstructured grids with a third-order reconstructed discontinuous Galerkin method. The developed scheme requires the minimum code intrusion and algorithm alteration for upgrading a legacy solver with the GPU computing capability at very little extra effort in programming, which leads to a unified and portable code development strategy. A face coloring algorithm is adopted to eliminate the memory contention because of the threading of internal and boundary face integrals. A number of flow problems are presented to verify the implementation of the developed scheme. Timing measurements were obtained by running the resulting GPU code on one Nvidia Tesla K20c GPU card (Nvidia Corporation, Santa Clara, CA, USA) and compared with those obtained by running the equivalent Message Passing Interface (MPI) parallel CPU code on a compute node (consisting of two AMD Opteron 6128 eight-core CPUs (Advanced Micro Devices, Inc., Sunnyvale, CA, USA)). Speedup factors of up to 24× and 1.6× for the GPU code were achieved with respect to one and 16 CPU cores, respectively. The numerical results indicate that this OpenACC-based parallel scheme is an effective and extensible approach to port unstructured high-order CFD solvers to GPU computing.

View Accepted Manuscript (DOE)

Research Organization:: Idaho National Laboratory, Idaho Falls, ID (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: AC07-05ID14517

OSTI ID:: 1249553

Report Number(s):: INL/JOU--15-34041

Journal Information:: International Journal for Numerical Methods in Fluids, Journal Name: International Journal for Numerical Methods in Fluids Journal Issue: 3 Vol. 78; ISSN 0271-2091

Publisher:: WileyCopyright Statement

Country of Publication:: United States

Language:: English

References (34)

Applied Computational Fluid Dynamics Techniques Löhner, Rainald https://doi.org/10.1002/9780470989746	journal	March 2008
Running unstructured grid-based CFD solvers on modern graphics hardware Corrigan, Andrew; Camelli, Fernando F.; Löhner, Rainald International Journal for Numerical Methods in Fluids, Vol. 66, Issue 2 https://doi.org/10.1002/fld.2254	journal	April 2011
Unsteady CFD computations using vertex-centered finite volumes for unstructured grids on Graphics Processing Units Asouti, V. G.; Trompoukis, X. S.; Kampolis, I. C. International Journal for Numerical Methods in Fluids, Vol. 67, Issue 2 https://doi.org/10.1002/fld.2352	journal	May 2010
Semi-automatic porting of a large-scale Fortran CFD code to GPUs Corrigan, Andrew; Camelli, Fernando; Löhner, Rainald International Journal for Numerical Methods in Fluids, Vol. 69, Issue 2 https://doi.org/10.1002/fld.2560	journal	May 2011
High-order CFD methods: current status and perspective: HIGH-ORDER CFD METHODS Wang, Z. J.; Fidkowski, Krzysztof; Abgrall, Rémi International Journal for Numerical Methods in Fluids, Vol. 72, Issue 8 https://doi.org/10.1002/fld.3767	journal	January 2013
Average-State Jacobians and Implicit Methods for Compressible Viscous and Turbulent Flows Batten, P.; Leschziner, M. A.; Goldberg, U. C. Journal of Computational Physics, Vol. 137, Issue 1 https://doi.org/10.1006/jcph.1997.5793	journal	October 1997
The Runge–Kutta Discontinuous Galerkin Method for Conservation Laws V Cockburn, Bernardo; Shu, Chi-Wang Journal of Computational Physics, Vol. 141, Issue 2 https://doi.org/10.1006/jcph.1998.5892	journal	April 1998
Using Compiler Directives for Accelerating CFD Applications on GPUs Jin, Haoqiang; Kellogg, Mark; Mehrotra, Piyush OpenMP in a Heterogeneous World https://doi.org/10.1007/978-3-642-30961-8_12	book	January 2012
OpenACC — First Experiences with Real-World Applications Wienke, Sandra; Springer, Paul; Terboven, Christian Euro-Par 2012 Parallel Processing https://doi.org/10.1007/978-3-642-32820-6_85	book	January 2012
High-Re solutions for incompressible flow using the Navier-Stokes equations and a multigrid method Ghia, U.; Ghia, K. N.; Shin, C. T. Journal of Computational Physics, Vol. 48, Issue 3 https://doi.org/10.1016/0021-9991(82)90058-4	journal	December 1982
Discontinuous Galerkin solution of the Reynolds-averaged Navier–Stokes and k–ω turbulence model equations Bassi, Francesco; Crivellini, Andrea; Rebay, Stefano Computers & Fluids, Vol. 34, Issue 4-5 https://doi.org/10.1016/j.compfluid.2003.08.004	journal	May 2005
A set of parallel, implicit methods for a reconstructed discontinuous Galerkin method for compressible flows on 3D hybrid grids Xia, Yidong; Luo, Hong; Frisbey, Megan Computers & Fluids, Vol. 98 https://doi.org/10.1016/j.compfluid.2014.01.023	journal	July 2014
An implicit Hermite WENO reconstruction-based discontinuous Galerkin method on tetrahedral grids Xia, Yidong; Luo, Hong; Nourgaliev, Robert Computers & Fluids, Vol. 96 https://doi.org/10.1016/j.compfluid.2014.02.027	journal	June 2014
A p-multigrid discontinuous Galerkin method for the Euler equations on unstructured grids Luo, Hong; Baum, Joseph D.; Löhner, Rainald Journal of Computational Physics, Vol. 211, Issue 2 https://doi.org/10.1016/j.jcp.2005.06.019	journal	January 2006
A discontinuous Galerkin method based on a Taylor basis for the compressible flows on arbitrary grids Luo, Hong; Baum, Joseph D.; Löhner, Rainald Journal of Computational Physics, Vol. 227, Issue 20 https://doi.org/10.1016/j.jcp.2008.06.035	journal	October 2008
Large calculation of the flow over a hypersonic vehicle using a GPU Elsen, Erich; LeGresley, Patrick; Darve, Eric Journal of Computational Physics, Vol. 227, Issue 24 https://doi.org/10.1016/j.jcp.2008.08.023	journal	December 2008
Nodal discontinuous Galerkin methods on graphics processors Klöckner, A.; Warburton, T.; Bridge, J. Journal of Computational Physics, Vol. 228, Issue 21 https://doi.org/10.1016/j.jcp.2009.06.041	journal	November 2009
A reconstructed discontinuous Galerkin method for the compressible Navier–Stokes equations on arbitrary grids Luo, Hong; Luo, Luqing; Nourgaliev, Robert Journal of Computational Physics, Vol. 229, Issue 19 https://doi.org/10.1016/j.jcp.2010.05.033	journal	September 2010
A class of hybrid DG/FV methods for conservation laws II: Two-dimensional cases Zhang, Laiping; Wei, Liu; Lixin, He Journal of Computational Physics, Vol. 231, Issue 4 https://doi.org/10.1016/j.jcp.2011.03.032	journal	February 2012
A Hermite WENO reconstruction-based discontinuous Galerkin method for the Euler equations on tetrahedral grids Luo, Hong; Xia, Yidong; Li, Shujie Journal of Computational Physics, Vol. 231, Issue 16 https://doi.org/10.1016/j.jcp.2012.05.011	journal	June 2012
A reconstructed discontinuous Galerkin method based on a Hierarchical WENO reconstruction for compressible flows on tetrahedral grids Luo, Hong; Xia, Yidong; Spiegel, Seth Journal of Computational Physics, Vol. 236 https://doi.org/10.1016/j.jcp.2012.11.026	journal	March 2013
GPU acceleration of an unmodified parallel finite element Navier-Stokes solver Goddeke, Dominik; Buijssen, Sven H. M.; Wobker, Hilmar Simulation (HPCS), 2009 International Conference on High Performance Computing & Simulation https://doi.org/10.1109/HPCSIM.2009.5191718	conference	June 2009
OpenCL: A Parallel Programming Standard for Heterogeneous Computing Systems Stone, John E.; Gohara, David; Shi, Guochun Computing in Science & Engineering, Vol. 12, Issue 3, p. 66-73 https://doi.org/10.1109/MCSE.2010.69	journal	May 2010
A Survey of General-Purpose Computation on Graphics Hardware Owens, John D.; Luebke, David; Govindaraju, Naga Computer Graphics Forum, Vol. 26, Issue 1 https://doi.org/10.1111/j.1467-8659.2007.01012.x	journal	March 2007
Acceleration of a CFD Code with a GPU Jespersen, Dennis C. Scientific Programming, Vol. 18, Issue 3-4 https://doi.org/10.1155/2010/564806	journal	January 2010
Acceleration of a 3D Euler Solver Using Commodity Graphics Hardware Brandvik, Tobias; Pullan, Graham 46th AIAA Aerospace Sciences Meeting and Exhibit https://doi.org/10.2514/6.2008-607	conference	June 2012
Rapid Aerodynamic Performance Prediction on a Cluster of Graphics Processing Units Phillips, Everett; Zhang, Yao; Davis, Roger 47th AIAA Aerospace Sciences Meeting including The New Horizons Forum and Aerospace Exposition https://doi.org/10.2514/6.2009-565	conference	June 2012
CUDA Implementation of a Navier-Stokes Solver on Multi-GPU Desktop Platforms for Incompressible Flows Thibault, Julien; Senocak, Inanc 47th AIAA Aerospace Sciences Meeting including The New Horizons Forum and Aerospace Exposition https://doi.org/10.2514/6.2009-758	conference	June 2012
An MPI-CUDA Implementation for Massively Parallel Incompressible Flow Computations on Multi-GPU Clusters Jacobsen, Dana; Thibault, Julien; Senocak, Inanc 48th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition https://doi.org/10.2514/6.2010-522	conference	June 2012
Porting of an Edge-Based CFD Solver to GPUs Corrigan, Andrew; Camelli, Fernando; Löhner, Rainald 48th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition https://doi.org/10.2514/6.2010-523	conference	June 2012
High-Order Spectral Difference: Verification and Acceleration using GPU Computing Zimmerman, Ben J.; Wang, Zhi J.; Visbal, Miguel R. 21st AIAA Computational Fluid Dynamics Conference https://doi.org/10.2514/6.2013-2941	conference	June 2013
Directive-Based GPU Programming for Computational Fluid Dynamics Pickering, Brent P.; Roy, Christopher J. 52nd Aerospace Sciences Meeting https://doi.org/10.2514/6.2014-1131	conference	January 2014
A Parallel, Reconstructed Discontinuous Galerkin Method for the Compressible Flows on Arbitrary Grids Luo, Hong; Luo, Luqing; Ali, Amjad Communications in Computational Physics, Vol. 9, Issue 2 https://doi.org/10.4208/cicp.070210.020610a	journal	February 2011
A Reconstructed Discontinuous Galerkin Method for the Euler Equations on Arbitrary Grids Luo, Hong; Luo, Luqing; Nourgaliev, Robert Communications in Computational Physics, Vol. 12, Issue 5 https://doi.org/10.4208/cicp.250911.030212a	journal	November 2012

Cited By (3)

Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations Gong, Jing; Markidis, Stefano; Laure, Erwin The Journal of Supercomputing, Vol. 72, Issue 11 https://doi.org/10.1007/s11227-016-1744-5	journal	July 2016
Optimization and acceleration of flow simulations for CFD on CPU/GPU architecture Lei, Jiang; Li, Da-li; Zhou, Yun-long Journal of the Brazilian Society of Mechanical Sciences and Engineering, Vol. 41, Issue 7 https://doi.org/10.1007/s40430-019-1793-9	journal	June 2019
High-Order Hyperbolic Navier-Stokes Reconstructed Discontinuous Galerkin Method for Unsteady Flows Li, Lingquan; Lou, Jialin; Luo, Hong AIAA Aviation 2019 Forum https://doi.org/10.2514/6.2019-3060	conference	June 2019

Similar Records

Developing an ELM Ecosystem Dynamics Model on GPU with OpenACC

Conference · Wed Jun 01 00:00:00 EDT 2022 · OSTI ID:1879940

Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations

Journal Article · Mon Jul 18 00:00:00 EDT 2016 · Journal of Supercomputing · OSTI ID:1565549

Nek5000 with OpenACC

Journal Article · · Lecture Notes in Computer Science · OSTI ID:1567377

Related Subjects

97 MATHEMATICS AND COMPUTING
CUDA
GPU computing
Navier-Stokes equations
OpenACC
WENO
compressible flow
discontinuous Galerkin

OpenACC acceleration of an unstructured CFD solver based on a reconstructed discontinuous Galerkin method for compressible flows

Citation Formats

References (34)

Cited By (3)

Similar Records

Related Subjects