Reverse-Mode Algorithmic Differentiation of an OpenMP-Parallel Compressible Flow Solver

Huckelheim, Jan; Hovland, Paul; Strout, Michelle Mills; Muller, Jens-Dominik

doi:10.1177/1094342017712060

Reverse-Mode Algorithmic Differentiation of an OpenMP-Parallel Compressible Flow Solver

Journal Article · 01 January 2019 · International Journal of High Performance Computing Applications

DOI:https://doi.org/10.1177/1094342017712060· OSTI ID:1510005

Huckelheim, Jan; Hovland, Paul; Strout, Michelle Mills; Muller, Jens-Dominik

y Reverse-mode algorithmic differentiation (AD) is an established method for obtaining adjoint derivatives of computer simulation applications. In computational fluid dynamics (CFD), adjoint derivatives of a cost function output such as drag or lift with respect to design parameters such as surface coordinates or geometry control points are a key ingredient for shape optimization, uncertainty quantification and flow control. The computational cost of CFD applications and their derivatives makes it essential to use high-performance computing hardware efficiently, including multi- and many-core architectures. Nevertheless, OpenMP is not supported in most AD tools, and previously shown methods achieve poor scalability of the derivative code. We present the AD of an OpenMP-parallelized finite volume compressible flow solver for unstructured meshes. Our approach enables us to reuse the parallelization of the original code in the computation of adjoint derivatives. The method works by identifying code segments that can be differentiated in reverse-mode without changing their memory access pattern. The OpenMP parallelization is integrated into the derivative code during the build process in a way that is robust to modifications of the original code and independent of the OpenMP support of the differentiation tool. We show the scalability of our adjoint CFD solver on test cases ranging from thousands to millions of finite volume mesh cells on CPUs with up to 16 threads as well as on an Intel XeonPhi card with 236 threads. We demonstrate that our approach is more practical to implement for production-sized CFD codes and produces more efficient adjoint derivative code than previously shown AD methods.

Research Organization:: Argonne National Laboratory (ANL), Argonne, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC); USDOE Office of Science - Office of Advanced Scientific Computing Research; European Commission - Community Research and Development Information Service (CORDIS) - Seventh Framework Programme (FP7)

DOE Contract Number:: AC02-06CH11357

OSTI ID:: 1510005

Journal Information:: International Journal of High Performance Computing Applications, Vol. 33, Issue 1

Country of Publication:: United States

Language:: English

References (34)

On the performance of discrete adjoint CFD codes using automatic differentiation No authors listed International Journal for Numerical Methods in Fluids https://doi.org/10.1002/fld.885		January 2005
Comparison of Node-Centered and Cell-Centered Unstructured Finite-Volume Discretizations: Viscous Fluxes No authors listed AIAA Journal https://doi.org/10.2514/1.44940		July 2010
MPI-Parallel Discrete Adjoint OpenFOAM No authors listed Procedia Computer Science https://doi.org/10.1016/j.procs.2015.05.181		January 2015
Sparse Matrices in MATLAB: Design and Implementation No authors listed SIAM Journal on Matrix Analysis and Applications https://doi.org/10.1137/0613024		January 1992
A continuous adjoint formulation for the computation of topological and surface sensitivities of ducted flows No authors listed International Journal for Numerical Methods in Fluids https://doi.org/10.1002/fld.1770		November 2008
Approximate Riemann solvers, parameter vectors, and difference schemes No authors listed Journal of Computational Physics https://doi.org/10.1016/0021-9991(81)90128-5		October 1981
Toward adjoinable MPI No authors listed Distributed Processing (IPDPS), 2009 IEEE International Symposium on Parallel & Distributed Processing https://doi.org/10.1109/IPDPS.2009.5161165		May 2009
A class of OpenMP applications involving nested parallelism No authors listed Proceedings of the 2004 ACM symposium on Applied computing - SAC '04 https://doi.org/10.1145/967900.967948		January 2004
Explicit loop scheduling in OpenMP for parallel automatic differentiation No authors listed Proceedings 16th Annual International Symposium on High Performance Computing Systems and Applications https://doi.org/10.1109/HPCSA.2002.1019144		January 2002
Towards High-Performance Optimizations of the Unstructured Open-Source SU2 Suite No authors listed AIAA Infotech @ Aerospace https://doi.org/10.2514/6.2015-1949		January 2015
OpenMP: an industry standard API for shared-memory programming No authors listed IEEE Computational Science and Engineering https://doi.org/10.1109/99.660313		January 1998
Developing a scalable hybrid MPI/OpenMP unstructured finite element model No authors listed Computers & Fluids https://doi.org/10.1016/j.compfluid.2014.09.007		March 2015
On the chromatic index of almost all graphs No authors listed Journal of Combinatorial Theory, Series B https://doi.org/10.1016/0095-8956(77)90039-9		October 1977
Tangent Linear and Adjoint Versions of NASA/GMAO’s Fortran 90 Global Weather Forecast Model No authors listed Automatic Differentiation: Applications, Theory, and Implementations https://doi.org/10.1007/3-540-28438-9_24		January 2006
Continuous Adjoint Method for Unstructured Grids No authors listed AIAA Journal https://doi.org/10.2514/1.25362		May 2008
Code generation for parallel execution of a class of irregular loops on distributed memory systems No authors listed 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.30		November 2012
A graph coloring algorithm for large scale scheduling problems No authors listed Computers & Operations Research https://doi.org/10.1016/0305-0548(86)90061-4		January 1986
Performance analysis of the OP2 framework on many-core architectures No authors listed ACM SIGMETRICS Performance Evaluation Review https://doi.org/10.1145/1964218.1964221		March 2011
Adjoining Hybrid Parallel Code No authors listed The Eighth International Conference on Engineering Computational Technology, Civil-Comp Proceedings https://doi.org/10.4203/ccp.100.7		September 2012
Hybrid OpenMP/MPI Anisotropic Mesh Smoothing No authors listed Procedia Computer Science https://doi.org/10.1016/j.procs.2012.04.166		January 2012
Run-time parallelization and scheduling of loops No authors listed IEEE Transactions on Computers https://doi.org/10.1109/12.88484		May 1991
Pseudo-timestepping and verification for automatic differentiation derived CFD codes No authors listed Computers & Fluids https://doi.org/10.1016/j.compfluid.2011.01.039		July 2011
Algorithm 755; ADOL-C: a package for the automatic differentiation of algorithms written in C/C++ No authors listed ACM Transactions on Mathematical Software https://doi.org/10.1145/229473.229474		June 1996
OpenAD/F: A Modular Open-Source Tool for Automatic Differentiation of Fortran Codes No authors listed ACM Transactions on Mathematical Software https://doi.org/10.1145/1377596.1377598		July 2008
Automatic parallelization of irregular applications No authors listed Parallel Computing https://doi.org/10.1016/S0167-8191(00)00052-1		December 2000
Handling tens of thousands of cores with industrial/legacy codes: Approaches, implementation and timings No authors listed Computers & Fluids https://doi.org/10.1016/j.compfluid.2012.09.030		October 2013
Interpretative adjoints for numerical simulation codes using MPI No authors listed Procedia Computer Science https://doi.org/10.1016/j.procs.2010.04.204		May 2010
An Introduction to the Adjoint Approach to Design No authors listed Flow, Turbulence and Combustion https://doi.org/10.1023/A:1011430410075		December 2000
Bringing together automatic differentiation and OpenMP No authors listed Proceedings of the 15th international conference on Supercomputing - ICS '01 https://doi.org/10.1145/377792.377842		January 2001
Automated Derivation of the Adjoint of High-Level Transient Finite Element Programs No authors listed SIAM Journal on Scientific Computing https://doi.org/10.1137/120873558		January 2013
The Tapenade automatic differentiation tool: Principles, model, and specification No authors listed ACM Transactions on Mathematical Software https://doi.org/10.1145/2450153.2450158		April 2013
Detached-Eddy Simulation No authors listed Annual Review of Fluid Mechanics https://doi.org/10.1146/annurev.fluid.010908.165130		January 2009
OpenUH: an optimizing, portable OpenMP compiler No authors listed Concurrency and Computation: Practice and Experience https://doi.org/10.1002/cpe.1174		January 2007
Realistic Performance Characterization of CFD Applications on Intel Many Integrated Core Architecture No authors listed The Computer Journal https://doi.org/10.1093/comjnl/bxv022		April 2015

Similar Records

Parallelizable adjoint stencil computations using transposed forward-mode algorithmic differentiation

Journal Article · 2018 · Optimization Methods and Software · OSTI ID:1477730

Hückelheim, J. C.; Hovland, P. D.; Strout, M. M.; +1 more

Deploy threading in Nalu solver stack

Technical Report · 2018 · OSTI ID:1481562

Prokopenko, Andrey; Thomas, Stephen; Swirydowicz, Kasia; +4 more

Symmetric Indefinite Linear Solver Using OpenMP Task on Multicore Architectures

Journal Article · 2018 · IEEE Transactions on Parallel and Distributed Systems · OSTI ID:1565696

Yamazaki, Ichitaro; Kurzak, Jakub; Wu, Panruo; +2 more

Reverse-Mode Algorithmic Differentiation of an OpenMP-Parallel Compressible Flow Solver

Citation Formats

References (34)

Similar Records

Related Subjects