Title: Source-to-Source Automatic Differentiation of OpenMP Parallel Loops

Journal Article · · ACM Transactions on Mathematical Software
DOI: https://doi.org/10.1145/3472796 · OSTI ID:1883228
 [1];  [2]
  1. Argonne National Lab. (ANL), Argonne, IL (United States)
  2. Inria Sophia-Antipolis, Biot (France)
+ Show Author Affiliations

This article presents our work toward correct and efficient automatic differentiation of OpenMP parallel worksharing loops in forward and reverse mode. Automatic differentiation is a method to obtain gradients of numerical programs, which are crucial in optimization, uncertainty quantification, and machine learning. The computational cost to compute gradients is a common bottleneck in practice. For applications that are parallelized for multicore CPUs or GPUs using OpenMP, one also wishes to compute the gradients in parallel. Here, we propose a framework to reason about the correctness of the generated derivative code, from which we justify our OpenMP extension to the differentiation model. We implement this model in the automatic differentiation tool Tapenade and present test cases that are differentiated following our extended differentiation procedure. Performance of the generated derivative programs in forward and reverse mode is better than sequential, although our reverse mode often scales worse than the input programs.

Research Organization:
Argonne National Laboratory (ANL), Argonne, IL (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Basic Energy Sciences (BES)
Grant/Contract Number:
AC02-06CH11357
OSTI ID:
1883228
Journal Information:
ACM Transactions on Mathematical Software, Journal Name: ACM Transactions on Mathematical Software Journal Issue: 1 Vol. 48; ISSN 0098-3500
Publisher:
Association for Computing MachineryCopyright Statement
Country of Publication:
United States
Language:
English

References (17)

Applying TAF to generate efficient derivative code of Fortran 77-95 programs journal March 2003
Parallel Reverse Mode Automatic Differentiation for OpenMP Programs with ADOL-C book January 2008
Exploiting Sparsity in Automatic Differentiation on Multicore Architectures book January 2012
Adjoint computations by algorithmic differentiation of a parallel solver for time-dependent PDEs journal September 2020
A usability case study of algorithmic differentiation tools on the ISSM ice sheet model journal November 2017
SIMPLE adjoint message passing journal March 2018
Parallelizable adjoint stencil computations using transposed forward-mode algorithmic differentiation journal May 2018
OpenMP: an industry standard API for shared-memory programming journal January 1998
Explicit loop scheduling in OpenMP for parallel automatic differentiation conference January 2002
The Design, Deployment, and Evaluation of the CORAL Pre-Exascale Systems
  • Vazhkudai, Sudharshan S.; de Supinski, Bronis R.; Bland, Arthur S.
  • SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00055
conference November 2018
Evaluating Derivatives book January 2008
A framework for enhancing data reuse via associative reordering journal June 2014
Automatic Differentiation for Adjoint Stencil Loops conference August 2019
Bringing together automatic differentiation and OpenMP conference January 2001
A class of OpenMP applications involving nested parallelism conference January 2004
Lattice Boltzmann Method for Fluid Flows journal January 1998
Reverse-mode algorithmic differentiation of an OpenMP-parallel compressible flow solver journal February 2017

Similar Records

Related Subjects

97 MATHEMATICS AND COMPUTING
Automatic Differentiation
Multicore
OpenMP
Shared-Memory Parallel