Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation

Conference ·
 [1];  [1];  [1];  [1];  [1]
  1. ORNL
+ Show Author Affiliations

We evaluate AI-assisted generative capabilities on fundamental numerical kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV, Jacobi Stencil, and CG. We test the generated kernel codes for a variety of language-supported programming models, including (1) C++ (e.g., OpenMP [including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g., OpenMP [including offload] and OpenACC), (3) Python (e.g., numpy, Numba, cuPy, and pyCUDA), and (4) Julia (e.g., Threads, CUDA.jl, AMDGPU.jl, and KernelAbstractions.jl). We use the GitHub Copilot capabilities powered by the GPT-based OpenAI Codex available in Visual Studio Code as of April 2023 to generate a vast amount of implementations given simple + + prompt variants. To quantify and compare the results, we propose a proficiency metric around the initial 10 suggestions given for each prompt. Results suggest that the OpenAI Codex outputs for C++ correlate with the adoption and maturity of programming models. For example, OpenMP and CUDA score really high, whereas HIP is still lacking. We found that prompts from either a targeted language such as Fortran or the more general purpose Python can benefit from adding code keywords, while Julia prompts perform acceptably well for its mature programming models (e.g., Threads and CUDA.jl). We expect for these benchmarks to provide a point of reference for each programming model's community. Overall, understanding the convergence of large language models, AI, and HPC is crucial due to its rapidly evolving nature and how it is redefining human-computer interactions.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Organization:
USDOE
DOE Contract Number:
AC05-00OR22725
OSTI ID:
2000371
Resource Relation:
Conference: The 52nd International Conference on Parallel Processing (ICPP) - Salt Lake City, Utah, United States of America - 8/7/2023 4:00:00 AM-8/10/2023 4:00:00 AM
Country of Publication:
United States
Language:
English

References (22)

Fortran journal August 1964
Julia: A Fresh Approach to Numerical Computing journal January 2017
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns journal December 2014
Code Generation Using Machine Learning: A Systematic Review journal January 2022
Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language conference March 2023
The International Exascale Software Project roadmap journal January 2011
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming conference February 2022
GPT-3: Its Nature, Scope, Limits, and Consequences journal November 2020
Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes conference May 2023
Psb2 conference June 2021
Is GitHub copilot a substitute for human pair-programming? conference May 2022
PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation journal March 2012
Experimental Multi-threading Support for the Julia Programming Language conference November 2014
Exascale Computing in the United States journal January 2019
Numba: a LLVM-based Python JIT compiler conference January 2015
Analysis of the popularity of programming languages in open source software communities conference August 2020
An empirical evaluation of GitHub copilot's code suggestions conference May 2022
Choose your programming copilot conference July 2022
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models conference April 2022
Extreme Heterogeneity 2018 - Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity report December 2018
Using GitHub Copilot to Solve Simple Programming Problems conference March 2023
Assessing the quality of GitHub copilot’s code generation
  • Yetistiren, Burak; Ozsoy, Isik; Tuzun, Eray
  • Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering https://doi.org/10.1145/3558489.3559072
conference November 2022

Similar Records

Related Subjects