Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation

Godoy, William; Valero Lara, Pedro; Teranishi, Keita; Balaprakash, Prasanna; Vetter, Jeffrey

doi:10.1145/3605731.3605886

Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation

Conference · Tue Aug 01 00:00:00 EDT 2023

DOI:https://doi.org/10.1145/3605731.3605886· OSTI ID:2000371

^[1]; ^[1]; Teranishi, Keita ^[1]; ^[1]; ^[1]

ORNL

We evaluate AI-assisted generative capabilities on fundamental numerical kernels in high-performance computing (HPC), including AXPY, GEMV, GEMM, SpMV, Jacobi Stencil, and CG. We test the generated kernel codes for a variety of language-supported programming models, including (1) C++ (e.g., OpenMP [including offload], OpenACC, Kokkos, SyCL, CUDA, and HIP), (2) Fortran (e.g., OpenMP [including offload] and OpenACC), (3) Python (e.g., numpy, Numba, cuPy, and pyCUDA), and (4) Julia (e.g., Threads, CUDA.jl, AMDGPU.jl, and KernelAbstractions.jl). We use the GitHub Copilot capabilities powered by the GPT-based OpenAI Codex available in Visual Studio Code as of April 2023 to generate a vast amount of implementations given simple + + prompt variants. To quantify and compare the results, we propose a proficiency metric around the initial 10 suggestions given for each prompt. Results suggest that the OpenAI Codex outputs for C++ correlate with the adoption and maturity of programming models. For example, OpenMP and CUDA score really high, whereas HIP is still lacking. We found that prompts from either a targeted language such as Fortran or the more general purpose Python can benefit from adding code keywords, while Julia prompts perform acceptably well for its mature programming models (e.g., Threads and CUDA.jl). We expect for these benchmarks to provide a point of reference for each programming model's community. Overall, understanding the convergence of large language models, AI, and HPC is crucial due to its rapidly evolving nature and how it is redefining human-computer interactions.

View Conference

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 2000371

Country of Publication:: United States

Language:: English

References (22)

Exascale Computing in the United States Kothe, Douglas; Lee, Stephen; Qualters, Irene Computing in Science & Engineering, Vol. 21, Issue 1 https://doi.org/10.1109/MCSE.2018.2875366	journal	January 2019
Julia: A Fresh Approach to Numerical Computing Bezanson, Jeff; Edelman, Alan; Karpinski, Stefan SIAM Review, Vol. 59, Issue 1 https://doi.org/10.1137/141000671	journal	January 2017
An empirical evaluation of GitHub copilot's code suggestions Nguyen, Nhan; Nadi, Sarah Proceedings of the 19th International Conference on Mining Software Repositories https://doi.org/10.1145/3524842.3528470	conference	May 2022
Choose your programming copilot Sobania, Dominik; Briesch, Martin; Rothlauf, Franz Proceedings of the Genetic and Evolutionary Computation Conference https://doi.org/10.1145/3512290.3528700	conference	July 2022
Is GitHub copilot a substitute for human pair-programming? Imai, Saki Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings https://doi.org/10.1145/3510454.3522684	conference	May 2022
Numba: a LLVM-based Python JIT compiler Lam, Siu Kwan; Pitrou, Antoine; Seibert, Stanley Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC - LLVM '15 https://doi.org/10.1145/2833157.2833162	conference	January 2015
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming Finnie-Ansley, James; Denny, Paul; Becker, Brett A. Australasian Computing Education Conference https://doi.org/10.1145/3511861.3511863	conference	February 2022
Assessing the quality of GitHub copilot’s code generation Yetistiren, Burak; Ozsoy, Isik; Tuzun, Eray Proceedings of the 18th International Conference on Predictive Models and Data Analytics in Software Engineering https://doi.org/10.1145/3558489.3559072	conference	November 2022
Kokkos: Enabling manycore performance portability through polymorphic memory access patterns Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel Journal of Parallel and Distributed Computing, Vol. 74, Issue 12 https://doi.org/10.1016/j.jpdc.2014.07.003	journal	December 2014
Experimental Multi-threading Support for the Julia Programming Language Knopp, Tobias 2014 First Workshop for High Performance Technical Computing in Dynamic Languages https://doi.org/10.1109/HPTCDL.2014.11	conference	November 2014
Psb2 Helmuth, Thomas; Kelly, Peter Proceedings of the Genetic and Evolutionary Computation Conference https://doi.org/10.1145/3449639.3459285	conference	June 2021
Code Generation Using Machine Learning: A Systematic Review Dehaerne, Enrique; Dey, Bappaditya; Halder, Sandip IEEE Access, Vol. 10 https://doi.org/10.1109/ACCESS.2022.3196347	journal	January 2022
The International Exascale Software Project roadmap Dongarra, Jack; Beckman, Pete; Moore, Terry The International Journal of High Performance Computing Applications, Vol. 25, Issue 1 https://doi.org/10.1177/1094342010391989	journal	January 2011
Fortran Backus, J. W.; Heising, W. P. IEEE Transactions on Electronic Computers, Vol. EC-13, Issue 4 https://doi.org/10.1109/PGEC.1964.263818	journal	August 1964
Using GitHub Copilot to Solve Simple Programming Problems Wermelinger, Michel Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 https://doi.org/10.1145/3545945.3569830	conference	March 2023
GPT-3: Its Nature, Scope, Limits, and Consequences Floridi, Luciano; Chiriatti, Massimo Minds and Machines, Vol. 30, Issue 4 https://doi.org/10.1007/s11023-020-09548-1	journal	November 2020
Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language Denny, Paul; Kumar, Viraj; Giacaman, Nasser Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 https://doi.org/10.1145/3545945.3569823	conference	March 2023
Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models Vaithilingam, Priyan; Zhang, Tianyi; Glassman, Elena L. CHI Conference on Human Factors in Computing Systems Extended Abstracts https://doi.org/10.1145/3491101.3519665	conference	April 2022
Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes Godoy, William F.; Valero-Lara, Pedro; Dettling, T. Elise 2023 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) https://doi.org/10.1109/IPDPSW59300.2023.00068	conference	May 2023
Extreme Heterogeneity 2018 - Productive Computational Science in the Era of Extreme Heterogeneity: Report for DOE ASCR Workshop on Extreme Heterogeneity Vetter, Jeffrey S.; Brightwell, Ron; Gokhale, Maya https://doi.org/10.2172/1473756	report	December 2018
Analysis of the popularity of programming languages in open source software communities Lu, Dongdong; Wu, Jie; Sheng, Yongxiang 2020 International Conference on Big Data and Social Sciences (ICBDSS) https://doi.org/10.1109/ICBDSS51270.2020.00033	conference	August 2020
PyCUDA and PyOpenCL: A scripting-based approach to GPU run-time code generation Klöckner, Andreas; Pinto, Nicolas; Lee, Yunsup Parallel Computing, Vol. 38, Issue 3 https://doi.org/10.1016/j.parco.2011.09.001	journal	March 2012

Similar Records

Large language model evaluation for high–performance computing software development

Journal Article · Wed Sep 04 00:00:00 EDT 2024 · Concurrency and Computation. Practice and Experience · OSTI ID:2474767

Evaluating performance and portability of high-level programming models: Julia, Python/Numba, and Kokkos on exascale nodes

Conference · Mon May 01 00:00:00 EDT 2023 · OSTI ID:1994693

KokkACC: Enhancing Kokkos with OpenACC

Conference · Tue Nov 01 00:00:00 EDT 2022 · OSTI ID:2000279

Evaluation of OpenAI Codex for HPC Parallel Programming Models Kernel Generation

Citation Formats

References (22)

Similar Records

Related Subjects