Overcoming Challenges to Continuous Integration in HPC
Journal Article
·
· Computing in Science and Engineering
- Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
Continuous integration (CI) has become a ubiquitous practice in modern software development, with major code hosting services offering free automation on popular platforms. CI offers major benefits, as it enables detecting bugs in code prior to committing changes. While high-performance computing (HPC) research relies heavily on software, HPC machines are not considered “common” platforms. This presents several challenges that hinder the adoption of CI in HPC environments, making it difficult to maintain bug-free HPC projects, and resulting in adverse effects on the research community. Here we explore the challenges that impede HPC CI, such as hardware diversity, security, isolation, administrative policies, and non-standard authentication, environments, and job submission mechanisms. We propose several solutions that could enhance the quality of HPC software and the experience of developers. Implementing these solutions would require significant changes at HPC centers, but if these changes are made, it would ultimately enable faster and better science.
- Research Organization:
- Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
- Sponsoring Organization:
- USDOE National Nuclear Security Administration (NNSA)
- Grant/Contract Number:
- AC52-07NA27344
- OSTI ID:
- 2283634
- Report Number(s):
- LLNL--JRNL-846623; 1071069
- Journal Information:
- Computing in Science and Engineering, Journal Name: Computing in Science and Engineering Journal Issue: 6 Vol. 24; ISSN 1521-9615
- Publisher:
- IEEE Computer SocietyCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Feasibility of Running Singularity Containers with Hybrid MPI on NASA High-End Computing Resources
|
conference | November 2021 |
EasyBuild: Building Software with Ease
|
conference | November 2012 |
The Spack package manager: bringing order to HPC software chaos
|
conference | January 2015 |
Similar Records
Creating Continuous Integration Infrastructure for Software Development on U.S. Department of Energy High-Performance Computing Systems
Tandem Predictions for HPC Jobs
Tandem Predictions for HPC Jobs: Preprint
Journal Article
·
Tue Feb 06 19:00:00 EST 2024
· Computing in Science and Engineering
·
OSTI ID:2345319
Tandem Predictions for HPC Jobs
Conference
·
Wed Jul 17 00:00:00 EDT 2024
·
OSTI ID:2447811
Tandem Predictions for HPC Jobs: Preprint
Conference
·
Mon Aug 04 20:00:00 EDT 2025
·
OSTI ID:2575622