Continuous Integration and Delivery for HPC

Sampedro, Zebula; Holt, Aaron; Hauser, Thomas

doi:10.1145/3219104.3219147

Continuous Integration and Delivery for HPC

Journal Article · Sun Jul 22 00:00:00 EDT 2018 · Proceedings of the Practice and Experience on Advanced Research Computing

DOI:https://doi.org/10.1145/3219104.3219147· OSTI ID:1544252

Sampedro, Zebula ^[1]; Holt, Aaron ^[1]; Hauser, Thomas ^[1]

Univ. of Colorado, Boulder, CO (United States)

Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Organization:: USDOE Office of Energy Efficiency and Renewable Energy (EERE), Renewable Power Office. Wind Energy Technologies Office

Grant/Contract Number:: FE0026298

OSTI ID:: 1544252

Journal Information:: Proceedings of the Practice and Experience on Advanced Research Computing, Journal Name: Proceedings of the Practice and Experience on Advanced Research Computing Vol. 2018

Publisher:: ACM PressCopyright Statement

Country of Publication:: United States

Language:: English

References (12)

SLURM: Simple Linux Utility for Resource Management Yoo, Andy B.; Jette, Morris A.; Grondona, Mark Job Scheduling Strategies for Parallel Processing https://doi.org/10.1007/10968987_3	book	January 2003
Orchestrating Docker Containers in the HPC Environment Higgins, Joshua; Holmes, Violeta; Venters, Colin Lecture Notes in Computer Science https://doi.org/10.1007/978-3-319-20119-1_36	book	January 2015
Management of an academic HPC cluster: The UL experience Varrette, Sebastien; Bouvry, Pascal; Cartiaux, Hyacinthe 2014 International Conference on High Performance Computing & Simulation (HPCS) https://doi.org/10.1109/HPCSim.2014.6903792	conference	July 2014
Modern Scientific Software Management Using EasyBuild and Lmod Geimer, Markus; Hoste, Kenneth; McLay, Robert 2014 First International Workshop on HPC User Support Tools (HUST) https://doi.org/10.1109/HUST.2014.8	conference	November 2014
To Docker or Not to Docker: A Security Perspective Combe, Theo; Martin, Antony; Di Pietro, Roberto IEEE Cloud Computing, Vol. 3, Issue 5 https://doi.org/10.1109/MCC.2016.100	journal	September 2016
Docker [Software engineering] Anderson, Charles IEEE Software, Vol. 32, Issue 3 https://doi.org/10.1109/MS.2015.62	journal	May 2015
The Spack package manager: bringing order to HPC software chaos Gamblin, Todd; LeGendre, Matthew; Collette, Michael R. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807623	conference	January 2015
Deploying RMACC Summit: An HPC Resource for the Rocky Mountain Region Anderson, Jonathon; Burns, Patrick J.; Milroy, Daniel Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact - PEARC17 https://doi.org/10.1145/3093338.3093379	conference	January 2017
Charliecloud: unprivileged containers for user-defined software stacks in HPC Priedhorsky, Reid; Randles, Tim Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17 https://doi.org/10.1145/3126908.3126925	conference	January 2017
Automated System Health and Performance Benchmarking Platform: High Performance Computing Test Harness with Jenkins Voss, Joseph; Garcia, Joe A.; Proctor, W. Cyrus Proceedings of the HPC Systems Professionals Workshop on - HPCSYSPROS'17 https://doi.org/10.1145/3155105.3155106	conference	January 2017
Singularity: Scientific containers for mobility of compute Kurtzer, Gregory M.; Sochat, Vanessa; Bauer, Michael W. PLOS ONE, Vol. 12, Issue 5 https://doi.org/10.1371/journal.pone.0177459	journal	May 2017
Singularity Registry: Open Source Registry for Singularity Images Sochat, Vanessa The Journal of Open Source Software, Vol. 2, Issue 18 https://doi.org/10.21105/joss.00426	journal	October 2017

Cited By (1)

A Continuous Integration-Based Framework for Software Management Khuvis, Samuel; You, Zhi-Qiang; Na, Heechang PEARC '19: Practice and Experience in Advanced Research Computing, Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning) https://doi.org/10.1145/3332186.3332219	conference	July 2019

Similar Records

Mastering HPC Runtime Prediction: From Observing Patterns to a Methodological Approach

Conference · Sun Sep 10 00:00:00 EDT 2023 · OSTI ID:2246634

MFIX-Exa: A path toward exascale CFD-DEM simulations

Journal Article · Thu Apr 15 20:00:00 EDT 2021 · International Journal of High Performance Computing Applications · OSTI ID:1841162

Mastering HPC Runtime Prediction: From Observing Patterns to a Methodological Approach: Preprint

Conference · Sun Jun 25 20:00:00 EDT 2023 · OSTI ID:1988023

Related Subjects

97 MATHEMATICS AND COMPUTING
MFIX-Exa
Singularity
containers
continuous delivery
continuous deployment
continuous integration
software automation
software builds

Continuous Integration and Delivery for HPC

Citation Formats

References (12)

Cited By (1)

Similar Records

Related Subjects