skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Continuous Integration and Delivery for HPC: Using Singularity and Jenkins

Abstract

Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.

Authors:
 [1];  [1];  [1]
  1. Univ. of Colorado, Boulder, CO (United States). Research Computing
Publication Date:
Research Org.:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
Sponsoring Org.:
USDOE Office of Science (SC); National Science Foundation (NSF)
OSTI Identifier:
1544252
Grant/Contract Number:  
FE0026298
Resource Type:
Accepted Manuscript
Resource Relation:
Conference: Proceedings of the Practice and Experience on Advanced Research Computing - PEARC '18, Pittsburgh, PA (United States), 22-26 Jul 2018
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; continuous integration; continuous delivery; continuous deployment; Singularity; containers; software builds; software automation; MFIX-Exa

Citation Formats

Sampedro, Zebula, Holt, Aaron, and Hauser, Thomas. Continuous Integration and Delivery for HPC: Using Singularity and Jenkins. United States: N. p., 2018. Web. doi:10.1145/3219104.3219147.
Sampedro, Zebula, Holt, Aaron, & Hauser, Thomas. Continuous Integration and Delivery for HPC: Using Singularity and Jenkins. United States. doi:10.1145/3219104.3219147.
Sampedro, Zebula, Holt, Aaron, and Hauser, Thomas. Sun . "Continuous Integration and Delivery for HPC: Using Singularity and Jenkins". United States. doi:10.1145/3219104.3219147. https://www.osti.gov/servlets/purl/1544252.
@article{osti_1544252,
title = {Continuous Integration and Delivery for HPC: Using Singularity and Jenkins},
author = {Sampedro, Zebula and Holt, Aaron and Hauser, Thomas},
abstractNote = {Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.},
doi = {10.1145/3219104.3219147},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share: