skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Continuous Integration and Delivery for HPC: Using Singularity and Jenkins

Abstract

Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.

Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
Sponsoring Org.:
USDOE Office of Science (SC); National Science Foundation (NSF)
OSTI Identifier:
1544252
Grant/Contract Number:  
FE0026298
Resource Type:
Accepted Manuscript
Journal Name:
International Conference Proceedings Series (ICPS)
Additional Journal Information:
Journal Name: International Conference Proceedings Series (ICPS); Journal Volume: 2018; Conference: Practice and Experience in Advanced Research Computing - PEARC '18, Pittsburgh, PA (United States), 22-26 Jul 2018; Related Information: ISBN 978-1-4503-6446-1; Journal ID: ISSN 0000-2002
Publisher:
ACM Press
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; continuous integration; continuous delivery; continuous deployment; Singularity; containers; software builds; software automation; MFIX-Exa

Citation Formats

Continuous Integration and Delivery for HPC: Using Singularity and Jenkins. United States: N. p., 2018. Web. doi:10.1145/3219104.3219147.
Continuous Integration and Delivery for HPC: Using Singularity and Jenkins. United States. doi:10.1145/3219104.3219147.
Sun . "Continuous Integration and Delivery for HPC: Using Singularity and Jenkins". United States. doi:10.1145/3219104.3219147. https://www.osti.gov/servlets/purl/1544252.
@article{osti_1544252,
title = {Continuous Integration and Delivery for HPC: Using Singularity and Jenkins},
author = {None, None},
abstractNote = {Continuous integration, delivery, and deployment (CICD) is widely used in DevOps communities, as it allows for teams of all sizes to deploy rapidly-changing hardware and software resources quickly and confidently. In this paper, we will describe how University of Colorado Boulder Research Computing has adopted these practices on the RMACC Summit supercomputer to allow system engineers and researchers alike to capitalize on the benefits of CICD-centric development workflows. We will introduce the topic of CICD at a high level and describe how such practices can ease common software management challenges for High-Performance Computing (HPC) resources. We will then document the infrastructure deployed for Summit, and explain how software such as Jenkins and Singularity enabled adaptation for an HPC environment. We will conclude with two case studies discussing the use of our CICD infrastructure: one case study from the perspective of a system engineer maintaining user-facing resources, and the other case study from the perspective of a researcher developing, maintaining, and using the MFiX-Exa codebase.},
doi = {10.1145/3219104.3219147},
journal = {International Conference Proceedings Series (ICPS)},
number = ,
volume = 2018,
place = {United States},
year = {2018},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

To Docker or Not to Docker: A Security Perspective
journal, September 2016

  • Combe, Theo; Martin, Antony; Di Pietro, Roberto
  • IEEE Cloud Computing, Vol. 3, Issue 5
  • DOI: 10.1109/MCC.2016.100

Automated System Health and Performance Benchmarking Platform: High Performance Computing Test Harness with Jenkins
conference, January 2017

  • Voss, Joseph; Garcia, Joe A.; Proctor, W. Cyrus
  • Proceedings of the HPC Systems Professionals Workshop on - HPCSYSPROS'17
  • DOI: 10.1145/3155105.3155106

Deploying RMACC Summit: An HPC Resource for the Rocky Mountain Region
conference, January 2017

  • Anderson, Jonathon; Burns, Patrick J.; Milroy, Daniel
  • Proceedings of the Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact - PEARC17
  • DOI: 10.1145/3093338.3093379

Management of an academic HPC cluster: The UL experience
conference, July 2014

  • Varrette, Sebastien; Bouvry, Pascal; Cartiaux, Hyacinthe
  • 2014 International Conference on High Performance Computing & Simulation (HPCS)
  • DOI: 10.1109/HPCSim.2014.6903792

The Spack package manager: bringing order to HPC software chaos
conference, January 2015

  • Gamblin, Todd; LeGendre, Matthew; Collette, Michael R.
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15
  • DOI: 10.1145/2807591.2807623

Singularity: Scientific containers for mobility of compute
journal, May 2017


Docker [Software engineering]
journal, May 2015


Charliecloud: unprivileged containers for user-defined software stacks in HPC
conference, January 2017

  • Priedhorsky, Reid; Randles, Tim
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17
  • DOI: 10.1145/3126908.3126925

Singularity Registry: Open Source Registry for Singularity Images
journal, October 2017

  • Sochat, Vanessa
  • The Journal of Open Source Software, Vol. 2, Issue 18
  • DOI: 10.21105/joss.00426

Modern Scientific Software Management Using EasyBuild and Lmod
conference, November 2014

  • Geimer, Markus; Hoste, Kenneth; McLay, Robert
  • 2014 First International Workshop on HPC User Support Tools (HUST)
  • DOI: 10.1109/HUST.2014.8

    Works referencing / citing this record:

    A Continuous Integration-Based Framework for Software Management
    conference, July 2019

    • Khuvis, Samuel; You, Zhi-Qiang; Na, Heechang
    • PEARC '19: Practice and Experience in Advanced Research Computing, Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (learning)
    • DOI: 10.1145/3332186.3332219