Linux containers for fun and profit in HPC

Priedhorsky, Reid; Randles, Timothy C.

Title: Linux containers for fun and profit in HPC

Full Record
Other Related Research

Abstract

This article outlines options for user-defined software stacks from an HPC perspective. Here, we argue that a lightweight approach based on Linux containers is most suitable for HPC centers because it provides the best balance between maximizing service of user needs and minimizing risks. We also discuss how containers work and several implementations, including Charliecloud, our own open-source solution developed at Los Alamos.

Authors:

^[1];

^[1]

Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Publication Date:: Sun Oct 01 00:00:00 EDT 2017

Research Org.:: Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1398926

Report Number(s):: LA-UR-17-25168
Journal ID: ISSN 1044-6397

Grant/Contract Number:: AC52-06NA25396

Resource Type:: Accepted Manuscript

Journal Name:: ;Login

Additional Journal Information:: Journal Volume: 42; Journal Issue: 3; Journal ID: ISSN 1044-6397

Publisher:: USENIX Association

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Priedhorsky, Reid, and Randles, Timothy C. Linux containers for fun and profit in HPC.  United States: N. p., 2017. 
Web.

Copy to clipboard


                    Priedhorsky, Reid, & Randles, Timothy C. Linux containers for fun and profit in HPC.  United States.

Copy to clipboard


                    Priedhorsky, Reid, and Randles, Timothy C. Sun .  
"Linux containers for fun and profit in HPC".  United States.  https://www.osti.gov/servlets/purl/1398926.

Copy to clipboard


                    
@article{osti_1398926,

  title        = {Linux containers for fun and profit in HPC},

  author       = {Priedhorsky, Reid and Randles, Timothy C.},

  abstractNote = {This article outlines options for user-defined software stacks from an HPC perspective. Here, we argue that a lightweight approach based on Linux containers is most suitable for HPC centers because it provides the best balance between maximizing service of user needs and minimizing risks. We also discuss how containers work and several implementations, including Charliecloud, our own open-source solution developed at Los Alamos.},

  doi          = {},

  journal      = {;Login},

  number       = 3,

  volume       = 42,

  place        = {United States},

  year         = {Sun Oct 01 00:00:00 EDT 2017},

  month        = {Sun Oct 01 00:00:00 EDT 2017}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

The DOI is not currently available

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Charliecloud: Unprivileged containers for user-defined software stacks in HPC

Conference Priedhorsky, Reid ; Randles, Timothy C.

Supercomputing centers are seeing increasing demand for user-defined software stacks (UDSS), instead of or in addition to the stack provided by the center. These UDSS support user needs such as complex dependencies or build requirements, externally required configurations, portability, and consistency. The challenge for centers is to provide these services in a usable manner while minimizing the risks: security, support burden, missing functionality, and performance. We present Charliecloud, which uses the Linux user and mount namespaces to run industry-standard Docker containers with no privileged operations or daemons on center resources. Our simple approach avoids most security risks while maintaining accessmore »« less
https://doi.org/10.2172/1296650

Full Text Available
Balancing Performance and Portability with Containers in HPC: An OpenSHMEM Example

Conference Naughton III, Thomas ; Sorrillo, Lawrence ; Simpson, Adam B. ; ...

There is a growing interest in using Linux containers to streamline software development and application deployment. A container enables the user to bundle the salient elements of the software stack from an application’s perspective. In this paper, we discuss initial experiences in using the Open MPI implementation of OpenSHMEM with containers on HPC resources. We provide a brief overview of two container runtimes, Docker & Singularity, highlighting elements that are of interest for HPC users. The Docker platform offers a rich set of services that are widely used in enterprise environments, whereas Singularity is an emerging container runtime that ismore »« less
https://doi.org/10.1007/978-3-319-73814-7_9

Full Text Available
STAR Data Production Workflow on HPC: Lessons Learned & Best Practices

Conference Poat, Michael ; Poat, MD ; Lauret, J ; ...

The Solenoidal Tracker at RHIC (STAR) is a multi-national supported experiment located at Brookhaven National Lab. The raw physics data captured from the detector is on the order of tens of PBytes per data acquisition campaign, which makes STAR fit well within the definition of a big data science experiment. The production of the data has typically run on standard nodes or on standard Grid computing environments. All embedding simulations (complex workflow mixing real and simulated events) have been run on standard Linux resources at the National Energy Research Scientific Computing Center (NERSC) aka PDSF. However, HPC resources such asmore »« less
https://doi.org/10.1088/1742-6596/1525/1/012068

Full Text Available
Userspace Squash Filesystem for Launching Linux Containers on HPC Systems [Thesis]

Technical Report Goff, Ronald Shane

The demand for user defined software stacks (UDSS) has been increasing in the high-performance computing (HPC) community. Container technology has become popular due to the flexibility and isolation it provides to HPC users. Container images must be available to all nodes involved for use in HPC and can be distributed to compute nodes in a variety of ways. A common method for container image distribution is to simply copy the container image to memory on each compute node, which can be time consuming at scale, and uses valuable memory on each node. The kernel mounted squash filesystem (squashfs) has provenmore »« less
https://doi.org/10.2172/1874141

Full Text Available
HPC-Colony: Services and Interfaces to Aupport Systems With Very Large Numbers of Processors

Technical Report Jones, T ; Kale, L ; Moreira, J ; ...

The HPC-Colony Project, a collaboration with Lawrence Livermore National Laboratory, the University of Illinois at Urbana-Champaign and IBM, is focused on services and interfaces for very large numbers of processors. Advances in parallel systems in the last decade have delivered phenomenal progress in the overall capability available to a single parallel application. Several systems with peak capability of over 100TF are already available and systems are expected to exceed 1PF within a few years. Despite these impressive advances in peak performance capability, the sustained performance of these systems continues to fall as a percentage of the peak capability. Initial analysismore »« less
https://doi.org/10.2172/902273

Full Text Available

Similar Records