Virtualization of the ATLAS software environment on a shared HPC system

Gamel, Anton J.; Schnoor, U.; Meier, K.; Bührer, F.; Schumacher, M.

doi:10.1088/1742-6596/1085/3/032036

Title: Virtualization of the ATLAS software environment on a shared HPC system

Abstract

The shared HPC cluster NEMO at the University of Freiburg has been made available to local ATLAS users through the provisioning of virtual machines incorporating the ATLAS software environment analogously to a WLCG center. This concept allows to run both data analysis and production on the HPC host system which is connected to the existing Tier2/Tier3 infrastructure. Schedulers of the two clusters were integrated in a dynamic, on-demand way. An automatically generated, fully functional virtual machine image provides access to the local user environment. The performance in the virtualized environment is evaluated for typical High-Energy Physics applications.

Authors:

Gamel, Anton J. ^[1]; Schnoor, U. ^[1]; Meier, K. ^[1]; Bührer, F. ^[1]; Schumacher, M. ^[1]

Albert-Ludwigs-Univ. Freiburg, Freiburg im Breisgau (Germany)

Publication Date:: Sat Sep 01 00:00:00 EDT 2018

Research Org.:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1544179

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Physics. Conference Series

Additional Journal Information:: Journal Volume: 1085; Journal ID: ISSN 1742-6588

Publisher:: IOP Publishing

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS

Citation Formats


                    Gamel, Anton J., Schnoor, U., Meier, K., Bührer, F., and Schumacher, M. Virtualization of the ATLAS software environment on a shared HPC system.  United States: N. p., 2018. 
Web.  doi:10.1088/1742-6596/1085/3/032036.

Copy to clipboard


                    Gamel, Anton J., Schnoor, U., Meier, K., Bührer, F., & Schumacher, M. Virtualization of the ATLAS software environment on a shared HPC system.  United States.  https://doi.org/10.1088/1742-6596/1085/3/032036

Copy to clipboard


                    Gamel, Anton J., Schnoor, U., Meier, K., Bührer, F., and Schumacher, M. Sat .  
"Virtualization of the ATLAS software environment on a shared HPC system".  United States.  https://doi.org/10.1088/1742-6596/1085/3/032036.  https://www.osti.gov/servlets/purl/1544179.

Copy to clipboard


                    
@article{osti_1544179,

  title        = {Virtualization of the ATLAS software environment on a shared HPC system},

  author       = {Gamel, Anton J. and Schnoor, U. and Meier, K. and Bührer, F. and Schumacher, M.},

  abstractNote = {The shared HPC cluster NEMO at the University of Freiburg has been made available to local ATLAS users through the provisioning of virtual machines incorporating the ATLAS software environment analogously to a WLCG center. This concept allows to run both data analysis and production on the HPC host system which is connected to the existing Tier2/Tier3 infrastructure. Schedulers of the two clusters were integrated in a dynamic, on-demand way. An automatically generated, fully functional virtual machine image provides access to the local user environment. The performance in the virtualized environment is evaluated for typical High-Energy Physics applications.},

  doi          = {10.1088/1742-6596/1085/3/032036},

  journal      = {Journal of Physics. Conference Series},

  number       = ,

  volume       = 1085,

  place        = {United States},

  year         = {Sat Sep 01 00:00:00 EDT 2018},

  month        = {Sat Sep 01 00:00:00 EDT 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1088/1742-6596/1085/3/032036

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

The Worldwide LHC Computing Grid (worldwide LCG)
journal, July 2007

Shiers, Jamie
Computer Physics Communications, Vol. 177, Issue 1-2
DOI: 10.1016/j.cpc.2007.02.021

Dynamic provisioning of a HEP computing infrastructure on a shared hybrid HPC system
journal, October 2016

Meier, Konrad; Fleig, Georg; Hauth, Thomas
Journal of Physics: Conference Series, Vol. 762
DOI: 10.1088/1742-6596/762/1/012012

A brief introduction to PYTHIA 8.1
journal, June 2008

Sjöstrand, Torbjörn; Mrenna, Stephen; Skands, Peter
Computer Physics Communications, Vol. 178, Issue 11
DOI: 10.1016/j.cpc.2008.01.036

Similar Records in DOE PAGES and OSTI.GOV collections:

Lifting and Dropping VMs to Dynamically Transition Between Time- and Space-sharing for Large-Scale HPC Systems

Conference Gordon, Nick ; Lange, Jack

As HPC environments increasingly integrate with edge based systems, system architectures will need to handle a broader class of workloads and scheduling requirements. One result of this shift will be the need to simultaneously support bulk-synchronous parallel (BSP) and on-demand service based applications on the same infrastructure. This in turn will require that future resource management approaches utilize both space-shared as well as time-shared resource scheduling strategies. In this work we introduce the concept of "VM-lifting'' (and its inverse "VM-Dropping'') which allows dynamically switching an HPC workload between space-shared and time-shared scheduling regimes. Our work targets co-kernel based HPC systemmore »« less
https://doi.org/10.1145/3502181.3531471

Full Text Available
Virtual Environment for Testing Software-Defined Networking Solutions for Scientific Workflows

Conference Liu, Qiang ; Rao, Nageswara S. ; Sen, Satyabrata ; ...

Recent developments in software-defined infrastructures promise that scientific workflows utilizing supercomputers, instruments, and storage systems will be dynamically composed and orchestrated using software at unprecedented speed and scale in the near future. Testing of the underlying networking software, particularly during initial exploratory stages, remains a challenge due to potential disruptions, and resource allocation and coordination needed over the multi-domain physical infrastructure. To overcome these challenges, we develop the Virtual Science Network Environment (VSNE) that emulates the multi-site host, storage, and network infrastructure using Virtual Machines (VMs), wherein the production and nascent software can be tested. Within each VM, which representsmore »« less
https://doi.org/10.1145/3217197.3217202

Full Text Available
Virtual Environment for Testing Software-Defined Networking Solutions for Scientific Workflows

Conference Liu, Qiang ; Katramatos, Dimitrios

Recent developments in software-defined infrastructures promise that scientific workflows utilizing supercomputers, instruments, and storage systems will be dynamically composed and orchestrated using software at unprecedented speed and scale in the near future. Testing of the underlying networking software, particularly during initial exploratory stages, remains a challenge due to potential disruptions, and resource allocation and coordination needed over the multi-domain physical infrastructure. To overcome these challenges, we develop the Virtual Science Network Environment (VSNE) that emulates the multi-site host, storage, and network infrastructure using Virtual Machines (VMs), wherein the production and nascent software can be tested. Within each VM, which representsmore »« less
https://doi.org/10.1145/3217197.3217202

Full Text Available
The HARNESS Workbench: Unified and Adaptive Access to Diverse HPC Platforms

Technical Report Sunderam, Vaidy S

The primary goal of the Harness WorkBench (HWB) project is to investigate innovative software environments that will help enhance the overall productivity of applications science on diverse HPC platforms. Two complementary frameworks were designed: one, a virtualized command toolkit for application building, deployment, and execution, that provides a common view across diverse HPC systems, in particular the DOE leadership computing platforms (Cray, IBM, SGI, and clusters); and two, a unified runtime environment that consolidates access to runtime services via an adaptive framework for execution-time and post processing activities. A prototype of the first was developed based on the concept ofmore »« less
https://doi.org/10.2172/1036700

Full Text Available
Using Pilot Jobs and CernVM File System for Simplified Use of Containers and Software Distribution

Conference Urs, Namratha ; Mambelli, Marco ; Dykstra, David - TBD

High Energy Physics (HEP) experiments entail an abundance of computing resources, i.e. sites, to run simulations and analyses by processing data. This requirement is fulfilled by local batch farms, grid sites, private/commercial clouds, and supercomputing centers via High Throughput Computing (HTC). The growing needs of such experiments and resources being prone to trends of heterogeneity make it difficult for physicists to handle these resources directly. Additionally, HEP collaborations heavily rely on data and software releases, typically in the order of tens of gigabytes, while conducting simulations and analyses. Hence, aspects of scalability, reliability, and maintenance become crucial with regards tomore »« less
https://doi.org/10.2172/1824852

Full Text Available

Similar Records

Title: Virtualization of the ATLAS software environment on a shared HPC system

Abstract

Citation Formats

The Worldwide LHC Computing Grid (worldwide LCG) journal, July 2007

Dynamic provisioning of a HEP computing infrastructure on a shared hybrid HPC system journal, October 2016

A brief introduction to PYTHIA 8.1 journal, June 2008

The Worldwide LHC Computing Grid (worldwide LCG)
journal, July 2007

Dynamic provisioning of a HEP computing infrastructure on a shared hybrid HPC system
journal, October 2016

A brief introduction to PYTHIA 8.1
journal, June 2008