Distributing User Code with the CernVM FileSystem
Journal Article
·
· EPJ Web of Conferences (Online)
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
The CernVM FileSystem (CVMFS) is widely used in High Throughput Computing to efficiently distributed experiment code. However, the standard CVMFS publishing tools are designed for a small group of people from each experiment to maintain common software, and the tools are not a good fit for publishing software from numerous users in each experiment. As a result, most user code, such as code to do specific physics analyses, is still sent with every job to the place the job is run. That process is relatively inefficient, especially when the user code is large. To overcome these limitations, we have built a CVMFS user code publication system. This publication system enables users to still submit their code with their jobs but the code is distributed and accessed through the standard CVMFS infrastructure. The user code is automatically deleted from CVMFS after a period of no use. Most of the software for the system is available as a single self-contained open source rpm called cvmfs-user-pub and is available for other deployments.
- Research Organization:
- Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
- Contributing Organization:
- FIFE
- Grant/Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1623372
- Alternate ID(s):
- OSTI ID: 23171940
- Report Number(s):
- FERMILAB--CONF-20-108-SCD; oai:inspirehep.net:1795488
- Journal Information:
- EPJ Web of Conferences (Online), Journal Name: EPJ Web of Conferences (Online) Vol. 245; ISSN 2100-014X
- Publisher:
- EDP SciencesCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Status and future perspectives of CernVM-FS
|
journal | December 2012 |
FIFE-Jobsub: a grid submission system for intensity frontier experiments at Fermilab
|
journal | June 2014 |
Similar Records
Distributing User Code with the CernVM FileSystem
Engineering the CernVM-Filesystem as a High Bandwidth Distributed Filesystem for Auxiliary Physics Data
Using Pilot Jobs and CernVM File System for Simplified Use of Containers and Software Distribution
Conference
·
Wed Nov 06 23:00:00 EST 2019
·
OSTI ID:1630708
Engineering the CernVM-Filesystem as a High Bandwidth Distributed Filesystem for Auxiliary Physics Data
Conference
·
Tue Dec 22 23:00:00 EST 2015
· J.Phys.Conf.Ser.
·
OSTI ID:1247506
Using Pilot Jobs and CernVM File System for Simplified Use of Containers and Software Distribution
Conference
·
Thu Dec 31 23:00:00 EST 2020
· No journal information
·
OSTI ID:1824852