skip to main content

DOE PAGESDOE PAGES

Title: Optimizing CMS build infrastructure via Apache Mesos

The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC) at CERN consists of 6M lines of in-house code, developed over a decade by nearly 1000 physicists, as well as a comparable amount of general use open-source code. A critical ingredient to the success of the construction and early operation of the WLCG was the convergence, around the year 2000, on the use of a homogeneous environment of commodity x86-64 processors and Linux.Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other applications on a dynamically shared pool of nodes. Lastly, we present how we migrated our continuous integration system to schedule jobs on a relatively small Apache Mesos enabled cluster and how this resulted in better resource usage, higher peak performance and lower latency thanks to the dynamic scheduling capabilities of Mesos.
Authors:
 [1] ;  [2] ;  [3] ;  [1] ;  [4] ;  [1]
  1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  2. Univ. di Torino, Torino (Italy)
  3. Princeton Univ., Princeton, NJ (United States)
  4. Univ. de los Andes, Bogota (Colombia)
Publication Date:
Report Number(s):
arXiv:1507.07429; FERMILAB-CONF-15-661-CMS
Journal ID: ISSN 1742-6588; 1385108
Grant/Contract Number:
AC02-07CH11359
Type:
Accepted Manuscript
Journal Name:
Journal of Physics. Conference Series
Additional Journal Information:
Journal Volume: 664; Journal Issue: 6; Journal ID: ISSN 1742-6588
Publisher:
IOP Publishing
Research Org:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
Country of Publication:
United States
Language:
English
Subject:
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS
OSTI Identifier:
1346389