skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Optimizing CMS build infrastructure via Apache Mesos

Journal Article · · Journal of Physics. Conference Series
 [1];  [2];  [3];  [1];  [4];  [1]
  1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  2. Univ. di Torino, Torino (Italy)
  3. Princeton Univ., Princeton, NJ (United States)
  4. Univ. de los Andes, Bogota (Colombia)
+ Show Author Affiliations

The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC) at CERN consists of 6M lines of in-house code, developed over a decade by nearly 1000 physicists, as well as a comparable amount of general use open-source code. A critical ingredient to the success of the construction and early operation of the WLCG was the convergence, around the year 2000, on the use of a homogeneous environment of commodity x86-64 processors and Linux.Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora, and other applications on a dynamically shared pool of nodes. Lastly, we present how we migrated our continuous integration system to schedule jobs on a relatively small Apache Mesos enabled cluster and how this resulted in better resource usage, higher peak performance and lower latency thanks to the dynamic scheduling capabilities of Mesos.

Research Organization:
Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
Sponsoring Organization:
USDOE Office of Science (SC), High Energy Physics (HEP)
Grant/Contract Number:
AC02-07CH11359
OSTI ID:
1346389
Report Number(s):
arXiv:1507.07429; FERMILAB-CONF-15-661-CMS; 1385108
Journal Information:
Journal of Physics. Conference Series, Vol. 664, Issue 6; ISSN 1742-6588
Publisher:
IOP PublishingCopyright Statement
Country of Publication:
United States
Language:
English

References (1)

Large-scale cluster management at Google with Borg conference April 2015

Cited By (1)

A cloud-agnostic queuing system to support the implementation of deadline-based application execution policies journal December 2019

Similar Records

Center for Technology for Advanced Scientific Componet Software (TASCS)
Technical Report · Sun Oct 31 00:00:00 EDT 2010 · OSTI ID:1346389
Analyzing petabytes of data with Hadoop
Multimedia · Fri Aug 21 00:00:00 EDT 2009 · OSTI ID:1346389
Pooling the resources of the CMS Tier-1 sites
Journal Article · Wed Dec 23 00:00:00 EST 2015 · Journal of Physics. Conference Series · OSTI ID:1346389
+11 more

Related Subjects

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS