Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Exploiting analytics techniques in CMS computing monitoring

Journal Article · · Journal of Physics. Conference Series
 [1];  [2];  [3];  [4];  [3]
  1. Univ. di Balogna (Italy)
  2. Cornell Univ., Ithaca, NY (United States)
  3. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  4. Univ. of Vilnius (Lithuania)
+ Show Author Affiliations
The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.
Research Organization:
Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
Sponsoring Organization:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
Grant/Contract Number:
AC02-07CH11359
OSTI ID:
1415640
Report Number(s):
FERMILAB-CONF--16-735-CD; 1638624
Journal Information:
Journal of Physics. Conference Series, Journal Name: Journal of Physics. Conference Series Journal Issue: 9 Vol. 898; ISSN 1742-6588
Publisher:
IOP PublishingCopyright Statement
Country of Publication:
United States
Language:
English

Figures / Tables (5)

Figure 1(p. 3)figure Figure 1
Table 1(p. 5)table Table 1
Table 2(p. 5)table Table 2
Figure 2(p. 6)figure Figure 2
Figure 3(p. 6)figure Figure 3

Similar Records

XRootD popularity on hadoop clusters
Journal Article · Tue Nov 21 19:00:00 EST 2017 · Journal of Physics. Conference Series · OSTI ID:1831862
Large-scale seismic waveform quality metric calculation using Hadoop
Journal Article · Thu May 26 20:00:00 EDT 2016 · Computers and Geosciences · OSTI ID:1262167
MARIANE: MApReduce Implementation Adapted for HPC Environments
Conference · Wed Jul 06 00:00:00 EDT 2011 · OSTI ID:1093573

Related Subjects

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS
97 MATHEMATICS AND COMPUTING