skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Exploiting Analytics Techniques in CMS Computing Monitoring

Abstract

The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.

Authors:
 [1];  [2];  [3];  [4];  [3]
  1. Bologna U.
  2. Cornell U.
  3. Fermilab
  4. Vilnius U.
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
OSTI Identifier:
1415640
Report Number(s):
FERMILAB-CONF-16-735-CD
1638624
DOE Contract Number:  
AC02-07CH11359
Resource Type:
Conference
Resource Relation:
Journal Name: J.Phys.Conf.Ser.; Journal Volume: 898; Journal Issue: 9; Conference: 22nd International Conference on Computing in High Energy and Nuclear Physics, San Francisco, CA, 10/10-10/14/2016
Country of Publication:
United States
Language:
English

Citation Formats

Bonacorsi, D., Kuznetsov, V., Magini, N., Repečka, A., and Vaandering, E. Exploiting Analytics Techniques in CMS Computing Monitoring. United States: N. p., 2017. Web. doi:10.1088/1742-6596/898/9/092030.
Bonacorsi, D., Kuznetsov, V., Magini, N., Repečka, A., & Vaandering, E. Exploiting Analytics Techniques in CMS Computing Monitoring. United States. doi:10.1088/1742-6596/898/9/092030.
Bonacorsi, D., Kuznetsov, V., Magini, N., Repečka, A., and Vaandering, E. Wed . "Exploiting Analytics Techniques in CMS Computing Monitoring". United States. doi:10.1088/1742-6596/898/9/092030. https://www.osti.gov/servlets/purl/1415640.
@article{osti_1415640,
title = {Exploiting Analytics Techniques in CMS Computing Monitoring},
author = {Bonacorsi, D. and Kuznetsov, V. and Magini, N. and Repečka, A. and Vaandering, E.},
abstractNote = {The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.},
doi = {10.1088/1742-6596/898/9/092030},
journal = {J.Phys.Conf.Ser.},
number = 9,
volume = 898,
place = {United States},
year = {Wed Nov 22 00:00:00 EST 2017},
month = {Wed Nov 22 00:00:00 EST 2017}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: