Exploiting analytics techniques in CMS computing monitoring
- Univ. di Balogna (Italy)
- Cornell Univ., Ithaca, NY (United States)
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Univ. of Vilnius (Lithuania)
The CMS experiment has collected an enormous volume of metadata about its computing operations in its monitoring systems, describing its experience in operating all of the CMS workflows on all of the Worldwide LHC Computing Grid Tiers. Data mining efforts into all these information have rarely been done, but are of crucial importance for a better understanding of how CMS did successful operations, and to reach an adequate and adaptive modelling of the CMS operations, in order to allow detailed optimizations and eventually a prediction of system behaviours. These data are now streamed into the CERN Hadoop data cluster for further analysis. Specific sets of information (e.g. data on how many replicas of datasets CMS wrote on disks at WLCG Tiers, data on which datasets were primarily requested for analysis, etc) were collected on Hadoop and processed with MapReduce applications profiting of the parallelization on the Hadoop cluster. We present the implementation of new monitoring applications on Hadoop, and discuss the new possibilities in CMS computing monitoring introduced with the ability to quickly process big data sets from mulltiple sources, looking forward to a predictive modeling of the system.
- Research Organization:
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), High Energy Physics (HEP)
- Grant/Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1415640
- Report Number(s):
- FERMILAB-CONF-16-735-CD; 1638624
- Journal Information:
- Journal of Physics. Conference Series, Vol. 898, Issue 9; Conference: 22nd International Conference on Computing in High Energy and Nuclear Physics, San Francisco, CA, 10/10-10/14/2016; ISSN 1742-6588
- Publisher:
- IOP PublishingCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Similar Records
Center for Technology for Advanced Scientific Componet Software (TASCS)
Large scale and low latency analysis facilities for the CMS experiment: Development and operational aspects