skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Using Big Data Technologies for HEP Analysis

Abstract

The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysis in HEP, making the overall scientific process faster and smoother. In this paper, we are presenting the latest developments and the most recent results on the usage of Apache Spark for HEP analysis. The study aims at evaluating the efficiency of the application of the new tools both quantitatively, by measuring the performances, and qualitatively, focusing on the user experience. The first goal is achieved by developing a data reduction facility: working together with CERN Openlab and Intel, CMS replicates a real physics search using Spark-based technologies, with the ambition of reducing 1 PB of public data in 5 hours,more » collected by the CMS experiment, to 1 TB of data in a format suitable for physics analysis. The second goal is achieved by implementing multiple physics use-cases in Apache Spark using as input preprocessed datasets derived from official CMS data and simulation. By performing different end-analyses up to the publication plots on different hardware, feasibility, usability and portability are compared to the ones of a traditional ROOT-based workflow.« less

Authors:
 [1];  [2];  [2];  [3];  [3];  [4];  [5];  [3];  [1];  [6];  [1];  [3];  [2];  [7];  [3];  [8];  [6];  [4];  [4];  [6]
  1. Fermilab
  2. Intel, Santa Clara
  3. CERN
  4. Princeton U.
  5. Flatiron Inst., New York
  6. Genoa U.
  7. Vanderbilt U.
  8. UC, San Diego
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
OSTI Identifier:
1529354
Report Number(s):
arXiv:1901.07143; FERMILAB-PUB-19-037-CD-PPD
1716266
DOE Contract Number:  
AC02-07CH11359
Resource Type:
Journal Article
Journal Name:
TBD
Additional Journal Information:
Journal Name: TBD
Country of Publication:
United States
Language:
English

Citation Formats

Cremonesi, Matteo, Bellini, Claudio, Bian, Bianny, Canali, Luca, Dimakopoulos, Vasileios, Elmer, Peter, Fisk, Ian, Girone, Maria, Gutsche, Oliver, Hoh, Siew-Yan, Jayatilaka, Bo, Khristenko, Viktor, Luiselli, Andrea, Melo, Andrew, Evangelos, Evangelos, Olivito, Dominick, Pazzini, Jacopo, Pivarski, Jim, Svyatkovskiy, Alexey, and Zanetti, Marco. Using Big Data Technologies for HEP Analysis. United States: N. p., 2019. Web.
Cremonesi, Matteo, Bellini, Claudio, Bian, Bianny, Canali, Luca, Dimakopoulos, Vasileios, Elmer, Peter, Fisk, Ian, Girone, Maria, Gutsche, Oliver, Hoh, Siew-Yan, Jayatilaka, Bo, Khristenko, Viktor, Luiselli, Andrea, Melo, Andrew, Evangelos, Evangelos, Olivito, Dominick, Pazzini, Jacopo, Pivarski, Jim, Svyatkovskiy, Alexey, & Zanetti, Marco. Using Big Data Technologies for HEP Analysis. United States.
Cremonesi, Matteo, Bellini, Claudio, Bian, Bianny, Canali, Luca, Dimakopoulos, Vasileios, Elmer, Peter, Fisk, Ian, Girone, Maria, Gutsche, Oliver, Hoh, Siew-Yan, Jayatilaka, Bo, Khristenko, Viktor, Luiselli, Andrea, Melo, Andrew, Evangelos, Evangelos, Olivito, Dominick, Pazzini, Jacopo, Pivarski, Jim, Svyatkovskiy, Alexey, and Zanetti, Marco. Mon . "Using Big Data Technologies for HEP Analysis". United States. https://www.osti.gov/servlets/purl/1529354.
@article{osti_1529354,
title = {Using Big Data Technologies for HEP Analysis},
author = {Cremonesi, Matteo and Bellini, Claudio and Bian, Bianny and Canali, Luca and Dimakopoulos, Vasileios and Elmer, Peter and Fisk, Ian and Girone, Maria and Gutsche, Oliver and Hoh, Siew-Yan and Jayatilaka, Bo and Khristenko, Viktor and Luiselli, Andrea and Melo, Andrew and Evangelos, Evangelos and Olivito, Dominick and Pazzini, Jacopo and Pivarski, Jim and Svyatkovskiy, Alexey and Zanetti, Marco},
abstractNote = {The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysis in HEP, making the overall scientific process faster and smoother. In this paper, we are presenting the latest developments and the most recent results on the usage of Apache Spark for HEP analysis. The study aims at evaluating the efficiency of the application of the new tools both quantitatively, by measuring the performances, and qualitatively, focusing on the user experience. The first goal is achieved by developing a data reduction facility: working together with CERN Openlab and Intel, CMS replicates a real physics search using Spark-based technologies, with the ambition of reducing 1 PB of public data in 5 hours, collected by the CMS experiment, to 1 TB of data in a format suitable for physics analysis. The second goal is achieved by implementing multiple physics use-cases in Apache Spark using as input preprocessed datasets derived from official CMS data and simulation. By performing different end-analyses up to the publication plots on different hardware, feasibility, usability and portability are compared to the ones of a traditional ROOT-based workflow.},
doi = {},
journal = {TBD},
number = ,
volume = ,
place = {United States},
year = {2019},
month = {1}
}