skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Big Data in HEP: A comprehensive use case study

Abstract

Experimental Particle Physics has been at the forefront of analyzing the worlds largest datasets for decades. The HEP community was the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems collectively called Big Data technologies have emerged to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and promise a fresh look at analysis of very large datasets and could potentially reduce the time-to-physics with increased interactivity. In this talk, we present an active LHC Run 2 analysis, searching for dark matter with the CMS detector, as a testbed for Big Data technologies. We directly compare the traditional NTuple-based analysis with an equivalent analysis using Apache Spark on the Hadoop ecosystem and beyond. In both cases, we start the analysis with the official experiment data formats and produce publication physics plots. Lastly, we will discuss advantages and disadvantages of each approach and give an outlook on further studies needed.

Authors:
ORCiD logo [1];  [1];  [2];  [1];  [1];  [2];  [1];  [3];  [2];  [1]
  1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  2. Princeton Univ., Princeton, NJ (United States)
  3. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Johns Hopkins Univ., Baltimore, MD (United States)
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
OSTI Identifier:
1343953
Report Number(s):
arXiv:1703.04171; FERMILAB-CONF-17-028-CD
Journal ID: ISSN 1742-6588; 1517266
Grant/Contract Number:
AC02-07CH11359
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Journal of Physics. Conference Series
Additional Journal Information:
Journal Volume: 898; Conference: Dark Interactions: perspective from theory and experiment, Upton, NY (United States), 4-7 Oct 2016; Journal ID: ISSN 1742-6588
Publisher:
IOP Publishing
Country of Publication:
United States
Language:
English
Subject:
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats

Gutsche, Oliver, Cremonesi, Matteo, Elmer, Peter, Jayatilaka, Bo, Kowalkowski, Jim, Pivarski, Jim, Sehrish, Saba, Surez, Cristina Mantilla, Svyatkovskiy, Alexey, and Tran, Nhan. Big Data in HEP: A comprehensive use case study. United States: N. p., 2017. Web. doi:10.1088/1742-6596/898/7/072012.
Gutsche, Oliver, Cremonesi, Matteo, Elmer, Peter, Jayatilaka, Bo, Kowalkowski, Jim, Pivarski, Jim, Sehrish, Saba, Surez, Cristina Mantilla, Svyatkovskiy, Alexey, & Tran, Nhan. Big Data in HEP: A comprehensive use case study. United States. doi:10.1088/1742-6596/898/7/072012.
Gutsche, Oliver, Cremonesi, Matteo, Elmer, Peter, Jayatilaka, Bo, Kowalkowski, Jim, Pivarski, Jim, Sehrish, Saba, Surez, Cristina Mantilla, Svyatkovskiy, Alexey, and Tran, Nhan. Thu . "Big Data in HEP: A comprehensive use case study". United States. doi:10.1088/1742-6596/898/7/072012. https://www.osti.gov/servlets/purl/1343953.
@article{osti_1343953,
title = {Big Data in HEP: A comprehensive use case study},
author = {Gutsche, Oliver and Cremonesi, Matteo and Elmer, Peter and Jayatilaka, Bo and Kowalkowski, Jim and Pivarski, Jim and Sehrish, Saba and Surez, Cristina Mantilla and Svyatkovskiy, Alexey and Tran, Nhan},
abstractNote = {Experimental Particle Physics has been at the forefront of analyzing the worlds largest datasets for decades. The HEP community was the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems collectively called Big Data technologies have emerged to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and promise a fresh look at analysis of very large datasets and could potentially reduce the time-to-physics with increased interactivity. In this talk, we present an active LHC Run 2 analysis, searching for dark matter with the CMS detector, as a testbed for Big Data technologies. We directly compare the traditional NTuple-based analysis with an equivalent analysis using Apache Spark on the Hadoop ecosystem and beyond. In both cases, we start the analysis with the official experiment data formats and produce publication physics plots. Lastly, we will discuss advantages and disadvantages of each approach and give an outlook on further studies needed.},
doi = {10.1088/1742-6596/898/7/072012},
journal = {Journal of Physics. Conference Series},
number = ,
volume = 898,
place = {United States},
year = {Thu Nov 23 00:00:00 EST 2017},
month = {Thu Nov 23 00:00:00 EST 2017}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share: