Big Data in HEP: A comprehensive use case study

doi:https://doi.org/10.1088/1742-6596/898/7/072012

Title: Big Data in HEP: A comprehensive use case study

Full Record
Other Related Research

Abstract

Experimental Particle Physics has been at the forefront of analyzing the worlds largest datasets for decades. The HEP community was the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems collectively called Big Data technologies have emerged to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and promise a fresh look at analysis of very large datasets and could potentially reduce the time-to-physics with increased interactivity. In this talk, we present an active LHC Run 2 analysis, searching for dark matter with the CMS detector, as a testbed for Big Data technologies. We directly compare the traditional NTuple-based analysis with an equivalent analysis using Apache Spark on the Hadoop ecosystem and beyond. In both cases, we start the analysis with the official experiment data formats and produce publication physics plots. Lastly, we will discuss advantages and disadvantages of each approach and give an outlook on further studies needed.

Authors:

^[1]; Cremonesi, Matteo ^[1]; Elmer, Peter ^[2]; Jayatilaka, Bo ^[1]; Kowalkowski, Jim ^[1]; Pivarski, Jim ^[2]; Sehrish, Saba ^[1]; Surez, Cristina Mantilla ^[3]; Svyatkovskiy, Alexey ^[2]; Tran, Nhan ^[1]

Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Princeton Univ., Princeton, NJ (United States)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Johns Hopkins Univ., Baltimore, MD (United States)

Publication Date:: 2017-11-23

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)

OSTI Identifier:: 1343953

Report Number(s):: arXiv:1703.04171; FERMILAB-CONF-17-028-CD
Journal ID: ISSN 1742-6588; 1517266

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Journal Article: Accepted Manuscript

Journal Name:: Journal of Physics. Conference Series

Additional Journal Information:: Journal Volume: 898; Conference: Dark Interactions: perspective from theory and experiment, Upton, NY (United States), 4-7 Oct 2016; Journal ID: ISSN 1742-6588

Publisher:: IOP Publishing

Country of Publication:: United States

Language:: English

Subject:: 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats


                    Gutsche, Oliver, Cremonesi, Matteo, Elmer, Peter, Jayatilaka, Bo, Kowalkowski, Jim, Pivarski, Jim, Sehrish, Saba, Surez, Cristina Mantilla, Svyatkovskiy, Alexey, and Tran, Nhan. Big Data in HEP: A comprehensive use case study.  United States: N. p., 2017. 
        Web.  doi:10.1088/1742-6596/898/7/072012.

Copy to clipboard


                    Gutsche, Oliver, Cremonesi, Matteo, Elmer, Peter, Jayatilaka, Bo, Kowalkowski, Jim, Pivarski, Jim, Sehrish, Saba, Surez, Cristina Mantilla, Svyatkovskiy, Alexey, & Tran, Nhan. Big Data in HEP: A comprehensive use case study.  United States.  https://doi.org/10.1088/1742-6596/898/7/072012

Copy to clipboard


                    Gutsche, Oliver, Cremonesi, Matteo, Elmer, Peter, Jayatilaka, Bo, Kowalkowski, Jim, Pivarski, Jim, Sehrish, Saba, Surez, Cristina Mantilla, Svyatkovskiy, Alexey, and Tran, Nhan. Thu .  
        "Big Data in HEP: A comprehensive use case study".  United States.  https://doi.org/10.1088/1742-6596/898/7/072012.  https://www.osti.gov/servlets/purl/1343953.

Copy to clipboard


                    
@article{osti_1343953,

  title        = {Big Data in HEP: A comprehensive use case study},

  author       = {Gutsche, Oliver and Cremonesi, Matteo and Elmer, Peter and Jayatilaka, Bo and Kowalkowski, Jim and Pivarski, Jim and Sehrish, Saba and Surez, Cristina Mantilla and Svyatkovskiy, Alexey and Tran, Nhan},

  abstractNote = {Experimental Particle Physics has been at the forefront of analyzing the worlds largest datasets for decades. The HEP community was the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems collectively called Big Data technologies have emerged to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and promise a fresh look at analysis of very large datasets and could potentially reduce the time-to-physics with increased interactivity. In this talk, we present an active LHC Run 2 analysis, searching for dark matter with the CMS detector, as a testbed for Big Data technologies. We directly compare the traditional NTuple-based analysis with an equivalent analysis using Apache Spark on the Hadoop ecosystem and beyond. In both cases, we start the analysis with the official experiment data formats and produce publication physics plots. Lastly, we will discuss advantages and disadvantages of each approach and give an outlook on further studies needed.},

  doi          = {10.1088/1742-6596/898/7/072012},

  url          = {https://www.osti.gov/biblio/1343953},
  journal      = {Journal of Physics. Conference Series},

  issn         = {1742-6588},

  number       = ,

  volume       = 898,

  place        = {United States},

  year         = {2017},

  month        = {11}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1088/1742-6596/898/7/072012

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

CMS Analysis and Data Reduction with Apache Spark

Journal Article Gutsche, Oliver ; Canali, Luca ; Cremer, Illia ; ... - Journal of Physics. Conference Series

Experimental Particle Physics has been at the forefront of analyzing the world's largest datasets for decades. The HEP community was among the first to develop suitable software and computing tools for this task. In recent times, new toolkits and systems for distributed data processing, collectively called "Big Data" technologies have emerged from industry and open source projects to support the analysis of Petabyte and Exabyte datasets in industry. While the principles of data analysis in HEP have not changed (filtering and transforming experiment-specific data formats), these new technologies use different approaches and tools, promising a fresh look at analysis ofmore »« less
https://doi.org/10.1088/1742-6596/1085/4/042030

Full Text Available
Using Big Data Technologies for HEP Analysis

Journal Article Cremonesi, Matteo ; Bellini, Claudio ; Bian, Bianny ; ... - TBD

The HEP community is approaching an era were the excellent performances of the particle accelerators in delivering collision at high rate will force the experiments to record a large amount of information. The growing size of the datasets could potentially become a limiting factor in the capability to produce scientific results timely and efficiently. Recently, new technologies and new approaches have been developed in industry to answer to the necessity to retrieve information as quickly as possible to analyze PB and EB datasets. Providing the scientists with these modern computing tools will lead to rethinking the principles of data analysismore »« less
Full Text Available
Center for Technology for Advanced Scientific Componet Software (TASCS)

Technical Report Govindaraju, Madhusudhan

Advanced Scientific Computing Research Computer Science FY 2010Report Center for Technology for Advanced Scientific Component Software: Distributed CCA State University of New York, Binghamton, NY, 13902 Summary The overall objective of Binghamton's involvement is to work on enhancements of the CCA environment, motivated by the applications and research initiatives discussed in the proposal. This year we are working on re-focusing our design and development efforts to develop proof-of-concept implementations that have the potential to significantly impact scientific components. We worked on developing parallel implementations for non-hydrostatic code and worked on a model coupling interface for biogeochemical computations coded in MATLAB.more »« less
https://doi.org/10.2172/1092881

Full Text Available
Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing

Journal Article Klimentov, A. ; Buncic, P. ; De, K. ; ... - Journal of Physics. Conference Series

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS and ALICE are the largest collaborations ever assembled in the sciences and are at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, both experiments rely on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Managementmore »« less
Cited by 1
https://doi.org/10.1088/1742-6596/608/1/012040

Full Text Available
Large-scale seismic waveform quality metric calculation using Hadoop

Journal Article Magana-Zook, Steven ; Gaylord, Jessie M. ; Knapp, Douglas R. ; ... - Computers and Geosciences

Here in this work we investigated the suitability of Hadoop MapReduce and Apache Spark for large-scale computation of seismic waveform quality metrics by comparing their performance with that of a traditional distributed implementation. The Incorporated Research Institutions for Seismology (IRIS) Data Management Center (DMC) provided 43 terabytes of broadband waveform data of which 5.1 TB of data were processed with the traditional architecture, and the full 43 TB were processed using MapReduce and Spark. Maximum performance of ~0.56 terabytes per hour was achieved using all 5 nodes of the traditional implementation. We noted that I/O dominated processing, and that I/Omore »« less
Cited by 1
https://doi.org/10.1016/j.cageo.2016.05.012

Full Text Available

Similar Records