skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Supporting HEP Data as the Exascale Era Approaches

Abstract

We describe our work on the use of HPC data formats and storage technologies to efficiently describe, store and access High Energy Physics (HEP) data. In the HEP community, important data sets are large, currently at the scale of tens of PiB, and are expected to grow dramatically in the next decade. These data require different data representations—ranging from complex C++ objects to tabular data—that are used at the many varied stages of data analysis. Traditionally, these data have been stored in large collections of files at each stage. This approach does not scale well on HPC resources. To obtain good scalability, without requiring that analysts have parallel programming expertise, our work stresses the use of HPCoptimized tools and implicit data-parallelism. We describe data organization and the performance and scaling of data processing using examples from three HEP experiments (LArIAT, NOvA and DUNE), and for three different stages of analysis. We compare the performance of our HPC-oriented solutions to the traditional HEP approaches.

Authors:
 [1];  [2];  [1];  [3];  [3];  [2];  [3];  [3];  [2];  [3];  [2];  [4];  [3]
  1. Colorado State U., Fort Collins
  2. Argonne
  3. Fermilab
  4. sousaae@ucmail.uc.edu
Publication Date:
Research Org.:
Argonne National Lab. (ANL), Argonne, IL (United States); Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
OSTI Identifier:
1480092
Report Number(s):
FERMILAB-POSTER-18-103-CD
1701094
DOE Contract Number:  
AC02-07CH11359
Resource Type:
Conference
Country of Publication:
United States
Language:
English

Citation Formats

Buchanan, Norman, Dorier, Matthieu, Doyle, Derek, Green, Christopher, Kowalkowski, James, Latham, Robert, Norman, Andrew, Paterno, Marc, Ross, Robert, Sehrish, Saba, Snyder, Shane, Sousa, Alexandre, and White, Brandon. Supporting HEP Data as the Exascale Era Approaches. United States: N. p., 2018. Web.
Buchanan, Norman, Dorier, Matthieu, Doyle, Derek, Green, Christopher, Kowalkowski, James, Latham, Robert, Norman, Andrew, Paterno, Marc, Ross, Robert, Sehrish, Saba, Snyder, Shane, Sousa, Alexandre, & White, Brandon. Supporting HEP Data as the Exascale Era Approaches. United States.
Buchanan, Norman, Dorier, Matthieu, Doyle, Derek, Green, Christopher, Kowalkowski, James, Latham, Robert, Norman, Andrew, Paterno, Marc, Ross, Robert, Sehrish, Saba, Snyder, Shane, Sousa, Alexandre, and White, Brandon. Fri . "Supporting HEP Data as the Exascale Era Approaches". United States. https://www.osti.gov/servlets/purl/1480092.
@article{osti_1480092,
title = {Supporting HEP Data as the Exascale Era Approaches},
author = {Buchanan, Norman and Dorier, Matthieu and Doyle, Derek and Green, Christopher and Kowalkowski, James and Latham, Robert and Norman, Andrew and Paterno, Marc and Ross, Robert and Sehrish, Saba and Snyder, Shane and Sousa, Alexandre and White, Brandon},
abstractNote = {We describe our work on the use of HPC data formats and storage technologies to efficiently describe, store and access High Energy Physics (HEP) data. In the HEP community, important data sets are large, currently at the scale of tens of PiB, and are expected to grow dramatically in the next decade. These data require different data representations—ranging from complex C++ objects to tabular data—that are used at the many varied stages of data analysis. Traditionally, these data have been stored in large collections of files at each stage. This approach does not scale well on HPC resources. To obtain good scalability, without requiring that analysts have parallel programming expertise, our work stresses the use of HPCoptimized tools and implicit data-parallelism. We describe data organization and the performance and scaling of data processing using examples from three HEP experiments (LArIAT, NOvA and DUNE), and for three different stages of analysis. We compare the performance of our HPC-oriented solutions to the traditional HEP approaches.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {10}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: