Data handling with SAM and art at the NO vA experiment
During operations, NOvA produces between 5,000 and 7,000 raw files per day with peaks in excess of 12,000. These files must be processed in several stages to produce fully calibrated and reconstructed analysis files. In addition, many simulated neutrino interactions must be produced and processed through the same stages as data. To accommodate the large volume of data and Monte Carlo, production must be possible both on the Fermilab grid and on off-site farms, such as the ones accessible through the Open Science Grid. To handle the challenge of cataloging these files and to facilitate their off-line processing, we have adopted the SAM system developed at Fermilab. SAM indexes files according to metadata, keeps track of each file's physical locations, provides dataset management facilities, and facilitates data transfer to off-site grids. To integrate SAM with Fermilab's art software framework and the NOvA production workflow, we have developed methods to embed metadata into our configuration files, art files, and standalone ROOT files. A module in the art framework propagates the embedded information from configuration files into art files, and from input art files to output art files, allowing us to maintain a complete processing history within our files. Embedding metadata inmore »
- Authors:
-
[1]
; [2]
; [3]
; [4]
; [5]
; [4]
; [4]
; [6]
; [6]
- Univ. of Cincinnati, Cincinnati, OH (United States)
- California Institute of Technology, Pasadena, CA (United States)
- Indiana Univ., Bloomington, IN (United States)
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Tufts Univ., Medford, MA (United States)
- Univ. of Minnesota, Minneapolis, MN (United States)
- Publication Date:
- Report Number(s):
- FERMILAB-CONF-15-202-ND
Journal ID: ISSN 1742-6588; 1413833
- Grant/Contract Number:
- AC02-07CH11359
- Type:
- Accepted Manuscript
- Journal Name:
- Journal of Physics. Conference Series
- Additional Journal Information:
- Journal Volume: 664; Journal Issue: 4; Conference: 21st International Conference on Computing in High Energy and Nuclear Physics, Okinawa (Japan), 13-17 Apr 2015; Journal ID: ISSN 1742-6588
- Publisher:
- IOP Publishing
- Research Org:
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Sponsoring Org:
- USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
- OSTI Identifier:
- 1250776
Aurisano, A., Backhouse, C., Davies, G. S., Illingworth, R., Mayer, N., Mengel, M., Norman, A., Rocco, D., and Zirnstein, J.. Data handling with SAM and art at the NOvA experiment. United States: N. p.,
Web. doi:10.1088/1742-6596/664/4/042001.
Aurisano, A., Backhouse, C., Davies, G. S., Illingworth, R., Mayer, N., Mengel, M., Norman, A., Rocco, D., & Zirnstein, J.. Data handling with SAM and art at the NOvA experiment. United States. doi:10.1088/1742-6596/664/4/042001.
Aurisano, A., Backhouse, C., Davies, G. S., Illingworth, R., Mayer, N., Mengel, M., Norman, A., Rocco, D., and Zirnstein, J.. 2015.
"Data handling with SAM and art at the NOvA experiment". United States.
doi:10.1088/1742-6596/664/4/042001. https://www.osti.gov/servlets/purl/1250776.
@article{osti_1250776,
title = {Data handling with SAM and art at the NOvA experiment},
author = {Aurisano, A. and Backhouse, C. and Davies, G. S. and Illingworth, R. and Mayer, N. and Mengel, M. and Norman, A. and Rocco, D. and Zirnstein, J.},
abstractNote = {During operations, NOvA produces between 5,000 and 7,000 raw files per day with peaks in excess of 12,000. These files must be processed in several stages to produce fully calibrated and reconstructed analysis files. In addition, many simulated neutrino interactions must be produced and processed through the same stages as data. To accommodate the large volume of data and Monte Carlo, production must be possible both on the Fermilab grid and on off-site farms, such as the ones accessible through the Open Science Grid. To handle the challenge of cataloging these files and to facilitate their off-line processing, we have adopted the SAM system developed at Fermilab. SAM indexes files according to metadata, keeps track of each file's physical locations, provides dataset management facilities, and facilitates data transfer to off-site grids. To integrate SAM with Fermilab's art software framework and the NOvA production workflow, we have developed methods to embed metadata into our configuration files, art files, and standalone ROOT files. A module in the art framework propagates the embedded information from configuration files into art files, and from input art files to output art files, allowing us to maintain a complete processing history within our files. Embedding metadata in configuration files also allows configuration files indexed in SAM to be used as inputs to Monte Carlo production jobs. Further, SAM keeps track of the input files used to create each output file. Parentage information enables the construction of self-draining datasets which have become the primary production paradigm used at NOvA. In this study we will present an overview of SAM at NOvA and how it has transformed the file production framework used by the experiment.},
doi = {10.1088/1742-6596/664/4/042001},
journal = {Journal of Physics. Conference Series},
number = 4,
volume = 664,
place = {United States},
year = {2015},
month = {12}
}
- Free Publicly Accessible Full Text
-
Accepted Manuscript (DOE)
- Publisher's Version of Record
- https://doi.org/10.1088/1742-6596/664/4/042001