Grid collector: An event catalog with automated file management
Abstract
High Energy Nuclear Physics (HENP) experiments such as STAR at BNL and ATLAS at CERN produce large amounts of data that are stored as files on mass storage systems in computer centers. In these files, the basic unit of data is an event. Analysis is typically performed on a selected set of events. The files containing these events have to be located, copied from mass storage systems to disks before analysis, and removed when no longer needed. These file management tasks are tedious and time consuming. Typically, all events contained in the files are read into memory before a selection is made. Since the time to read the events dominate the overall execution time, reading the unwanted event needlessly increases the analysis time. The Grid Collector is a set of software modules that works together to address these two issues. It automates the file management tasks and provides ''direct'' access to the selected events for analyses. It is currently integrated with the STAR analysis framework. The users can select events based on tags, such as, ''production date between March 10 and 20, and the number of charged tracks > 100.'' The Grid Collector locates the files containing relevant events, transfersmore »
- Authors:
- Publication Date:
- Research Org.:
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
- Sponsoring Org.:
- USDOE Director. Office of Science. Office of Advanced Scientific Computing Research (US)
- OSTI Identifier:
- 835164
- Report Number(s):
- LBNL-55563
R&D Project: KS3310; TRN: US0407488
- DOE Contract Number:
- AC03-76SF00098
- Resource Type:
- Conference
- Resource Relation:
- Conference: IEEE Nuclear Science Symposium 2003, Portland, OR (US), 10/19/2003--10/25/2003; Other Information: PBD: 17 Oct 2003
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 73 NUCLEAR PHYSICS AND RADIATION PHYSICS; 99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; BNL; CERN; COMPUTERS; MANAGEMENT; NUCLEAR PHYSICS; PRODUCTION; STORAGE; GRID EVENT CATALOG DISK CACHE
Citation Formats
Wu, Kesheng, Zhang, Wei-Ming, Sim, Alexander, Gu, Junmin, and Shoshani, Arie. Grid collector: An event catalog with automated file management. United States: N. p., 2003.
Web.
Wu, Kesheng, Zhang, Wei-Ming, Sim, Alexander, Gu, Junmin, & Shoshani, Arie. Grid collector: An event catalog with automated file management. United States.
Wu, Kesheng, Zhang, Wei-Ming, Sim, Alexander, Gu, Junmin, and Shoshani, Arie. 2003.
"Grid collector: An event catalog with automated file management". United States. https://www.osti.gov/servlets/purl/835164.
@article{osti_835164,
title = {Grid collector: An event catalog with automated file management},
author = {Wu, Kesheng and Zhang, Wei-Ming and Sim, Alexander and Gu, Junmin and Shoshani, Arie},
abstractNote = {High Energy Nuclear Physics (HENP) experiments such as STAR at BNL and ATLAS at CERN produce large amounts of data that are stored as files on mass storage systems in computer centers. In these files, the basic unit of data is an event. Analysis is typically performed on a selected set of events. The files containing these events have to be located, copied from mass storage systems to disks before analysis, and removed when no longer needed. These file management tasks are tedious and time consuming. Typically, all events contained in the files are read into memory before a selection is made. Since the time to read the events dominate the overall execution time, reading the unwanted event needlessly increases the analysis time. The Grid Collector is a set of software modules that works together to address these two issues. It automates the file management tasks and provides ''direct'' access to the selected events for analyses. It is currently integrated with the STAR analysis framework. The users can select events based on tags, such as, ''production date between March 10 and 20, and the number of charged tracks > 100.'' The Grid Collector locates the files containing relevant events, transfers the files across the Grid if necessary, and delivers the events to the analysis code through the familiar iterators. There has been some research efforts to address the file management issues, the Grid Collector is unique in that it addresses the event access issue together with the file management issues. This makes it more useful to a large variety of users.},
doi = {},
url = {https://www.osti.gov/biblio/835164},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Fri Oct 17 00:00:00 EDT 2003},
month = {Fri Oct 17 00:00:00 EDT 2003}
}