Grid collector: An event catalog with automated file management
High Energy Nuclear Physics (HENP) experiments such as STAR at BNL and ATLAS at CERN produce large amounts of data that are stored as files on mass storage systems in computer centers. In these files, the basic unit of data is an event. Analysis is typically performed on a selected set of events. The files containing these events have to be located, copied from mass storage systems to disks before analysis, and removed when no longer needed. These file management tasks are tedious and time consuming. Typically, all events contained in the files are read into memory before a selection is made. Since the time to read the events dominate the overall execution time, reading the unwanted event needlessly increases the analysis time. The Grid Collector is a set of software modules that works together to address these two issues. It automates the file management tasks and provides ''direct'' access to the selected events for analyses. It is currently integrated with the STAR analysis framework. The users can select events based on tags, such as, ''production date between March 10 and 20, and the number of charged tracks > 100.'' The Grid Collector locates the files containing relevant events, transfers the files across the Grid if necessary, and delivers the events to the analysis code through the familiar iterators. There has been some research efforts to address the file management issues, the Grid Collector is unique in that it addresses the event access issue together with the file management issues. This makes it more useful to a large variety of users.
- Research Organization:
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
- Sponsoring Organization:
- USDOE Director. Office of Science. Office of Advanced Scientific Computing Research (US)
- DOE Contract Number:
- AC03-76SF00098
- OSTI ID:
- 835164
- Report Number(s):
- LBNL-55563; R&D Project: KS3310; TRN: US0407488
- Resource Relation:
- Conference: IEEE Nuclear Science Symposium 2003, Portland, OR (US), 10/19/2003--10/25/2003; Other Information: PBD: 17 Oct 2003
- Country of Publication:
- United States
- Language:
- English
Similar Records
Grid Collector: Using an event catalog to speed up user analysisin distributed environment
Grid Collector: Using an event catalog to speed up user analysisin distributed environment