An inconvenient truth : file-level metadata and in-file metadata caching in the (file-agnostic) ATLAS event store.

Conference · Tue Jan 01 04:00:00 EST 2008 · J. Phys.: Conf. Ser.

DOI:https://doi.org/10.1088/1742-6596/119/4/042022· OSTI ID:971479

Malon, D; van Gemmeren, P; Hawkings, R; Schaffer, A; CERN,

In the ATLAS event store, files are sometimes 'an inconvenient truth.' From the point of view of the ATLAS distributed data management system, files are too small - datasets are the units of interest. From the point of view of the ATLAS event store architecture, files are simply a physical clustering optimization: the units of interest are event collections - sets of events that satisfy common conditions or selection predicates - and such collections may or may not have been accumulated into files that contain those events and no others. It is nonetheless important to maintain file-level metadata, and to cache metadata in event data files. When such metadata may or may not be present in files, or when values may have been updated after files are written and replicated, a clear and transparent model for metadata retrieval from the file itself or from remote databases is required. In this paper we describe how ATLAS reconciles its file and non-file paradigms, the machinery for associating metadata with files and event collections, and the infrastructure for metadata propagation from input to output for provenance record management and related purposes.

• Research Organization: Argonne National Laboratory (ANL)
• Sponsoring Organization: SC
• DOE Contract Number: AC02-06CH11357
• OSTI ID: 971479
• Report Number(s): ANL-HEP-CP-07-73
• Conference Information: Journal Name: J. Phys.: Conf. Ser. Journal Issue: 2008 Journal Volume: 119
• Country of Publication: United States
• Language: ENGLISH

Related Subjects

73 NUCLEAR PHYSICS AND RADIATION PHYSICS
ARCHITECTURE
MACHINERY
MANAGEMENT
NUCLEAR PHYSICS
OPTIMIZATION