Label-based Virtual Directories In dCache

Conference · Wed Jan 01 04:00:00 EST 2025 · EPJ Web Conf.

DOI:https://doi.org/10.1051/epjconf/202533701274· OSTI ID:3009885

Sahakyan, Marina ^[1]; Green, Christopher ^[2]; Litvintsev, Dmitry ^[2]; Chitrapu, Krishnaveni ^[3]; Morschel, Lea ^[1]; Mkrtchyan, Tigran ^[1]; Meyer, Svenja ^[1]

---

1. DESY
2. Fermilab
3. Natl. Supercomputing Ctr., Tianjin

Traditional filesystems organize data in directories. These directories are typically a collection of files whose grouping is based on a single criterion, e.g., the starting date of an experiment, experiment name, beamline ID, measurement device, or instrument. However, each file in a directory can belong to several logical groups, such as a special event type, experiment condition, or a part of a selected dataset. dCache is a storage system developed to store large amounts of scientific data, used by many HEP and Photon Science experiments. With recent developments in dCache, we have introduced a concept of file tagging, which dynamically groups files with the same label into virtual directories. The file labels can be added, removed, renamed, and deleted through the admin interface or via REST API. The files in virtual directories are exposed through all protocols supported by dCache. This contribution will describe the details of the implementation for file tagging in dCache and present our future development plans on automatic metadata extractions, a feature that will significantly simplify data management. Additionally, we are exploring the future use of virtual directories as a way to translate scientific data catalogs into filesystem views for direct data analysis.

• Research Organization: Natl. Supercomputing Ctr., Tianjin; DESY; Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
• Sponsoring Organization: US Department of Energy
• DOE Contract Number: 89243024CSC000002
• OSTI ID: 3009885
• Report Number(s): FERMILAB-CONF-25-0940-CSAID; oai:inspirehep.net:3065643
• Resource Type: Conference paper
• Conference Information: Journal Name: EPJ Web Conf.
• Country of Publication: United States
• Language: English