dCache: The Storage System of Choice for Data-Intensive Applications

Mkrtchyan, Tigran; Green, Christopher; Hoyos, Karen; Litvintsev, Dmitry; Millar, Paul; Morschel, Lea; Rossi, Albert; Sahakyan, Marina; Starr, Darren

doi:10.1007/s41781-025-00152-5

dCache: The Storage System of Choice for Data-Intensive Applications

Journal Article · Tue Nov 25 00:00:00 EST 2025 · Computing and Software for Big Science

DOI:https://doi.org/10.1007/s41781-025-00152-5· OSTI ID:3006987

^[1]; ^[2]; Hoyos, Karen ^[1]; ^[2]; ^[1]; ^[1]; Rossi, Albert ^[2]; Sahakyan, Marina ^[1]; Starr, Darren ^[3]

Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)
Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
Univ. of Oslo, Blindern (Norway); NordForsk, Oslo (Norway)

The ever-increasing volumes of data produced by modern scientific facilities like EuXFEL and LHC put significant stress on data management infrastructure operated by laboratories and research centers. The challenges to be addressed span the entire data life cycle, from ingest and efficient data analysis to long-term preservation, typically involving large tape libraries. dCache, a storage system developed in collaboration between the Deutsches Elektronen-Synchrotron (DESY), Fermi National Accelerator Laboratory, and Nordic e-Infrastructure Collaboration (NeIC), is designed to manage a large number of disk servers and to facilitate transparent data migration to and from archival storage. Its multifaceted approach offers a unified method to support a variety of scientific use cases with the same storage infrastructure, including high-throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters, and long-term data preservation on tertiary storage. Initially developed for high energy physics (HEP) experiments, dCache is now used by various scientific communities, including astrophysics, biomedical research, and life sciences, each having specific requirements. This paper presents architecture, deployment strategies, performance and scalability enhancements, and recent advancements in dCache addressing the needs of scientific communities. Finally, we touch on the development and release process, ensuring the software’s high quality.

View Accepted Manuscript (DOE)

Research Organization:: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), High Energy Physics (HEP)

Grant/Contract Number:: 89243024CSC000002

OSTI ID:: 3006987

Report Number(s):: FERMILAB-PUB--25-0857-CSAID; oai:inspirehep.net:3089579

Journal Information:: Computing and Software for Big Science, Journal Name: Computing and Software for Big Science Journal Issue: 1 Vol. 9; ISSN 2510-2036; ISSN 2510-2044

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (27)

CERN Tape Archive — from development to production deployment Davis, Michael C.; Bahyl, Vladímir; Cancio, Germán EPJ Web of Conferences, Vol. 214 https://doi.org/10.1051/epjconf/201921404015	journal	January 2019
Architecture and prototype of a WLCG data lake for HL-LHC Bird, Ian; Campana, Simone; Girone, Maria EPJ Web of Conferences, Vol. 214 https://doi.org/10.1051/epjconf/201921404024	journal	January 2019
Beyond X.509: token-based authentication and authorization for HEP Ceccanti, Andrea; Vianello, Enrico; Caberletti, Marco EPJ Web of Conferences, Vol. 214 https://doi.org/10.1051/epjconf/201921409002	journal	January 2019
The GridKa Tape Storage: various performance test results and current improvements Musheghyan, Haykuhi; Petzold, Andreas; Heiss, Andreas EPJ Web of Conferences, Vol. 245 https://doi.org/10.1051/epjconf/202024504026	journal	January 2020
ATLAS Data Carousel Barisits, Martin; Borodin, Mikhail; Di Girolamo, Alessandro EPJ Web of Conferences, Vol. 245 https://doi.org/10.1051/epjconf/202024504035	journal	January 2020
dCache – Efficient Message Encoding For Inter-Service Communication in dCache: Evaluation of Existing Serialization Protocols as a Replacement for Java Object Serialization Morschel, Lea; Adeyemi, Olufemi; Garonne, Vincent EPJ Web of Conferences, Vol. 245 https://doi.org/10.1051/epjconf/202024505017	journal	January 2020
dCache: Inter-disciplinary storage system Mkrtchyan, Tigran; Chitrapu, Krishnaveni; Garonne, Vincent EPJ Web of Conferences, Vol. 251 https://doi.org/10.1051/epjconf/202125102010	journal	January 2021
Improving Performance of Tape Restore Request Scheduling in the Storage System dCache Morschel, Lea; Chitrapu, Krishnaveni; Garonne, Vincent EPJ Web of Conferences, Vol. 251 https://doi.org/10.1051/epjconf/202125102016	journal	January 2021
An HTTP REST API for Tape-backed Storage Afonso, João; Caffy, Cédric; Patrascoiu, Mihai EPJ Web of Conferences, Vol. 295 https://doi.org/10.1051/epjconf/202429501008	journal	January 2024
dCache integration with CERN Tape Archive Mkrtchyan, Tigran; Chodak, Jacek; Karimi, Mwai EPJ Web of Conferences, Vol. 295 https://doi.org/10.1051/epjconf/202429501016	journal	January 2024
The CERN Tape Archive Beyond CERN An Open Source Data Archival System for HEP Davis, Michael; Afonso, João; Bachmann, Richard EPJ Web of Conferences, Vol. 295 https://doi.org/10.1051/epjconf/202429501048	journal	January 2024
Memorandum on design-oriented information systems research Österle, Hubert; Becker, Jörg; Frank, Ulrich European Journal of Information Systems, Vol. 20, Issue 1 https://doi.org/10.1057/ejis.2010.55	journal	January 2011
A distributed storage system with dCache Behrmann, G.; Fuhrmann, P.; Grønager, M. Journal of Physics: Conference Series, Vol. 119, Issue 6 https://doi.org/10.1088/1742-6596/119/6/062014	journal	July 2008
LHC Data Analysis Using NFSv4.1 (pNFS): A Detailed Evaluation Elmsheuser, Johannes; Fuhrmann, Patrick; Kemp, Yves Journal of Physics: Conference Series, Vol. 331, Issue 5 https://doi.org/10.1088/1742-6596/331/5/052010	journal	December 2011
dCache, agile adoption of storage technology Millar, A. P.; Baranova, T.; Behrmann, G. Journal of Physics: Conference Series, Vol. 396, Issue 3 https://doi.org/10.1088/1742-6596/396/3/032077	journal	December 2012
Experience with HEP analysis on mounted filesystems. Fuhrmann, Patrick; Gasthuber, Martin; Kemp, Yves Journal of Physics: Conference Series, Vol. 396, Issue 4 https://doi.org/10.1088/1742-6596/396/4/042020	journal	December 2012
Transparent handling of small files with dCache to optimize tape access Schwank, Karsten; Krücker, Dirk; Fuhrmann, Patrick Journal of Physics: Conference Series, Vol. 664, Issue 4 https://doi.org/10.1088/1742-6596/664/4/042048	journal	December 2015
Data Resilience in the dCache Storage System Rossi, A. L.; Adeyemi, F.; Ashish, A. Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/6/062024	journal	October 2017
EOS developments Sindrilaru, Elvin A.; Peters, Andreas J.; Adde, Geoffray M. Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/6/062032	journal	October 2017
Exporting Storage Systems in a Scalable Manner with pNFS Hildebrand, D.; Honeyman, P. 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST'05) https://doi.org/10.1109/MSST.2005.14	conference	January 2005
A cost-effective, high-bandwidth storage architecture Gibson, Garth A.; Zelenka, Jim; Nagle, David F. ACM SIGOPS Operating Systems Review, Vol. 32, Issue 5 https://doi.org/10.1145/384265.291029	journal	December 1998
Distributed file systems: concepts and examples Levy, Eliezer; Silberschatz, Abraham ACM Computing Surveys, Vol. 22, Issue 4 https://doi.org/10.1145/98163.98169	journal	December 1990
Data Management Infrastructure for European XFEL Malka, Janusz; Aplin, Steve; Boukhelef, Djelloul JACoW Publishing, Geneva, Switzerland https://doi.org/10.18429/JACoW-ICALEPCS2023-WE1BCO02	conferencepaper	January 2024
Improving Tape Restore Request Scheduling in the Storage System dCache Morschel, Lea Deutsches Elektronen-Synchrotron, DESY, Hamburg https://doi.org/10.3204/PUBDB-2020-01394	text	January 2020
Chimera - a new, fast, extensible and Grid enabled namespace service Gasthuber, M.; Fuhrmann, P.; Mkrtchyan, T. CERN https://doi.org/10.5170/CERN-2005-002.1180	null	January 2005
The cathedral and the bazaar Raymond, Eric S. First Monday, Vol. 3, Issue 2 https://doi.org/10.5210/FM.V3I2.578	journal	March 1998
root-project/root: v6.18/02 Brun, Rene; Rademakers, Fons; Canal, Philippe https://doi.org/10.5281/zenodo.3895860	software	August 2019

Similar Records

dCache: Inter-disciplinary storage system

Conference · Thu Dec 31 23:00:00 EST 2020 · EPJ Web Conf. · OSTI ID:1854764

dCache project status and update

Conference · Tue Dec 31 23:00:00 EST 2024 · EPJ Web Conf. · OSTI ID:3009879

dCache - Keeping up With the Evolution of Science

Conference · Tue Dec 31 23:00:00 EST 2019 · EPJ Web Conf. · OSTI ID:1842718

Related Subjects

DCache
Data access
HSM
Storage

dCache: The Storage System of Choice for Data-Intensive Applications

Citation Formats

References (27)

Similar Records

Related Subjects