Rucio: Scientific Data Management

Barisits, Martin; Beermann, Thomas; Berghaus, Frank; Bockelman, Brian; Bogado, Joaquin; Cameron, David; Christidis, Dimitrios; Ciangottini, Diego; Dimitrov, Gancho; Elsing, Markus; Garonne, Vincent; di Girolamo, Alessandro; Goossens, Luc; Guan, Wen; Guenther, Jaroslav; Javurek, Tomas; Kuhn, Dietmar; Lassnig, Mario; Lopez, Fernando; Magini, Nicolo; Molfetas, Angelos; Nairz, Armin; Ould-Saada, Farid; Prenner, Stefan; Serfon, Cedric; Stewart, Graeme; Vaandering, Eric; Vasileva, Petya; Vigne, Ralph; Wegner, Tobias

doi:10.1007/s41781-019-0026-3

Title: Rucio: Scientific Data Management

Journal Article · Fri Aug 09 00:00:00 EDT 2019 · Computing and Software for Big Science

DOI:https://doi.org/10.1007/s41781-019-0026-3· OSTI ID:1542972

Barisits, Martin; Beermann, Thomas; Berghaus, Frank; Bockelman, Brian; Bogado, Joaquin; Cameron, David; Christidis, Dimitrios; Ciangottini, Diego; Dimitrov, Gancho; Elsing, Markus; Garonne, Vincent; di Girolamo, Alessandro; Goossens, Luc; Guan, Wen; Guenther, Jaroslav; Javurek, Tomas; Kuhn, Dietmar;

; Lopez, Fernando; Magini, Nicolo more »

Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. The data can be distributed across heterogeneous data centers at widely distributed locations. Rucio was originally developed to meet the requirements of the high-energy physics experiment ATLAS, and now is continuously extended to support the LHC experiments and other diverse scientific communities. In this article, we detail the fundamental concepts of Rucio, describe the architecture along with implementation details, and report operational experience from production usage.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), High Energy Physics (HEP)

Contributing Organization:: CERN Collaboration

Grant/Contract Number:: AC02-07CH11359

OSTI ID:: 1542972

Report Number(s):: arXiv:1902.09857; ATL-COM-SOFT-2018-100; FERMILAB-PUB-19-095-CD; 2510-2044; oai:inspirehep.net:1722117

Journal Information:: Computing and Software for Big Science, Vol. 3, Issue 1; ISSN 2510-2036

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (31)

Machine learning of network metrics in ATLAS Distributed Data Management Lassnig, Mario; Toler, Wesley; Vamosi, Ralf Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/6/062009	journal	October 2017
Testing as a Service with HammerCloud Llamas, Ramón Medrano; Barrand, Quentin; Elmsheuser, Johannes Journal of Physics: Conference Series, Vol. 513, Issue 6 https://doi.org/10.1088/1742-6596/513/6/062031	journal	June 2014
ATLAS Replica Management in Rucio: Replication Rules and Subscriptions Barisits, M.; Serfon, C.; Garonne, V. Journal of Physics: Conference Series, Vol. 513, Issue 4 https://doi.org/10.1088/1742-6596/513/4/042003	journal	June 2014
The GridSite Web/Grid security system McNab, Andrew; Li, Yibiao Journal of Physics: Conference Series, Vol. 219, Issue 6 https://doi.org/10.1088/1742-6596/219/6/062058	journal	April 2010
FTS3: New Data Movement Service For WLCG Ayllon, A. A.; Salichos, M.; Simon, M. K. Journal of Physics: Conference Series, Vol. 513, Issue 3 https://doi.org/10.1088/1742-6596/513/3/032081	journal	June 2014
LHCOPN and LHCONE: Status and Future Evolution Martelli, E.; Stancu, S. Journal of Physics: Conference Series, Vol. 664, Issue 5 https://doi.org/10.1088/1742-6596/664/5/052025	journal	December 2015
Overview of ATLAS PanDA Workload Management Maeno, T.; De, K.; Wenaus, T. Journal of Physics: Conference Series, Vol. 331, Issue 7 https://doi.org/10.1088/1742-6596/331/7/072024	journal	December 2011
Multilevel Workflow System in the ATLAS Experiment Borodin, M.; De, K.; Navarro, J. Garcia Journal of Physics: Conference Series, Vol. 608 https://doi.org/10.1088/1742-6596/608/1/012015	journal	May 2015
EOS as the present and future solution for data storage at CERN Peters, Aj; Sindrilaru, Ea; Adde, G. Journal of Physics: Conference Series, Vol. 664, Issue 4 https://doi.org/10.1088/1742-6596/664/4/042042	journal	December 2015
Managing ATLAS data on a petabyte-scale with DQ2 Branco, M.; Cameron, D.; Gaidioz, B. Journal of Physics: Conference Series, Vol. 119, Issue 6 https://doi.org/10.1088/1742-6596/119/6/062017	journal	July 2008
The ATLAS Data Acquisition System in LHC Run 2 Panduro Vazquez, William Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/3/032017	journal	October 2017
The new CERN tape software - getting ready for total performance Cano, E.; Murray, S.; Kruse, D. F. Journal of Physics: Conference Series, Vol. 664, Issue 4 https://doi.org/10.1088/1742-6596/664/4/042007	journal	December 2015
DIRAC in Large Particle Physics Experiments Stagni, F.; Tsaregorodtsev, A.; Arrabito, L. Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/9/092020	journal	October 2017
ROOT — An object oriented data analysis framework Brun, Rene; Rademakers, Fons Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 389, Issue 1-2 https://doi.org/10.1016/S0168-9002(97)00048-X	journal	April 1997
AthenaMT: upgrading the ATLAS software framework for the many-core world with multi-threading Leggett, Charles; Baines, John; Bold, Tomasz Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/4/042009	journal	October 2017
AGIS: The ATLAS Grid Information System Anisenkov, Alexey; Belov, Sergey; Di Girolamo, Alessandro Journal of Physics: Conference Series, Vol. 396, Issue 3 https://doi.org/10.1088/1742-6596/396/3/032006	journal	December 2012
Caching Servers for ATLAS Gardner, R. W.; Hanushevsky, A.; Vukotic, I. Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/6/062017	journal	October 2017
Dynamic federation of grid and cloud storage Furano, Fabrizio; Keeble, Oliver; Field, Laurence Physics of Particles and Nuclei Letters, Vol. 13, Issue 5 https://doi.org/10.1134/S1547477116050186	journal	September 2016
A Survey of Software-Defined Networking: Past, Present, and Future of Programmable Networks Nunes, Bruno Astuto A.; Mendonca, Marc; Nguyen, Xuan-Nam IEEE Communications Surveys & Tutorials, Vol. 16, Issue 3 https://doi.org/10.1109/SURV.2014.012214.00180	journal	October 2014
Search for TeV-scale gravity signatures in high-mass final states with leptons and jets with the ATLAS detector at $\sqrt{s} = 13 TeV$ Aaboud, M.; Aad, G.; Abbott, B. Physics Letters B, Vol. 760 https://doi.org/10.1016/j.physletb.2016.07.030	journal	September 2016
DPM evolution: a disk operations management engine for DPM Manzi, A.; Furano, F.; Keeble, O. Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/6/062011	journal	October 2017
Storage resource manager version 2.2: design, implementation, and testing experience Donno, F.; Abadie, L.; Badino, P. Journal of Physics: Conference Series, Vol. 119, Issue 6 https://doi.org/10.1088/1742-6596/119/6/062028	journal	July 2008
LHC Machine Evans, Lyndon; Bryant, Philip Journal of Instrumentation, Vol. 3, Issue 08 https://doi.org/10.1088/1748-0221/3/08/S08001	journal	August 2008
The ATLAS Tier-0: Overview and operational experience Elsing, Markus; Goossens, Luc; Nairz, Armin Journal of Physics: Conference Series, Vol. 219, Issue 7 https://doi.org/10.1088/1742-6596/219/7/072011	journal	April 2010
The ATLAS Experiment at the CERN Large Hadron Collider Collaboration, The ATLAS; Aad, G.; Abat, E. Journal of Instrumentation, Vol. 3, Issue 08 https://doi.org/10.1088/1748-0221/3/08/S08003	journal	August 2008
The CMS experiment at the CERN LHC Collaboration, The CMS; Chatrchyan, S.; Hmayakyan, G. Journal of Instrumentation, Vol. 3, Issue 08 https://doi.org/10.1088/1748-0221/3/08/S08004	journal	August 2008
Unified Monitoring Architecture for IT and Grid Services Aimar, A.; Corman, A. Aguado; Andrade, P. Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/9/092033	journal	October 2017
ATLAS Distributed Computing Experience and Performance During the LHC Run-2 Filipčič, A. Journal of Physics: Conference Series, Vol. 898 https://doi.org/10.1088/1742-6596/898/5/052015	journal	October 2017
Distributed computing in practice: the Condor experience Thain, Douglas; Tannenbaum, Todd; Livny, Miron Concurrency and Computation: Practice and Experience, Vol. 17, Issue 2-4, p. 323-356 https://doi.org/10.1002/cpe.938	journal	January 2005
Globus Toolkit Version 4: Software for Service-Oriented Systems Foster, Ian Journal of Computer Science and Technology, Vol. 21, Issue 4 https://doi.org/10.1007/s11390-006-0513-y	journal	July 2006
AGIS: The ATLAS Grid Information System Anisenkov, A.; Girolamo, A. Di; Klimentov, A. Journal of Physics: Conference Series, Vol. 513, Issue 3 https://doi.org/10.1088/1742-6596/513/3/032001	journal	June 2014