skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond

Abstract

With this contribution we present some recent developments made to Rucio, the data management system of the High-Energy Physics Experiment ATLAS. Already managing 300 Petabytes of both official and user data, Rucio has seen incremental improvements throughout LHC Run-2, and is currently laying the groundwork for HEP computing in the HL-LHC era. The focus of this contribution are (a) the automations that have been put in place such as data rebalancing or dynamic replication of user data, as well as their supporting infrastructures such as real-time networking metrics or transfer time predictions; (b) the flexible approach towards inclusion of heterogeneous storage systems, including object stores, while unifying the potential access paths using generally available tools and protocols; (c) machine learning approaches to help with transfer throughput estimation; and (d) the adoption of Rucio for two other experiments, AMS and Xenon1t. We conclude by presenting operational numbers and figures to quantify these improvements, and extrapolate the necessary changes and developments for future LHC runs.

Authors:
; ; ; ; ;
Publication Date:
Research Org.:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
Sponsoring Org.:
USDOE Office of Science (SC)
Contributing Org.:
ATLAS collaboration
OSTI Identifier:
1544176
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Journal of Physics. Conference Series
Additional Journal Information:
Journal Volume: 1085; Journal ID: ISSN 1742-6588
Publisher:
IOP Publishing
Country of Publication:
United States
Language:
English
Subject:
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats

Barisits, M., Beermann, T., Garonne, V., Javurek, T., Lassnig, M., and Serfon, C. The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond. United States: N. p., 2018. Web. doi:10.1088/1742-6596/1085/3/032030.
Barisits, M., Beermann, T., Garonne, V., Javurek, T., Lassnig, M., & Serfon, C. The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond. United States. https://doi.org/10.1088/1742-6596/1085/3/032030
Barisits, M., Beermann, T., Garonne, V., Javurek, T., Lassnig, M., and Serfon, C. 2018. "The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond". United States. https://doi.org/10.1088/1742-6596/1085/3/032030. https://www.osti.gov/servlets/purl/1544176.
@article{osti_1544176,
title = {The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond},
author = {Barisits, M. and Beermann, T. and Garonne, V. and Javurek, T. and Lassnig, M. and Serfon, C.},
abstractNote = {With this contribution we present some recent developments made to Rucio, the data management system of the High-Energy Physics Experiment ATLAS. Already managing 300 Petabytes of both official and user data, Rucio has seen incremental improvements throughout LHC Run-2, and is currently laying the groundwork for HEP computing in the HL-LHC era. The focus of this contribution are (a) the automations that have been put in place such as data rebalancing or dynamic replication of user data, as well as their supporting infrastructures such as real-time networking metrics or transfer time predictions; (b) the flexible approach towards inclusion of heterogeneous storage systems, including object stores, while unifying the potential access paths using generally available tools and protocols; (c) machine learning approaches to help with transfer throughput estimation; and (d) the adoption of Rucio for two other experiments, AMS and Xenon1t. We conclude by presenting operational numbers and figures to quantify these improvements, and extrapolate the necessary changes and developments for future LHC runs.},
doi = {10.1088/1742-6596/1085/3/032030},
url = {https://www.osti.gov/biblio/1544176}, journal = {Journal of Physics. Conference Series},
issn = {1742-6588},
number = ,
volume = 1085,
place = {United States},
year = {Sat Sep 01 00:00:00 EDT 2018},
month = {Sat Sep 01 00:00:00 EDT 2018}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 5 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Long Short-Term Memory
journal, November 1997


Automatic rebalancing of data in ATLAS distributed data management
journal, October 2017


C3PO - A Dynamic Data Placement Agent for ATLAS Distributed Data Management
journal, October 2017