The ATLAS Data Management System Rucio: Supporting LHC Run-2 and beyond
With this contribution we present some recent developments made to Rucio, the data management system of the High-Energy Physics Experiment ATLAS. Already managing 300 Petabytes of both official and user data, Rucio has seen incremental improvements throughout LHC Run-2, and is currently laying the groundwork for HEP computing in the HL-LHC era. The focus of this contribution are (a) the automations that have been put in place such as data rebalancing or dynamic replication of user data, as well as their supporting infrastructures such as real-time networking metrics or transfer time predictions; (b) the flexible approach towards inclusion of heterogeneous storage systems, including object stores, while unifying the potential access paths using generally available tools and protocols; (c) machine learning approaches to help with transfer throughput estimation; and (d) the adoption of Rucio for two other experiments, AMS and Xenon1t. We conclude by presenting operational numbers and figures to quantify these improvements, and extrapolate the necessary changes and developments for future LHC runs.
- Research Organization:
- Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- Contributing Organization:
- ATLAS collaboration
- OSTI ID:
- 1544176
- Journal Information:
- Journal of Physics. Conference Series, Vol. 1085; ISSN 1742-6588
- Publisher:
- IOP PublishingCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Web of Science
Long Short-Term Memory
|
journal | November 1997 |
Automatic rebalancing of data in ATLAS distributed data management
|
journal | October 2017 |
C3PO - A Dynamic Data Placement Agent for ATLAS Distributed Data Management
|
journal | October 2017 |
Similar Records
Transitioning CMS to Rucio Data Managment
Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON