Extending Rucio with modern cloud storage support

Journal Article · Mon May 06 00:00:00 UTC 2024 · EPJ Web of Conferences (Online)

DOI: https://doi.org/10.1051/epjconf/202429501030 · OSTI ID:2429541

Barisits, Martin ^[1]; Barnsley, Robert ^[2]; Barreiro Megino, Fernando Harald ^[3]; Elmsheuser, Johannes ^[4]; Lassnig, Mario ^[1]; Patrascoiu, Mihai ^[1]; Perry, James ^[5]; Serfon, Cedric ^[4]; Vendrell Moya, Alba ^[1]; Wegner, Tobias ^[1]

---

1. European Organization for Nuclear Research (CERN), Geneva (Switzerland)
2. Square Kilometre Array Observatory (SKAO), Cheshire (United Kingdom)
3. Univ. of Texas, Austin, TX (United States)
4. Brookhaven National Laboratory (BNL), Upton, NY (United States)
5. Univ. of Edinburgh (United Kingdom)

Rucio is a software framework designed to facilitate scientific collaborations in efficiently organising, managing, and accessing extensive volumes of data through customizable policies. The framework enables data distribution across globally distributed locations and heterogeneous data centres, integrating various storage and network technologies into a unified federated entity. Rucio offers advanced features like distributed data recovery and adaptive replication, and it exhibits high scalability, modularity, and extensibility. Originally developed to meet the requirements of the high-energy physics experiment ATLAS, Rucio has been continuously expanded to support LHC experiments and diverse scientific communities. Recent R&D projects within these communities have evaluated the integration of both private and commercially-provided cloud storage systems, leading to the development of additional functionalities for seamless integration within Rucio. Furthermore, the underlying systems, FTS and GFAL/Davix, have been extended to cater to specific use cases. This contribution focuses on the technical aspects of this work, particularly the challenges encountered in building a generic interface for self-hosted cloud storage, such as MinIO or CEPH S3 Gateway, and established providers like Google Cloud Storage and Amazon Simple Storage Service. Additionally, the integration of decentralised clouds like SEAL is explored. Key aspects, including authentication and authorisation, direct and remote access, throughput and cost estimation, are highlighted, along with shared experiences in daily operations.

• Research Organization: Brookhaven National Laboratory (BNL), Upton, NY (United States)
• Sponsoring Organization: USDOE Office of Science (SC), High Energy Physics (HEP)
• Grant/Contract Number: SC0012704
• OSTI ID: 2429541
• Report Number(s): BNL--225905-2024-JAAM
• Journal Information: EPJ Web of Conferences (Online), Journal Name: EPJ Web of Conferences (Online) Vol. 295; ISSN 2100-014X
• Publisher: EDP SciencesCopyright Statement
• Country of Publication: United States
• Language: English