Accelerating science: The usage of commercial clouds in ATLAS Distributed Computing

Journal Article · 06 May 2024 · EPJ Web of Conferences (Online)

DOI: https://doi.org/10.1051/epjconf/202429507002 · OSTI ID:2429542

Barreiro Megino, Fernando ^[1]; Borodin, Mikhail ^[2]; De, Kaushik ^[1]; Elmsheuser, Johannes ^[3]; Di Girolamo, Alessandro ^[4]; Hartmann, Nikolai ^[5]; Heinrich, Lukas ^[6]; Klimentov, Alexei ^[3]; Lassnig, Mario ^[4]; Lin, FaHui ^[1]; Maeno, Tadashi ^[3]; Marshall, Zachary ^[7]; Merino, Gonzalo ^[8]; Nilsson, Paul ^[3]; Sandesara, Jay ^[9]; Serfon, Cedric ^[3]; South, David ^[10]; Singh, Harinder ^[11]

---

1. Univ. of Texas, Arlington, TX (United States)
2. Univ. of Iowa, Iowa City, IA (United States)
3. Brookhaven National Laboratory (BNL), Upton, NY (United States)
4. European Organization for Nuclear Research (CERN), Geneva (Switzerland)
5. Ludwig Maximilian Univ. of Munich, Munich (Germany)
6. Max Planck Institute for Physics, Munich (Germany)
7. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
8. Port d’Informació Científica, Barcelona (Spain)
9. Univ. of Massachusetts, Amherst, MA (United States)
10. Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)
11. California State University, Fresno, CA (United States)

The ATLAS experiment at CERN is one of the largest scientific machines built to date and will have ever growing computing needs as the Large Hadron Collider collects an increasingly larger volume of data over the next 20 years. ATLAS is conducting R&D projects on Amazon Web Services and Google Cloud as complementary resources for distributed computing, focusing on some of the key features of commercial clouds: lightweight operation, elasticity and availability of multiple chip architectures. The proof of concept phases have concluded with the cloud-native, vendoragnostic integration with the experiment’s data and workload management frameworks. Google Cloud has been used to evaluate elastic batch computing, ramping up ephemeral clusters of up to O(100k) cores to process tasks requiring quick turnaround. Amazon Web Services has been exploited for the successful physics validation of the Athena simulation software on ARM processors. We have also set up an interactive facility for physics analysis allowing endusers to spin up private, on-demand clusters for parallel computing with up to 4 000 cores, or run GPU enabled notebooks and jobs for machine learning applications. The success of the proof of concept phases has led to the extension of the Google Cloud project, where ATLAS will study the total cost of ownership of a production cloud site during 15 months with 10k cores on average, fully integrated with distributed grid computing resources and continue the R&D projects.

• Research Organization: Brookhaven National Laboratory (BNL), Upton, NY (United States)
• Sponsoring Organization: USDOE Office of Science (SC), High Energy Physics (HEP)
• Grant/Contract Number: SC0012704
• OSTI ID: 2429542
• Report Number(s): BNL--225906-2024-JAAM
• Journal Information: EPJ Web of Conferences (Online), Journal Name: EPJ Web of Conferences (Online) Vol. 295; ISSN 2100-014X
• Publisher: EDP SciencesCopyright Statement
• Country of Publication: United States
• Language: English