Automated Network Services for Exascale Data Movement

Conference · 01 January 2024 · EPJ Web Conf.

DOI:https://doi.org/10.1051/epjconf/202429501009· OSTI ID:2468770

Balcas, Justas ^[1]; Newman, Harvey ^[1]; Bhat, Preeti P. ^[1]; Würthwein, Frank ^[2]; Guiang, Jonathan ^[2]; Arora, Aashay ^[2]; Davila, Diego ^[2]; Graham, John ^[2]; Hutton, Thomas ^[2]; Lehman, Tom ^[3]; Yang, Xi ^[3]; Guok, Chin ^[3]; Mason, David Alexander ^[4]; Gutsche, Oliver ^[4]; DeMar, Phil ^[4]; Huang, Chih-Hao ^[4]; Shah, Syed Asif ^[4]; Litvintsev, Dmitry ^[4]; Heath, Ryan ^[4]; Melo, Andrew Malone ^[5]

---

1. Caltech
2. UC, San Diego
3. LBL, Berkeley
4. Fermilab
5. Vanderbilt U.

The Large Hadron Collider (LHC) experiments distribute data by leveraging a diverse array of National Research and Education Networks (NRENs), where experiment data management systems treat networks as a “blackbox” resource. After the High Luminosity upgrade, the Compact Muon Solenoid (CMS) experiment alone will produce roughly 0.5 exabytes of data per year. NREN Networks are a critical part of the success of CMS and other LHC experiments. However, during data movement, NRENs are unaware of data priorities, importance, or need for quality of service, and this poses a challenge for operators to coordinate the movement of data and have predictable data flows across multi-domain networks. The overarching goal of SENSE (The Software-defined network for End-to-end Networked Science at Exascale) is to enable National Labs and universities to request and provision end-to-end intelligent network services for their application workflows leveraging SDN (Software-Defined Networking) capabilities. This work aims to allow LHC Experiments and Rucio, the data management software used by CMS Experiment, to allocate and prioritize certain data transfers over the wide area network. In this paper, we will present the current progress of the integration of SENSE, Multi-domain end-to-end SDN Orchestration with QoS (Quality of Service) capabilities, with Rucio, the data management software used by CMS Experiment.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
• Sponsoring Organization: USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
• Contributing Organization: CMS
• DOE Contract Number: AC02-07CH11359
• OSTI ID: 2468770
• Report Number(s): FERMILAB-CONF-24-0704-CMS; oai:inspirehep.net:2785604
• Conference Information: Journal Name: EPJ Web Conf. Journal Volume: 295
• Country of Publication: United States
• Language: English