Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory

Conference · Wed Oct 01 00:00:00 EDT 2025 · EPJ Web Conf.

DOI:https://doi.org/10.1051/epjconf/202533701234· OSTI ID:3003660

Yang, Wei ^[1]; Adelman-McCarthy, Jennifer ^[2]; Daues, Greg ^[3]; Dubois, Richard ^[1]; Guan, Wen ^[4]; Guo, Yuyi ^[2]; Gower, Michelle ^[3]; Hernandez, Fabio ^[5]; Jenness, Tim ^[6]; Karavakis, Edward ^[4]; Lim, Kian-Tat ^[1]; Love, Peter ^[7]; Noble, Timothy ^[8]; Pietrowicz, Stephen ^[3]; White, Brandon ^[2]; Yang, Zhaoyu ^[4]; Yanny, Brian ^[2]

---

1. SLAC
2. Fermilab
3. NCSA, Urbana
4. Brookhaven Natl. Lab.
5. CC, Villeurbanne
6. Rubin Observ.
7. Lancaster U.
8. Rutherford Appleton Laboratory

The Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST) Camera is scheduled to start taking data in the summer of 2025. The Data Release Production will run the LSST Science Pipe software at data facilities in the US, France and the UK. The LSST Science Pipeline consists of complex directed acyclic graphs (DAGs) of tasks. Rubin will use the Production and Distributed Analysis (PanDA) workflow and workload management system to orchestrate this complex workflow and the distribution of workloads to the data facilities. When run end-to-end by a team of data production staff, this processing (the Science Pipelines, distributed by the workflow and workload management system) is referred to as a 'campaign'. This paper describes the central services and data facility specific services that support this multi-site data process model, including the service deployment infrastructure, the workload and workflow system, the Campaign Management tools, and connection to Rubin Data Management. This paper will also mention the experience of processing the Rubin Commissioning Camera data. All these are part of the effort to scale up the processing capabilities for the expected very large data volume from the LSST Camera.

• Research Organization: SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States); Brookhaven National Laboratory (BNL), Upton, NY (United States); NCSA, Urbana; Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States); Lancaster U.; Rubin Observ.; Rutherford Appleton Laboratory; CC, Villeurbanne
• Sponsoring Organization: US Department of Energy
• DOE Contract Number: 89243024CSC000002
• OSTI ID: 3003660
• Report Number(s): FERMILAB-CONF-25-0809-CSAID; oai:inspirehep.net:3065468
• Journal Information: EPJ Web Conf., Journal Name: EPJ Web Conf.
• Country of Publication: United States
• Language: English