Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory

Journal Article · · EPJ Web of Conferences (Online)
 [1];  [1];  [1];  [1];  [1];  [2];  [3];  [3];  [4];  [5];  [6];  [1];  [7];  [5];  [8];  [1];  [1];  [4];  [1]
  1. Brookhaven National Laboratory (BNL), Upton, NY (United States)
  2. Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
  3. University of Texas at Arlington, TX (United States)
  4. SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)
  5. National Center for Supercomputing Applications, Urbana, IL (United States)
  6. Vera C. Rubin Observatory, Tucson, AZ (United States)
  7. University of Pittsburgh, PA (United States)
  8. 3University of Texas at Arlington, TX (United States)
+ Show Author Affiliations

The Vera C. Rubin Observatory will produce an unprecedented astronomical data set for studies of the deep and dynamic universe. Its Legacy Survey of Space and Time (LSST) will image the entire southern sky every three to four days and produce tens of petabytes of raw image data and associated calibration data over the course of the experiment’s run. More than 20 terabytes of data must be stored every night, and annual campaigns to reprocess the entire dataset since the beginning of the survey will be conducted over ten years. The Production and Distributed Analysis (PanDA) system was evaluated by the Rubin Observatory Data Management team and selected to serve the Observatory’s needs due to its demonstrated scalability and flexibility over the years, for its Directed Acyclic Graph (DAG) support, its support for multi-site processing, and its highly scalable complex workflows via the intelligent Data Delivery Service (iDDS). PanDA is also being evaluated for prompt processing where data must be processed within 60 seconds after image capture. This paper will briefly describe the Rubin Data Management system and its Data Facilities (DFs). Finally, it will describe in depth the work performed in order to integrate the PanDA system with the Rubin Observatory to be able to run the Rubin Science Pipelines using PanDA.

Research Organization:
Brookhaven National Laboratory (BNL), Upton, NY (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Nuclear Physics (NP); National Science Foundation (NSF)
Grant/Contract Number:
SC0012704; AC02-76SF00515
OSTI ID:
2281342
Report Number(s):
BNL--225137-2023-JAAM
Journal Information:
EPJ Web of Conferences (Online), Journal Name: EPJ Web of Conferences (Online) Vol. 295; ISSN 2100-014X
Publisher:
EDP SciencesCopyright Statement
Country of Publication:
United States
Language:
English

References (9)

Rucio: Scientific Data Management journal August 2019
ATLAS Data Carousel journal January 2020
Overview of the distributed image processing infrastructure to produce the Legacy Survey of Space and Time journal January 2024
BigPanDA monitoring system evolution in the ATLAS Experiment journal January 2024
Distributed Machine Learning Workflow with PanDA and iDDS in LHC ATLAS journal January 2024
Utilizing Distributed Heterogeneous Computing with PanDA in ATLAS journal January 2024
The Vera C. Rubin Observatory Data Butler and pipeline execution system conference August 2022
Design and development of the 3.2 gigapixel camera for the Large Synoptic Survey Telescope conference July 2010
LSST: From Science Drivers to Reference Design and Anticipated Data Products journal March 2019

Similar Records

Related Subjects

79 ASTRONOMY AND ASTROPHYSICS