Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

PanDA: Production and Distributed Analysis System

Journal Article · · Computing and Software for Big Science
 [1];  [2];  [2];  [2];  [1];  [1];  [1];  [3];  [2];  [1];  [1];  [1];  [1]
  1. Brookhaven National Laboratory (BNL), Upton, NY (United States)
  2. University of Texas at Arlington, TX (United States)
  3. University of Pittsburgh, PA (United States)
+ Show Author Affiliations
The Production and Distributed Analysis (PanDA) system is a data-driven workload management system engineered to operate at the LHC data processing scale. The PanDA system provides a solution for scientific experiments to fully leverage their distributed heterogeneous resources, showcasing scalability, usability, flexibility, and robustness. The system has successfully proven itself through nearly two decades of steady operation in the ATLAS experiment, addressing the intricate requirements such as diverse resources distributed worldwide at about 200 sites, thousands of scientists analyzing the data remotely, the volume of processed data beyond the exabyte scale, dozens of scientific applications to support, and data processing over several billion hours of computing usage per year. PanDA’s flexibility and scalability make it suitable for the High Energy Physics community and wider science domains at the Exascale. Beyond High Energy Physics, PanDA’s relevance extends to other big data sciences, as evidenced by its adoption in the Vera C. Rubin Observatory and the sPHENIX experiment. As the significance of advanced workflows continues to grow, PanDA has transformed into a comprehensive ecosystem, effectively tackling challenges associated with emerging workflows and evolving computing technologies. The paper discusses PanDA’s prominent role in the scientific landscape, detailing its architecture, functionality, deployment strategies, project management approaches, results, and evolution into an ecosystem.
Research Organization:
Brookhaven National Laboratory (BNL), Upton, NY (United States)
Sponsoring Organization:
USDOE Office of Science (SC), High Energy Physics (HEP)
Grant/Contract Number:
SC0012704
OSTI ID:
2283314
Report Number(s):
BNL--225244-2024-JAAM
Journal Information:
Computing and Software for Big Science, Journal Name: Computing and Software for Big Science Journal Issue: 1 Vol. 8; ISSN 2510-2036
Publisher:
SpringerCopyright Statement
Country of Publication:
United States
Language:
English

References (29)

SLURM: Simple Linux Utility for Resource Management book January 2003
Design concepts for the Cherenkov Telescope Array CTA: an advanced facility for ground-based high-energy gamma-ray astronomy journal November 2011
Prospects for observing and localizing gravitational-wave transients with Advanced LIGO, Advanced Virgo and KAGRA journal September 2020
Rucio: Scientific Data Management journal August 2019
RHIC project overview
  • Harrison, M.; Ludlam, T.; Ozaki, S.
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 499, Issue 2-3 https://doi.org/10.1016/S0168-9002(02)01937-X
journal March 2003
Advanced Resource Connector middleware for lightweight computational Grids journal February 2007
Principles, technologies, and time: The translational journey of the HTCondor-CE journal May 2021
Grid production with the ATLAS Event Service journal January 2019
CRIC: Computing Resource Information Catalogue as a unified topology system for a large scale, heterogeneous and dynamic computing infrastructure journal January 2020
The DIRAC interware: current, upcoming and planned capabilities and technologies journal January 2020
ART ATLAS Release Tester using the Grid journal January 2020
Seamless integration of commercial Clouds with ATLAS Distributed Computing journal January 2021
The ATLAS Data Carousel Project Status journal January 2021
An intelligent Data Delivery Service for and beyond the ATLAS experiment journal January 2021
glideinWMS—a generic pilot-based workload management system journal July 2008
The GridSite Web/Grid security system journal April 2010
JAliEn – A new interface between the AliEn jobs and the central services journal June 2014
ARC Control Tower: A flexible generic distributed job management framework journal December 2015
The INDIGO-Datacloud Authentication and Authorization Infrastructure journal October 2017
Unified Monitoring Architecture for IT and Grid Services journal October 2017
LHC Machine journal August 2008
The ALICE experiment at the CERN LHC journal August 2008
The ATLAS Experiment at the CERN Large Hadron Collider journal August 2008
The CMS experiment at the CERN LHC journal August 2008
The LHCb Detector at the LHC journal August 2008
The Evolution of the Pegasus Workflow Management Software journal July 2019
funcX: A Federated Function Serving Fabric for Science
  • Chard, Ryan; Babuji, Yadu; Li, Zhuozhao
  • HPDC '20: The 29th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3369583.3392683
conference June 2020
Nevergrad journal April 2021
LSST: From Science Drivers to Reference Design and Anticipated Data Products journal March 2019

Similar Records

Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory
Conference · Sun Dec 31 23:00:00 EST 2023 · EPJ Web Conf. · OSTI ID:2468771
Integrating the PanDA Workload Management System with the Vera C. Rubin Observatory
Journal Article · Sun May 05 20:00:00 EDT 2024 · EPJ Web of Conferences (Online) · OSTI ID:2281342
Preparation of the Multi-Site Data Processing at the Vera C. Rubin Observatory
Conference · Wed Oct 01 00:00:00 EDT 2025 · EPJ Web Conf. · OSTI ID:3003660

Related Subjects

97 MATHEMATICS AND COMPUTING
Distributed computing
Exascale
Heterogeneous computing
Workload management