Overview of ATLAS PanDA Workload Management

Maeno, T; De, K; Wenaus, T; Nilsson, P; A, Stewart G; Walker, R; Stradling, A; Caballero, J; Potekhin, M; Smith, D

doi:10.1088/1742-6596/331/7/072024

Title: Overview of ATLAS PanDA Workload Management

Full Record
Other Related Research

Abstract

The Production and Distributed Analysis System (PanDA) plays a key role in the ATLAS distributed computing infrastructure. All ATLAS Monte-Carlo simulation and data reprocessing jobs pass through the PanDA system. We will describe how PanDA manages job execution on the grid using dynamic resource estimation and data replication together with intelligent brokerage in order to meet the scaling and automation requirements of ATLAS distributed computing. PanDA is also the primary ATLAS system for processing user and group analysis jobs, bringing further requirements for quick, flexible adaptation to the rapidly evolving analysis use cases of the early datataking phase, in addition to the high reliability, robustness and usability needed to provide efficient and transparent utilization of the grid for analysis users. We will describe how PanDA meets ATLAS requirements, the evolution of the system in light of operational experience, how the system has performed during the first LHC data-taking phase and plans for the future.

Authors:: Maeno, T; De, K; Wenaus, T; Nilsson, P; A, Stewart G; Walker, R; Stradling, A; Caballero, J; Potekhin, M; Smith, D

Publication Date:: Sat Jan 01 00:00:00 EST 2011

Research Org.:: Brookhaven National Lab. (BNL), Upton, NY (United States)

Sponsoring Org.:: USDOE SC OFFICE OF SCIENCE (SC)

OSTI Identifier:: 1042942

Report Number(s):: BNL-97267-2012-JA
Journal ID: ISSN 1742-6596; KA1101021; TRN: US1202965

DOE Contract Number:: DE-AC02-98CH10886

Resource Type:: Journal Article

Journal Name:: Journal of Physics. Conference Series (Online)

Additional Journal Information:: Journal Volume: 331; Journal Issue: 7; Journal ID: ISSN 1742-6596

Country of Publication:: United States

Language:: English

Subject:: 11 NUCLEAR FUEL CYCLE AND FUEL MATERIALS; 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS; AUTOMATION; MANAGEMENT; PROCESSING; PRODUCTION; RELIABILITY; REPROCESSING; SIMULATION

Citation Formats


                    Maeno, T, De, K, Wenaus, T, Nilsson, P, A, Stewart G, Walker, R, Stradling, A, Caballero, J, Potekhin, M, and Smith, D. Overview of ATLAS PanDA Workload Management.  United States: N. p., 2011. 
        Web.  doi:10.1088/1742-6596/331/7/072024.

Copy to clipboard


                    Maeno, T, De, K, Wenaus, T, Nilsson, P, A, Stewart G, Walker, R, Stradling, A, Caballero, J, Potekhin, M, & Smith, D. Overview of ATLAS PanDA Workload Management.  United States.  https://doi.org/10.1088/1742-6596/331/7/072024

Copy to clipboard


                    Maeno, T, De, K, Wenaus, T, Nilsson, P, A, Stewart G, Walker, R, Stradling, A, Caballero, J, Potekhin, M, and Smith, D. 2011.  
        "Overview of ATLAS PanDA Workload Management".  United States.  https://doi.org/10.1088/1742-6596/331/7/072024.

Copy to clipboard


                    
@article{osti_1042942,

  title        = {Overview of ATLAS PanDA Workload Management},

  author       = {Maeno, T and De, K and Wenaus, T and Nilsson, P and A, Stewart G and Walker, R and Stradling, A and Caballero, J and Potekhin, M and Smith, D},

  abstractNote = {The Production and Distributed Analysis System (PanDA) plays a key role in the ATLAS distributed computing infrastructure. All ATLAS Monte-Carlo simulation and data reprocessing jobs pass through the PanDA system. We will describe how PanDA manages job execution on the grid using dynamic resource estimation and data replication together with intelligent brokerage in order to meet the scaling and automation requirements of ATLAS distributed computing. PanDA is also the primary ATLAS system for processing user and group analysis jobs, bringing further requirements for quick, flexible adaptation to the rapidly evolving analysis use cases of the early datataking phase, in addition to the high reliability, robustness and usability needed to provide efficient and transparent utilization of the grid for analysis users. We will describe how PanDA meets ATLAS requirements, the evolution of the system in light of operational experience, how the system has performed during the first LHC data-taking phase and plans for the future.},

  doi          = {10.1088/1742-6596/331/7/072024},

  url          = {https://www.osti.gov/biblio/1042942},
  journal      = {Journal of Physics. Conference Series (Online)},

  issn         = {1742-6596},

  number       = 7,

  volume       = 331,

  place        = {United States},

  year         = {Sat Jan 01 00:00:00 EST 2011},

  month        = {Sat Jan 01 00:00:00 EST 2011}

}

Copy to clipboard

Journal Article:

https://doi.org/10.1088/1742-6596/331/7/072024

Other availability

Find in Google Scholar

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

BigPanDA: PanDA Workload Management System and its Applications beyond ATLAS

Conference Svirin, Pavlo; De, Kaushik; Forti, Alessandra; ...

Modern experiments collect peta-scale volumes of data and utilize vast, geographically distributed computing infrastructure that serves thousands of scientists around the world. Requirements for rapid, near real-time data processing, fast analysis cycles and need to run massive detector simulations to support data analysis pose special premium on efficient use of available computational resources. A sophisticated Workload Management System (WMS) is needed to coordinate the distribution and processing of data and jobs in such environment. The ATLAS experiment at CERN uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 datamore »« less
Full Text Available
Integration Of PanDA Workload Management System With Supercomputers for ATLAS and Data Intensive Science

Conference De, K; Jha, S; Klimentov, A; ...

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Managementmore »« less
Full Text Available
Integration Of PanDA Workload Management System With Supercomputers for ATLAS and Data Intensive Science

Journal Article Klimentov, A.; De, K.; Jha, S.; ... - Journal of Physics. Conference Series

The.LHC, operating at CERN, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe. ATLAS, one of the largest collaborations ever assembled in the sciences, is at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, the ATLAS experiment is relying on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System for managing the workflow for all data processing on over 150 data centers. Through PanDA, ATLAS physicists see a singlemore »« less
https://doi.org/10.1088/1742-6596/762/1/012021

Full Text Available
Integration of PanDA workload management system with Titan supercomputer at OLCF

Journal Article De, K.; Klimentov, A.; Oleynik, D.; ... - Journal of Physics. Conference Series

The PanDA (Production and Distributed Analysis) workload management system (WMS) was developed to meet the scale and complexity of LHC distributed computing for the ATLAS experiment. While PanDA currently distributes jobs to more than 100,000 cores at well over 100 Grid sites, the future LHC data taking runs will require more resources than Grid computing can possibly provide. To alleviate these challenges, ATLAS is engaged in an ambitious program to expand the current computing model to include additional resources such as the opportunistic use of supercomputers. We will describe a project aimed at integration of PanDA WMS with Titan supercomputermore »« less
Cited by 3
https://doi.org/10.1088/1742-6596/664/9/092020

Full Text Available
Evolution of the ATLAS PanDA Production and Distributed Analysis System

Journal Article Maeno, T.; De, K.; Wenaus, T.; ... - Journal of Physics. Conference Series

Evolution of the ATLAS PanDA Production and Distributed Analysis System T Maeno1,5, K De2, T Wenaus1, P Nilsson2, R Walker3, A Stradling2, V Fine1, M Potekhin1, S Panitkin1 and G Compostella4 Published under licence by IOP Publishing Ltd Journal of Physics: Conference Series, Volume 396, Part 3 Article PDF References Citations Metrics 101 Total downloads Cited by 8 articles Turn on MathJax Share this article Article information Abstract The PanDA (Production and Distributed Analysis) system has been developed to meet ATLAS production and analysis requirements for a data-driven workload management system capable of operating at LHC data processing scale. PanDAmore »« less
https://doi.org/10.1088/1742-6596/396/3/032071

Similar Records