skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Evolution of the ATLAS PanDA Production and Distributed Analysis System

Abstract

Evolution of the ATLAS PanDA Production and Distributed Analysis System T Maeno1,5, K De2, T Wenaus1, P Nilsson2, R Walker3, A Stradling2, V Fine1, M Potekhin1, S Panitkin1 and G Compostella4 Published under licence by IOP Publishing Ltd Journal of Physics: Conference Series, Volume 396, Part 3 Article PDF References Citations Metrics 101 Total downloads Cited by 8 articles Turn on MathJax Share this article Article information Abstract The PanDA (Production and Distributed Analysis) system has been developed to meet ATLAS production and analysis requirements for a data-driven workload management system capable of operating at LHC data processing scale. PanDA has performed well with high reliability and robustness during the two years of LHC data-taking, while being actively evolved to meet the rapidly changing requirements for analysis use cases. We will present an overview of system evolution including automatic rebrokerage and reattempt for analysis jobs, adaptation for the CernVM File System, support for the multi-cloud model through which Tier-2 sites act as members of multiple clouds, pledged resource management and preferential brokerage, and monitoring improvements. We will also describe results from the analysis of two years of PanDA usage statistics, current issues, and plans for the future.

Authors:
; ; ; ; ; ; ; ; ;
Publication Date:
Research Org.:
Brookhaven National Lab. (BNL), Upton, NY (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1073017
Report Number(s):
BNL-100593-2013-JA
Journal ID: ISSN 1742-6588; KA1101021
DOE Contract Number:
AC02-98CH10886
Resource Type:
Journal Article
Resource Relation:
Journal Name: Journal of Physics. Conference Series; Journal Volume: 396; Journal Issue: 3
Country of Publication:
United States
Language:
English
Subject:
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats

Maeno, T., De, K., Wenaus, T., Nilsson, P., Walker, R., Stradling, A., Fine, V., Potekhin, M., Panitkin, S., and Compostella, G. Evolution of the ATLAS PanDA Production and Distributed Analysis System. United States: N. p., 2012. Web. doi:10.1088/1742-6596/396/3/032071.
Maeno, T., De, K., Wenaus, T., Nilsson, P., Walker, R., Stradling, A., Fine, V., Potekhin, M., Panitkin, S., & Compostella, G. Evolution of the ATLAS PanDA Production and Distributed Analysis System. United States. doi:10.1088/1742-6596/396/3/032071.
Maeno, T., De, K., Wenaus, T., Nilsson, P., Walker, R., Stradling, A., Fine, V., Potekhin, M., Panitkin, S., and Compostella, G. Thu . "Evolution of the ATLAS PanDA Production and Distributed Analysis System". United States. doi:10.1088/1742-6596/396/3/032071.
@article{osti_1073017,
title = {Evolution of the ATLAS PanDA Production and Distributed Analysis System},
author = {Maeno, T. and De, K. and Wenaus, T. and Nilsson, P. and Walker, R. and Stradling, A. and Fine, V. and Potekhin, M. and Panitkin, S. and Compostella, G.},
abstractNote = {Evolution of the ATLAS PanDA Production and Distributed Analysis System T Maeno1,5, K De2, T Wenaus1, P Nilsson2, R Walker3, A Stradling2, V Fine1, M Potekhin1, S Panitkin1 and G Compostella4 Published under licence by IOP Publishing Ltd Journal of Physics: Conference Series, Volume 396, Part 3 Article PDF References Citations Metrics 101 Total downloads Cited by 8 articles Turn on MathJax Share this article Article information Abstract The PanDA (Production and Distributed Analysis) system has been developed to meet ATLAS production and analysis requirements for a data-driven workload management system capable of operating at LHC data processing scale. PanDA has performed well with high reliability and robustness during the two years of LHC data-taking, while being actively evolved to meet the rapidly changing requirements for analysis use cases. We will present an overview of system evolution including automatic rebrokerage and reattempt for analysis jobs, adaptation for the CernVM File System, support for the multi-cloud model through which Tier-2 sites act as members of multiple clouds, pledged resource management and preferential brokerage, and monitoring improvements. We will also describe results from the analysis of two years of PanDA usage statistics, current issues, and plans for the future.},
doi = {10.1088/1742-6596/396/3/032071},
journal = {Journal of Physics. Conference Series},
number = 3,
volume = 396,
place = {United States},
year = {Thu Dec 13 00:00:00 EST 2012},
month = {Thu Dec 13 00:00:00 EST 2012}
}
  • The Production and Distributed Analysis System (PanDA) plays a key role in the ATLAS distributed computing infrastructure. All ATLAS Monte-Carlo simulation and data reprocessing jobs pass through the PanDA system. We will describe how PanDA manages job execution on the grid using dynamic resource estimation and data replication together with intelligent brokerage in order to meet the scaling and automation requirements of ATLAS distributed computing. PanDA is also the primary ATLAS system for processing user and group analysis jobs, bringing further requirements for quick, flexible adaptation to the rapidly evolving analysis use cases of the early datataking phase, in additionmore » to the high reliability, robustness and usability needed to provide efficient and transparent utilization of the grid for analysis users. We will describe how PanDA meets ATLAS requirements, the evolution of the system in light of operational experience, how the system has performed during the first LHC data-taking phase and plans for the future.« less
  • The PanDA (Production and Distributed Analysis) system plays a key role in the ATLAS distributed computing infrastructure. PanDA is the ATLAS workload management system for processing all Monte-Carlo (MC) simulation and data reprocessing jobs in addition to user and group analysis jobs. The PanDA Dynamic Data Placement (PD2P) system has been developed to cope with difficulties of data placement for ATLAS. We will describe the design of the new system, its performance during the past year of data taking, dramatic improvements it has brought about in the efficient use of storage and processing resources, and plans for the future.