Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing

Abstract

The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS and ALICE are the largest collaborations ever assembled in the sciences and are at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, both experiments rely on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System (WMS) for managing the workflow for all data processing on hundreds of data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. The scale is demonstrated by the following numbers: PanDA manages O(102) sites, O(105) cores, O(108) jobs per year, O(103) users, and ATLAS data volume is O(1017) bytes. In 2013 we started an ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF). Themore » project titled 'Next Generation Workload Management and Analysis System for Big Data' (BigPanDA) is funded by DOE ASCR and HEP. Extending PanDA to clouds and LCF presents new challenges in managing heterogeneity and supporting workflow. The BigPanDA project is underway to setup and tailor PanDA at the Oak Ridge Leadership Computing Facility (OLCF) and at the National Research Center "Kurchatov Institute" together with ALICE distributed computing and ORNL computing professionals. Our approach to integration of HPC platforms at the OLCF and elsewhere is to reuse, as much as possible, existing components of the PanDA system. Finally, we will present our current accomplishments with running the PanDA WMS at OLCF and other supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications.« less

Authors:
 [1];  [2];  [3];  [4];  [1];  [5];  [1];  [3];  [1];  [3];  [6];  [7];  [8];  [7];  [1]
+ Show Author Affiliations
  1. Brookhaven National Lab. (BNL), Upton, NY (United States)
  2. European Organization for Nuclear Research (CERN), Geneva (Switzerland)
  3. Univ. of Texas, Arlington, TX (United States)
  4. Rutgers Univ., Piscataway, NJ (United States)
  5. SLAC National Accelerator Lab., Menlo Park, CA (United States)
  6. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  7. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  8. Argonne National Lab. (ANL), Argonne, IL (United States)
Publication Date:
Research Org.:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP)
Contributing Org.:
Brookhaven National Lab. (BNL), Upton, NY (United States); Univ. of Texas, Arlington, TX (United States)
OSTI Identifier:
1265526
Grant/Contract Number:  
AC05-00OR22725; AC02-98CH10886; AC02-06CH11357
Resource Type:
Accepted Manuscript
Journal Name:
Journal of Physics. Conference Series
Additional Journal Information:
Journal Volume: 608; Journal Issue: 1; Journal ID: ISSN 1742-6588
Publisher:
IOP Publishing
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Klimentov, A., Buncic, P., De, K., Jha, S., Maeno, T., Mount, R., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., Porter, R. J., Read, K. F., Vaniachine, A., Wells, J. C., and Wenaus, T. Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing. United States: N. p., 2015. Web. doi:10.1088/1742-6596/608/1/012040.
Copy to clipboard
Klimentov, A., Buncic, P., De, K., Jha, S., Maeno, T., Mount, R., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., Porter, R. J., Read, K. F., Vaniachine, A., Wells, J. C., & Wenaus, T. Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing. United States. https://doi.org/10.1088/1742-6596/608/1/012040
Copy to clipboard
Klimentov, A., Buncic, P., De, K., Jha, S., Maeno, T., Mount, R., Nilsson, P., Oleynik, D., Panitkin, S., Petrosyan, A., Porter, R. J., Read, K. F., Vaniachine, A., Wells, J. C., and Wenaus, T. Fri . "Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing". United States. https://doi.org/10.1088/1742-6596/608/1/012040. https://www.osti.gov/servlets/purl/1265526.
Copy to clipboard
@article{osti_1265526,
title = {Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing},
author = {Klimentov, A. and Buncic, P. and De, K. and Jha, S. and Maeno, T. and Mount, R. and Nilsson, P. and Oleynik, D. and Panitkin, S. and Petrosyan, A. and Porter, R. J. and Read, K. F. and Vaniachine, A. and Wells, J. C. and Wenaus, T.},
abstractNote = {The Large Hadron Collider (LHC), operating at the international CERN Laboratory in Geneva, Switzerland, is leading Big Data driven scientific explorations. Experiments at the LHC explore the fundamental nature of matter and the basic forces that shape our universe, and were recently credited for the discovery of a Higgs boson. ATLAS and ALICE are the largest collaborations ever assembled in the sciences and are at the forefront of research at the LHC. To address an unprecedented multi-petabyte data processing challenge, both experiments rely on a heterogeneous distributed computational infrastructure. The ATLAS experiment uses PanDA (Production and Data Analysis) Workload Management System (WMS) for managing the workflow for all data processing on hundreds of data centers. Through PanDA, ATLAS physicists see a single computing facility that enables rapid scientific breakthroughs for the experiment, even though the data centers are physically scattered all over the world. The scale is demonstrated by the following numbers: PanDA manages O(102) sites, O(105) cores, O(108) jobs per year, O(103) users, and ATLAS data volume is O(1017) bytes. In 2013 we started an ambitious program to expand PanDA to all available computing resources, including opportunistic use of commercial and academic clouds and Leadership Computing Facilities (LCF). The project titled 'Next Generation Workload Management and Analysis System for Big Data' (BigPanDA) is funded by DOE ASCR and HEP. Extending PanDA to clouds and LCF presents new challenges in managing heterogeneity and supporting workflow. The BigPanDA project is underway to setup and tailor PanDA at the Oak Ridge Leadership Computing Facility (OLCF) and at the National Research Center "Kurchatov Institute" together with ALICE distributed computing and ORNL computing professionals. Our approach to integration of HPC platforms at the OLCF and elsewhere is to reuse, as much as possible, existing components of the PanDA system. Finally, we will present our current accomplishments with running the PanDA WMS at OLCF and other supercomputers and demonstrate our ability to use PanDA as a portal independent of the computing facilities infrastructure for High Energy and Nuclear Physics as well as other data-intensive science applications.},
doi = {10.1088/1742-6596/608/1/012040},
journal = {Journal of Physics. Conference Series},
number = 1,
volume = 608,
place = {United States},
year = {Fri May 22 00:00:00 EDT 2015},
month = {Fri May 22 00:00:00 EDT 2015}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1088/1742-6596/608/1/012040
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 16 works
Citation information provided by
Web of Science
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

PanDA: distributed production and distributed analysis system for ATLAS
journal, July 2008

Open Science Grid Study of the Coupling between Conformation and Water Content in the Interior of a Protein
journal, October 2008

  • Damjanović, Ana; Miller, Benjamin T.; Wenaus, Torre J.
  • Journal of Chemical Information and Modeling, Vol. 48, Issue 10
  • DOI: 10.1021/ci800263c

The antimatter spectrometer (AMS-02): A particle physics detector in space
journal, April 2008

  • Battiston, Roberto
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 588, Issue 1-2
  • DOI: 10.1016/j.nima.2008.01.044

Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC
journal, September 2012

    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    PanDA Workload Management System Meta-data Segmentation
    journal, January 2015

      Sort by:
      [ × clear filter / sort ]

      Similar Records in DOE PAGES and OSTI.GOV collections: