skip to main content

DOE PAGESDOE PAGES

This content will become publicly available on March 19, 2019

Title: The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

The CMS experiment at the CERN LHC developed the Workflow Management Archive system to persistently store unstructured framework job report documents produced by distributed workflow management agents. In this paper we present its architecture, implementation, deployment, and integration with the CMS and CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster. The system leverages modern technologies such as a document oriented database and the Hadoop eco-system to provide the necessary flexibility to reliably process, store, and aggregate $$\mathcal{O}$$(1M) documents on a daily basis. We describe the data transformation, the short and long term storage layers, the query language, along with the aggregation pipeline developed to visualize various performance metrics to assist CMS data operators in assessing the performance of the CMS computing system.
Authors:
ORCiD logo [1] ;  [2] ;  [3]
  1. Cornell Univ., Ithaca, NY (United States)
  2. Heidelberg Univ. (Germany)
  3. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Publication Date:
Report Number(s):
arXiv:1801.03872; FERMILAB-PUB-18-074-CD
Journal ID: ISSN 2510-2036; 1647570
Grant/Contract Number:
AC02-07CH11359
Type:
Accepted Manuscript
Journal Name:
Computing and Software for Big Science
Additional Journal Information:
Journal Volume: 2; Journal Issue: 1; Journal ID: ISSN 2510-2036
Publisher:
Springer
Research Org:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
Country of Publication:
United States
Language:
English
Subject:
72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS
OSTI Identifier:
1437402