The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

Kuznetsov, Valentin; Fischer, Nils Leif; Guo, Yuyi

doi:10.1007/s41781-018-0005-0

Title: The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

Journal Article · 19 March 2018 · Computing and Software for Big Science

DOI: https://doi.org/10.1007/s41781-018-0005-0 · OSTI ID:1437402

^[1]; Fischer, Nils Leif ^[2]; Guo, Yuyi ^[3]

Cornell Univ., Ithaca, NY (United States)
Heidelberg Univ. (Germany)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

The CMS experiment at the CERN LHC developed the Workflow Management Archive system to persistently store unstructured framework job report documents produced by distributed workflow management agents. In this paper we present its architecture, implementation, deployment, and integration with the CMS and CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster. The system leverages modern technologies such as a document oriented database and the Hadoop eco-system to provide the necessary flexibility to reliably process, store, and aggregate $$\mathcal{O}$$(1M) documents on a daily basis. We describe the data transformation, the short and long term storage layers, the query language, along with the aggregation pipeline developed to visualize various performance metrics to assist CMS data operators in assessing the performance of the CMS computing system.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)

Sponsoring Organization:: USDOE Office of Science (SC), High Energy Physics (HEP)

Grant/Contract Number:: AC02-07CH11359

OSTI ID:: 1437402

Report Number(s):: arXiv:1801.03872; FERMILAB-PUB-18-074-CD; 1647570; TRN: US1900324

Journal Information:: Computing and Software for Big Science, Vol. 2, Issue 1; ISSN 2510-2036

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (8)

Using the glideinWMS System as a Common Resource Provisioning Layer in CMS Balcas, J.; Belforte, S.; Bockelman, B. Journal of Physics: Conference Series, Vol. 664, Issue 6 https://doi.org/10.1088/1742-6596/664/6/062031	journal	December 2015
The CMS Data Management System Giffels, M.; Guo, Y.; Kuznetsov, V. Journal of Physics: Conference Series, Vol. 513, Issue 4 https://doi.org/10.1088/1742-6596/513/4/042052	journal	June 2014
The Pilot Way to Grid Resources Using glideinWMS Sfiligoi, Igor; Bradley, Daniel C.; Holzman, Burt 2009 WRI World Congress on Computer Science and Information Engineering https://doi.org/10.1109/CSIE.2009.950	conference	March 2009
CMS computing operations during run 1 Adelman, J.; Alderweireldt, S.; Artieda, J. Journal of Physics: Conference Series, Vol. 513, Issue 3 https://doi.org/10.1088/1742-6596/513/3/032040	journal	June 2014
Distributed computing in practice: the Condor experience Thain, Douglas; Tannenbaum, Todd; Livny, Miron Concurrency and Computation: Practice and Experience, Vol. 17, Issue 2-4, p. 323-356 https://doi.org/10.1002/cpe.938	journal	January 2005
The CMS Remote Analysis Builder (CRAB) Spiga, D.; Lacaprara, S.; Bacchi, W. High Performance Computing – HiPC 2007 https://doi.org/10.1007/978-3-540-77220-0_52	book	January 2007
CMS computing operations during run 1 Adelman, J.; Alderweireldt, S.; Artieda, J. Karlsruhe https://doi.org/10.5445/ir/1000042207	text	January 2014
The CMS Remote Analysis Builder (CRAB) Spiga, D.; Lacaprara, S.; Bacchi, W. High Performance Computing – HiPC 2007 https://doi.org/10.1007/978-3-540-77220-0_52	book	January 2007