The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

Kuznetsov, Valentin; Fischer, Nils Leif; Guo, Yuyi

doi:10.1007/s41781-018-0005-0

Title: The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

Abstract

The CMS experiment at the CERN LHC developed the Workflow Management Archive system to persistently store unstructured framework job report documents produced by distributed workflow management agents. In this paper we present its architecture, implementation, deployment, and integration with the CMS and CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster. The system leverages modern technologies such as a document oriented database and the Hadoop eco-system to provide the necessary flexibility to reliably process, store, and aggregate $$\mathcal{O}$$(1M) documents on a daily basis. We describe the data transformation, the short and long term storage layers, the query language, along with the aggregation pipeline developed to visualize various performance metrics to assist CMS data operators in assessing the performance of the CMS computing system.

Authors:

^[1]; Fischer, Nils Leif ^[2]; Guo, Yuyi ^[3]

Cornell Univ., Ithaca, NY (United States)
Heidelberg Univ. (Germany)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Publication Date:: Mon Mar 19 00:00:00 EDT 2018

Research Org.:: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

OSTI Identifier:: 1437402

Report Number(s):: arXiv:1801.03872; FERMILAB-PUB-18-074-CD
Journal ID: ISSN 2510-2036; 1647570; TRN: US1900324

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Accepted Manuscript

Journal Name:: Computing and Software for Big Science

Additional Journal Information:: Journal Volume: 2; Journal Issue: 1; Journal ID: ISSN 2510-2036

Publisher:: Springer

Country of Publication:: United States

Language:: English

Subject:: 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS

Citation Formats


                    Kuznetsov, Valentin, Fischer, Nils Leif, and Guo, Yuyi. The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC.  United States: N. p., 2018. 
Web.  doi:10.1007/s41781-018-0005-0.

Copy to clipboard


                    Kuznetsov, Valentin, Fischer, Nils Leif, & Guo, Yuyi. The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC.  United States.  https://doi.org/10.1007/s41781-018-0005-0

Copy to clipboard


                    Kuznetsov, Valentin, Fischer, Nils Leif, and Guo, Yuyi. Mon .  
"The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC".  United States.  https://doi.org/10.1007/s41781-018-0005-0.  https://www.osti.gov/servlets/purl/1437402.

Copy to clipboard


                    
@article{osti_1437402,

  title        = {The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC},

  author       = {Kuznetsov, Valentin and Fischer, Nils Leif and Guo, Yuyi},

  abstractNote = {The CMS experiment at the CERN LHC developed the Workflow Management Archive system to persistently store unstructured framework job report documents produced by distributed workflow management agents. In this paper we present its architecture, implementation, deployment, and integration with the CMS and CERN computing infrastructures, such as central HDFS and Hadoop Spark cluster. The system leverages modern technologies such as a document oriented database and the Hadoop eco-system to provide the necessary flexibility to reliably process, store, and aggregate $\mathcal{O}$(1M) documents on a daily basis. We describe the data transformation, the short and long term storage layers, the query language, along with the aggregation pipeline developed to visualize various performance metrics to assist CMS data operators in assessing the performance of the CMS computing system.},

  doi          = {10.1007/s41781-018-0005-0},

  journal      = {Computing and Software for Big Science},

  number       = 1,

  volume       = 2,

  place        = {United States},

  year         = {Mon Mar 19 00:00:00 EDT 2018},

  month        = {Mon Mar 19 00:00:00 EDT 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1007/s41781-018-0005-0

Other availability

Search WorldCat to find libraries that may hold this journal

Figures / Tables:

Fig. 1 : WMArchive architecture. MongoDB represents shortterm storage (STS) and HDFS refers to long-term storage (LTS) discussed in a text.

All figures and tables (4 total)

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Using the glideinWMS System as a Common Resource Provisioning Layer in CMS
journal, December 2015

Balcas, J.; Belforte, S.; Bockelman, B.
Journal of Physics: Conference Series, Vol. 664, Issue 6
DOI: 10.1088/1742-6596/664/6/062031

The CMS Data Management System
journal, June 2014

Giffels, M.; Guo, Y.; Kuznetsov, V.
Journal of Physics: Conference Series, Vol. 513, Issue 4
DOI: 10.1088/1742-6596/513/4/042052

The Pilot Way to Grid Resources Using glideinWMS
conference, March 2009

Sfiligoi, Igor; Bradley, Daniel C.; Holzman, Burt
2009 WRI World Congress on Computer Science and Information Engineering
DOI: 10.1109/CSIE.2009.950

CMS computing operations during run 1
journal, June 2014

Adelman, J.; Alderweireldt, S.; Artieda, J.
Journal of Physics: Conference Series, Vol. 513, Issue 3
DOI: 10.1088/1742-6596/513/3/032040

Distributed computing in practice: the Condor experience
journal, January 2005

Thain, Douglas; Tannenbaum, Todd; Livny, Miron
Concurrency and Computation: Practice and Experience, Vol. 17, Issue 2-4, p. 323-356
DOI: 10.1002/cpe.938

CMS computing operations during run 1
text, January 2014

Adelman, J.; Alderweireldt, S.; Artieda, J.
Karlsruhe
DOI: 10.5445/ir/1000042207

Figures / Tables found in this record:

Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

Similar Records in DOE PAGES and OSTI.GOV collections:

Deactivation and decommissioning web log analysis using big data technology - 15710

Conference Joshi, Santosh ; Upadhyay, Himanshu ; Lagos, Leonel

The D and D Knowledge Management Information Tool (D and D KM- IT) is a web-based knowledge management information tool built for the D and D user community. Big data is a massive volume of structured and unstructured datasets which is so large that it's difficult to process using traditional database techniques. Web logs are the repository of files which are generated automatically for any operation on the web site. Web log files generated from the D and D KM-IT will be processed using the Apache Hadoop Framework to extract meaningful data. Hadoop is an open-source software framework for storingmore »« less
Automatic log analysis with NLP for the CMS workflow handling

Conference Layer, Lukas ; Abercrombie, Daniel Robert ; Bakhshiansohi, Hamed ; ... - EPJ Web Conf.

The central Monte-Carlo production of the CMS experiment utilizes the WLCG infrastructure and manages daily thousands of tasks, each up to thousands of jobs. The distributed computing system is bound to sustain a certain rate of failures of various types, which are currently handled by computing operators a posteriori. Within the context of computing operations, and operation intelligence, we propose a Machine Learning technique to learn from the operators with a view to reduce the operational workload and delays. This work is in continuation of CMS work on operation intelligence to try and reach accurate predictions with Machine Learning. Wemore »« less
https://doi.org/10.1051/epjconf/202024503006

Full Text Available
Optimizing CMS build infrastructure via Apache Mesos

Journal Article Abdurachmanov, David ; Degano, Alessandro ; Elmer, Peter ; ... - Journal of Physics. Conference Series

The Offline Software of the CMS Experiment at the Large Hadron Collider (LHC) at CERN consists of 6M lines of in-house code, developed over a decade by nearly 1000 physicists, as well as a comparable amount of general use open-source code. A critical ingredient to the success of the construction and early operation of the WLCG was the convergence, around the year 2000, on the use of a homogeneous environment of commodity x86-64 processors and Linux.Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run Hadoop, Jenkins, Spark, Aurora,more »« less
https://doi.org/10.1088/1742-6596/664/6/062013

Full Text Available
CMS distributed computing workflow experience

Conference Adelman-McCarthy, Jennifer ; et al. - J.Phys.Conf.Ser.

The vast majority of the CMS Computing capacity, which is organized in a tiered hierarchy, is located away from CERN. The 7 Tier-1 sites archive the LHC proton-proton collision data that is initially processed at CERN. These sites provide access to all recorded and simulated data for the Tier-2 sites, via wide-area network (WAN) transfers. All central data processing workflows are executed at the Tier-1 level, which contain re-reconstruction and skimming workflows of collision data as well as reprocessing of simulated data to adapt to changing detector conditions. This paper describes the operation of the CMS processing infrastructure at themore »« less
https://doi.org/10.1088/1742-6596/331/7/072019

Full Text Available
Adding Data Management Services to Parallel File Systems

Technical Report Brandt, Scott

The objective of this project, called DAMASC for “Data Management in Scientific Computing”, is to coalesce data management with parallel file system management to present a declarative interface to scientists for managing, querying, and analyzing extremely large data sets efficiently and predictably. Managing extremely large data sets is a key challenge of exascale computing. The overhead, energy, and cost of moving massive volumes of data demand designs where computation is close to storage. In current architectures, compute/analysis clusters access data in a physically separate parallel file system and largely leave it scientist to reduce data movement. Over the past decadesmore »« less
https://doi.org/10.2172/1171718

Full Text Available

Similar Records

Title: The Archive Solution for Distributed Workflow Management Agents of the CMS Experiment at LHC

Abstract

Citation Formats

Figures / Tables:

Using the glideinWMS System as a Common Resource Provisioning Layer in CMS journal, December 2015

The CMS Data Management System journal, June 2014

The Pilot Way to Grid Resources Using glideinWMS conference, March 2009

CMS computing operations during run 1 journal, June 2014

Distributed computing in practice: the Condor experience journal, January 2005

CMS computing operations during run 1 text, January 2014

Using the glideinWMS System as a Common Resource Provisioning Layer in CMS
journal, December 2015

The CMS Data Management System
journal, June 2014

The Pilot Way to Grid Resources Using glideinWMS
conference, March 2009

CMS computing operations during run 1
journal, June 2014

Distributed computing in practice: the Condor experience
journal, January 2005

CMS computing operations during run 1
text, January 2014