Efficient monitoring of CRAB jobs at CMS

Silva, J. M. D.; Balcas, J.; Belforte, S.; Ciangottini, D.; Mascheroni, M.; Rupeika, E. A.; Ivanov, T. T.; Hernandez, J. M.; Vaandering, E.

doi:10.1088/1742-6596/898/9/092036

Title: Efficient monitoring of CRAB jobs at CMS

Abstract

CRAB is a tool used for distributed analysis of CMS data. Users can submit sets of jobs with similar requirements (tasks) with a single request. CRAB uses a client-server architecture, where a lightweight client, a server, and ancillary services work together and are maintained by CMS operators at CERN. As with most complex software, good monitoring tools are crucial for efficient use and longterm maintainability. This work gives an overview of the monitoring tools developed to ensure the CRAB server and infrastructure are functional, help operators debug user problems, and minimize overhead and operating cost. This work also illustrates the design choices and gives a report on our experience with the tools we developed and the external ones we used.

Authors:

Silva, J. M. D. ^[1]; Balcas, J. ^[2]; Belforte, S. ^[3]; Ciangottini, D. ^[4]; Mascheroni, M. ^[5]; Rupeika, E. A. ^[6]; Ivanov, T. T. ^[7]; Hernandez, J. M. ^[8]; Vaandering, E. ^[5]

Univ. Estadual Paulista, Sao Paulo (Brazil)
California Inst. of Technology (CalTech), Pasadena, CA (United States)
Istituto Nazionale di Fisica Nucleare (INFN), Trieste (Italy)
Istituto Nazionale di Fisica Nucleare (INFN), Perugia (Italy)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Vilnius Univ. (Lithuania)
Univ. of Sofia (Bulgaria)
Research Centre for Energy, Environment and Technology (CIEMAT), Madrid (Spain)

Publication Date:: Wed Nov 22 00:00:00 EST 2017

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

OSTI Identifier:: 1415638

Report Number(s):: FERMILAB-CONF-17-578-CD
Journal ID: ISSN 1742-6588; 1638628

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Physics. Conference Series

Additional Journal Information:: Journal Volume: 898; Journal Issue: 9; Conference: 22nd International Conference on Computing in High Energy and Nuclear Physics, San Francisco, CA, 10/10-10/14/2016; Journal ID: ISSN 1742-6588

Publisher:: IOP Publishing

Country of Publication:: United States

Language:: English

Subject:: 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS; 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Silva, J. M. D., Balcas, J., Belforte, S., Ciangottini, D., Mascheroni, M., Rupeika, E. A., Ivanov, T. T., Hernandez, J. M., and Vaandering, E. Efficient monitoring of CRAB jobs at CMS.  United States: N. p., 2017. 
Web.  doi:10.1088/1742-6596/898/9/092036.

Copy to clipboard


                    Silva, J. M. D., Balcas, J., Belforte, S., Ciangottini, D., Mascheroni, M., Rupeika, E. A., Ivanov, T. T., Hernandez, J. M., & Vaandering, E. Efficient monitoring of CRAB jobs at CMS.  United States.  https://doi.org/10.1088/1742-6596/898/9/092036

Copy to clipboard


                    Silva, J. M. D., Balcas, J., Belforte, S., Ciangottini, D., Mascheroni, M., Rupeika, E. A., Ivanov, T. T., Hernandez, J. M., and Vaandering, E. Wed .  
"Efficient monitoring of CRAB jobs at CMS".  United States.  https://doi.org/10.1088/1742-6596/898/9/092036.  https://www.osti.gov/servlets/purl/1415638.

Copy to clipboard


                    
@article{osti_1415638,

  title        = {Efficient monitoring of CRAB jobs at CMS},

  author       = {Silva, J. M. D. and Balcas, J. and Belforte, S. and Ciangottini, D. and Mascheroni, M. and Rupeika, E. A. and Ivanov, T. T. and Hernandez, J. M. and Vaandering, E.},

  abstractNote = {CRAB is a tool used for distributed analysis of CMS data. Users can submit sets of jobs with similar requirements (tasks) with a single request. CRAB uses a client-server architecture, where a lightweight client, a server, and ancillary services work together and are maintained by CMS operators at CERN. As with most complex software, good monitoring tools are crucial for efficient use and longterm maintainability. This work gives an overview of the monitoring tools developed to ensure the CRAB server and infrastructure are functional, help operators debug user problems, and minimize overhead and operating cost. This work also illustrates the design choices and gives a report on our experience with the tools we developed and the external ones we used.},

  doi          = {10.1088/1742-6596/898/9/092036},

  journal      = {Journal of Physics. Conference Series},

  number       = 9,

  volume       = 898,

  place        = {United States},

  year         = {Wed Nov 22 00:00:00 EST 2017},

  month        = {Wed Nov 22 00:00:00 EST 2017}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1088/1742-6596/898/9/092036

Other availability

Search WorldCat to find libraries that may hold this journal

Figures / Tables:

Figure 1: CRAB Server Architecture.

All figures and tables (10 total)

Save / Share:

Export Metadata

Save to My Library

Figures / Tables found in this record:

Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

Similar Records in DOE PAGES and OSTI.GOV collections:

Langston University - High Energy Physics (LU-HEP)

Technical Report Snow, Dr., Joel

This final report is presented by Langston University (LU) for the project entitled "Langston University High Energy Physics" (LUHEP) under the direction of principal investigator (PI) and project director Professor Joel Snow. The project encompassed high energy physics research performed at hadron colliders. The PI is a collaborator on the DZero experiment at Fermi National Accelerator Laboratory in Batavia, IL, USA and the ATLAS experiment at CERN in Geneva, Switzerland and was during the entire project period from April 1, 1999 until May 14, 2012. Both experiments seek to understand the fundamental constituents of the physical universe and the forcesmore »« less
https://doi.org/10.2172/1048139

Full Text Available
CMS distributed data analysis with CRAB3

Journal Article Mascheroni, M. ; Balcas, J. ; Belforte, S. ; ... - Journal of Physics. Conference Series

The CMS Remote Analysis Builder (CRAB) is a distributed workflow management tool which facilitates analysis tasks by isolating users from the technical details of the Grid infrastructure. Throughout LHC Run 1, CRAB has been successfully employed by an average of 350 distinct users each week executing about 200,000 jobs per day.CRAB has been significantly upgraded in order to face the new challenges posed by LHC Run 2. Components of the new system include 1) a lightweight client, 2) a central primary server which communicates with the clients through a REST interface, 3) secondary servers which manage user analysis tasks andmore »« less
Cited by 6
https://doi.org/10.1088/1742-6596/664/6/062038

Full Text Available
Distributed analysis with CRAB: The client-server architecture evolution and commissioning

Conference Codispoti, G ; /INFN, Bologna /Bologna U. ; Cinquilli, M ; ... - PoS ACAT08:029,2008

CRAB (CMS Remote Analysis Builder) is the tool used by CMS to enable running physics analysis in a transparent manner over data distributed across many sites. It abstracts out the interaction with the underlying batch farms, grid infrastructure and CMS workload management tools, such that it is easily usable by non-experts. CRAB can be used as a direct interface to the computing system or can delegate the user task to a server. Major efforts have been dedicated to the client-server system development, allowing the user to deal only with a simple and intuitive interface and to delegate all the workmore »« less
Full Text Available
Use of DAGMan in CRAB3 to improve the splitting of CMS user jobs

Journal Article Wolf, M. ; Mascheroni, M. ; Woodard, A. ; ... - Journal of Physics. Conference Series

CRAB3 is a workload management tool used by CMS physicists to analyze data acquired by the Compact Muon Solenoid (CMS) detector at the CERN Large Hadron Collider (LHC). Research in high energy physics often requires the analysis of large collections of files, referred to as datasets. The task is divided into jobs that are distributed among a large collection of worker nodes throughout the Worldwide LHC Computing Grid (WLCG). Splitting a large analysis task into optimally sized jobs is critical to efficient use of distributed computing resources. Jobs that are too big will have excessive runtimes and will not distributemore »« less
https://doi.org/10.1088/1742-6596/898/5/052035

Full Text Available
The PhEDEx next-gen website

Conference Egeland, R. ; Huang, C. H. ; Rossman, P. ; ... - J.Phys.Conf.Ser.

PhEDEx is the data-transfer management solution written by CMS. It consists of agents running at each site, a website for presentation of information, and a web-based data-service for scripted access to information. The website allows users to monitor the progress of data-transfers, the status of site agents and links between sites, and the overall status and behaviour of everything about PhEDEx. It also allows users to make and approve requests for data-transfers and for deletion of data. It is the main point-of-entry for all users wishing to interact with PhEDEx. For several years, the website has consisted of a singlemore »« less
https://doi.org/10.1088/1742-6596/396/3/032117

Full Text Available

Similar Records