Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

A comparison of different database technologies for the CMS AsyncStageOut transfer database

Journal Article · · Journal of Physics. Conference Series
 [1];  [2];  [3];  [4];  [3];  [5];  [6];  [7];  [8];  [9]
  1. INFN Sezione di Perugia, Perugia (Italy)
  2. California Inst. of Technology (CalTech), Pasadena, CA (United States)
  3. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  4. Vilnius Univ. (Lithuania)
  5. European Organization for Nuclear Research (CERN), Geneva (Switzerland)
  6. Univ. Estadual Paulista, Sao Paulo (Brazil)
  7. CIEMAT, Madrid (Spain)
  8. INFN Sezione di Trieste, Trieste (Italy)
  9. Univ. of Sofia"St. Kliment Ohridski" (Bulgaria)
+ Show Author Affiliations
AsyncStageOut (ASO) is the component of the CMS distributed data analysis system (CRAB) that manages users transfers in a centrally controlled way using the File Transfer System (FTS3) at CERN. It addresses a major weakness of the previous, decentralized model, namely that the transfer of the user’s output data to a single remote site was part of the job execution, resulting in inefficient use of job slots and an unacceptable failure rate. Currently ASO manages up to 600k files of various sizes per day from more than 500 users per month, spread over more than 100 sites. ASO uses a NoSQL database (CouchDB) as internal bookkeeping and as way to communicate with other CRAB components. Since ASO/CRAB were put in production in 2014, the number of transfers constantly increased up to a point where the pressure to the central CouchDB instance became critical, creating new challenges for the system scalability, performance, and monitoring. This forced a re-engineering of the ASO application to increase its scalability and lowering its operational effort. In this contribution we present a comparison of the performance of the current NoSQL implementation and a new SQL implementation, and how their different strengths and features influenced the design choices and operational experience. We also discuss other architectural changes introduced in the system to handle the increasing load and latency in delivering output to the user.
Research Organization:
Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
Sponsoring Organization:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
Grant/Contract Number:
AC02-07CH11359
OSTI ID:
1420917
Report Number(s):
FERMILAB-CONF--16-756-CD; 1638458
Journal Information:
Journal of Physics. Conference Series, Journal Name: Journal of Physics. Conference Series Journal Issue: 4 Vol. 898; ISSN 1742-6588
Publisher:
IOP PublishingCopyright Statement
Country of Publication:
United States
Language:
English

Figures / Tables (6)

Figure 1(p. 3)figure Figure 1
Figure 2(p. 4)figure Figure 2
Figure 3(p. 5)figure Figure 3
Figure 4(p. 5)figure Figure 4
Figure 5(p. 6)figure Figure 5
Figure 6(p. 6)figure Figure 6

Similar Records

AsyncStageOut: Distributed User Data Management for CMS Analysis
Conference · Tue Dec 22 23:00:00 EST 2015 · J.Phys.Conf.Ser. · OSTI ID:1247517
Comparison of the Frontier Distributed Database Caching System to NoSQL Databases
Conference · Fri Jul 20 00:00:00 EDT 2012 · J.Phys.Conf.Ser. · OSTI ID:1419283
A gLite FTS based solution for managing user output in CMS
Conference · Sat Dec 31 23:00:00 EST 2011 · J.Phys.Conf.Ser. · OSTI ID:1405168

Related Subjects

71 CLASSICAL AND QUANTUM MECHANICS
GENERAL PHYSICS