Integration of end-user Cloud storage for CMS analysis

Riahi, Hassen; Aimar, Alberto; Ayllon, Alejandro Alvarez; Balcas, Justas; Ciangottini, Diego; Hernandez, Jose M.; Keeble, Oliver; Magini, Nicolo; Manzi, Andrea; Mascetti, Luca; Mascheroni, Marco; Tanasijczuk, Andres Jorge; Vaandering, Eric Wayne

doi:10.1016/j.future.2017.04.021

Title: Integration of end-user Cloud storage for CMS analysis

Full Record
Other Related Research

Abstract

End-user Cloud storage is increasing rapidly in popularity in research communities thanks to the collaboration capabilities it offers, namely synchronisation and sharing. CERN IT has implemented a model of such storage named, CERNBox, integrated with the CERN AuthN and AuthZ services. To exploit the use of the end-user Cloud storage for the distributed data analysis activity, the CMS experiment has started the integration of CERNBox as a Grid resource. This will allow CMS users to make use of their own storage in the Cloud for their analysis activities as well as to benefit from synchronisation and sharing capabilities to achieve results faster and more effectively. It will provide an integration model of Cloud storages in the Grid, which is implemented and commissioned over the world’s largest computing Grid infrastructure, Worldwide LHC Computing Grid (WLCG). In this paper, we present the integration strategy and infrastructure changes needed in order to transparently integrate end-user Cloud storage with the CMS distributed computing model. We describe the new challenges faced in data management between Grid and Cloud and how they were addressed, along with details of the support for Cloud storage recently introduced into the WLCG data movement middleware, FTS3. Finally, the commissioning experiencemore »« less

Authors:

Riahi, Hassen ^[1]; Aimar, Alberto ^[1]; Ayllon, Alejandro Alvarez ^[1]; Balcas, Justas ^[2]; Ciangottini, Diego ^[3]; Hernandez, Jose M. ^[4]; Keeble, Oliver ^[1]; Magini, Nicolo ^[5]; Manzi, Andrea ^[1]; Mascetti, Luca ^[1]; Mascheroni, Marco ^[5]; Tanasijczuk, Andres Jorge ^[6]; Vaandering, Eric Wayne ^[5]

European Organization for Nuclear Research, Geneva (Switzerland)
California Inst. of Technology (CalTech), Pasadena, CA (United States)
Univ. and INFN Perugia, Perugia (Italy)
CIEMAT, Madrid (Spain)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Univ. of California, San Diego, La Jolla, CA (United States)

Publication Date:: Fri May 19 00:00:00 EDT 2017

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

OSTI Identifier:: 1420408

Report Number(s):: FERMILAB-PUB-17-650
Journal ID: ISSN 0167-739X; 1653645; TRN: US1801487

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Accepted Manuscript

Journal Name:: Future Generations Computer Systems

Additional Journal Information:: Journal Volume: 78; Journal Issue: P3; Journal ID: ISSN 0167-739X

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Data transfer protocols; Cloud storage; Distributed data management; Distributed data analysis

Citation Formats


                    Riahi, Hassen, Aimar, Alberto, Ayllon, Alejandro Alvarez, Balcas, Justas, Ciangottini, Diego, Hernandez, Jose M., Keeble, Oliver, Magini, Nicolo, Manzi, Andrea, Mascetti, Luca, Mascheroni, Marco, Tanasijczuk, Andres Jorge, and Vaandering, Eric Wayne. Integration of end-user Cloud storage for CMS analysis.  United States: N. p., 2017. 
Web.  doi:10.1016/j.future.2017.04.021.

Copy to clipboard


                    Riahi, Hassen, Aimar, Alberto, Ayllon, Alejandro Alvarez, Balcas, Justas, Ciangottini, Diego, Hernandez, Jose M., Keeble, Oliver, Magini, Nicolo, Manzi, Andrea, Mascetti, Luca, Mascheroni, Marco, Tanasijczuk, Andres Jorge, & Vaandering, Eric Wayne. Integration of end-user Cloud storage for CMS analysis.  United States.  https://doi.org/10.1016/j.future.2017.04.021

Copy to clipboard


                    Riahi, Hassen, Aimar, Alberto, Ayllon, Alejandro Alvarez, Balcas, Justas, Ciangottini, Diego, Hernandez, Jose M., Keeble, Oliver, Magini, Nicolo, Manzi, Andrea, Mascetti, Luca, Mascheroni, Marco, Tanasijczuk, Andres Jorge, and Vaandering, Eric Wayne. Fri .  
"Integration of end-user Cloud storage for CMS analysis".  United States.  https://doi.org/10.1016/j.future.2017.04.021.  https://www.osti.gov/servlets/purl/1420408.

Copy to clipboard


                    
@article{osti_1420408,

  title        = {Integration of end-user Cloud storage for CMS analysis},

  author       = {Riahi, Hassen and Aimar, Alberto and Ayllon, Alejandro Alvarez and Balcas, Justas and Ciangottini, Diego and Hernandez, Jose M. and Keeble, Oliver and Magini, Nicolo and Manzi, Andrea and Mascetti, Luca and Mascheroni, Marco and Tanasijczuk, Andres Jorge and Vaandering, Eric Wayne},

  abstractNote = {End-user Cloud storage is increasing rapidly in popularity in research communities thanks to the collaboration capabilities it offers, namely synchronisation and sharing. CERN IT has implemented a model of such storage named, CERNBox, integrated with the CERN AuthN and AuthZ services. To exploit the use of the end-user Cloud storage for the distributed data analysis activity, the CMS experiment has started the integration of CERNBox as a Grid resource. This will allow CMS users to make use of their own storage in the Cloud for their analysis activities as well as to benefit from synchronisation and sharing capabilities to achieve results faster and more effectively. It will provide an integration model of Cloud storages in the Grid, which is implemented and commissioned over the world’s largest computing Grid infrastructure, Worldwide LHC Computing Grid (WLCG). In this paper, we present the integration strategy and infrastructure changes needed in order to transparently integrate end-user Cloud storage with the CMS distributed computing model. We describe the new challenges faced in data management between Grid and Cloud and how they were addressed, along with details of the support for Cloud storage recently introduced into the WLCG data movement middleware, FTS3. Finally, the commissioning experience of CERNBox for the distributed data analysis activity is also presented.},

  doi          = {10.1016/j.future.2017.04.021},

  journal      = {Future Generations Computer Systems},

  number       = P3,

  volume       = 78,

  place        = {United States},

  year         = {Fri May 19 00:00:00 EDT 2017},

  month        = {Fri May 19 00:00:00 EDT 2017}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.future.2017.04.021

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 1 work

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

The CMS Remote Analysis Builder (CRAB)

Conference Spiga, D ; Cinquilli, M ; Servoli, L ; ... - Lect.Notes Comput.Sci.4873:580-586,2007

The CMS experiment will produce several Pbytes of data every year, to be distributed over many computing centers geographically distributed in different countries. Analysis of this data will be also performed in a distributed way, using grid infrastructure. CRAB (CMS Remote Analysis Builder) is a specific tool, designed and developed by the CMS collaboration, that allows a transparent access to distributed data to end physicist. Very limited knowledge of underlying technicalities are required to the user. CRAB interacts with the local user environment, the CMS Data Management services and with the Grid middleware. It is able to use WLCG, gLitemore »« less
https://doi.org/10.1007/978-3-540-77220-0_52
Large scale and low latency analysis facilities for the CMS experiment: Development and operational aspects

Conference Riahi, H. ; Gowdy, S. ; Kreuzer, P. ; ... - J.Phys.Conf.Ser.

While a majority of CMS data analysis activities rely on the distributed computing infrastructure on the WLCG Grid, dedicated local computing facilities have been deployed to address particular requirements in terms of latency and scale. The CMS CERN Analysis Facility (CAF) was primarily designed to host a large variety of latency-critical workflows. These break down into alignment and calibration, detector commissioning and diagnosis, and high-interest physics analysis requiring fast turnaround. In order to reach the goal for fast turnaround tasks, the Workload Management group has designed a CRABServer based system to fit with two main needs: to provide a simple,more »« less
https://doi.org/10.1088/1742-6596/331/7/072030

Full Text Available
Experience with dynamic resource provisioning of the CMS online cluster using a cloud overlay

Journal Article Andre, Jean-Marc ; Behrens, Ulf ; Branson, James ; ... - EPJ Web of Conferences

The primary goal of the online cluster of the Compact Muon Solenoid (CMS) experiment at the Large Hadron Collider (LHC) is to build event data from the detector and to select interesting collisions in the High Level Trigger (HLT) farm for offline storage. With more than 1500 nodes and a capacity of about 850 kHEPSpecInt06, the HLT machines represent similar computing capacity of all the CMS Tier1 Grid sites together. Moreover, it is currently connected to the CERN IT datacenter via a dedicated 160 Gbps network connection and hence can access the remote EOS based storage with a high bandwidth.more »« less
https://doi.org/10.1051/epjconf/201921407017

Full Text Available
Langston University - High Energy Physics (LU-HEP)

Technical Report Snow, Dr., Joel

This final report is presented by Langston University (LU) for the project entitled "Langston University High Energy Physics" (LUHEP) under the direction of principal investigator (PI) and project director Professor Joel Snow. The project encompassed high energy physics research performed at hadron colliders. The PI is a collaborator on the DZero experiment at Fermi National Accelerator Laboratory in Batavia, IL, USA and the ATLAS experiment at CERN in Geneva, Switzerland and was during the entire project period from April 1, 1999 until May 14, 2012. Both experiments seek to understand the fundamental constituents of the physical universe and the forcesmore »« less
https://doi.org/10.2172/1048139

Full Text Available
Final report

Technical Report Weissman, Jon B

High performance computational science and engineering simulations have become an increasingly important part of the scientist's problem solving toolset. A key reason is the development of widely used codes and libraries that support these applications, for example, Netlib, a collection of numerical libraries [33]. The term community codes refers to those libraries or applications that have achieved some critical level of acceptance by a user community. Many of these applications are on the high-end in terms of required resources: computation, storage, and communication. Recently, there has been considerable interest in putting such applications on-line and packaging them as network servicesmore »« less
https://doi.org/10.2172/1007978

Full Text Available

Similar Records