Experience in using commercial clouds in CMS

Bauerdick, L.; Bockelman, B.; Dykstra, D.; Fuess, S.; Garzoglio, G.; Girone, M.; Gutsche, O.; Holzman, B.; Hufnagel, D.; Kim, H.; Kennedy, R.; Mason, D.; Spentzouris, P.; Timm, S.; Tiradani, A.; Vaandering, E.

doi:10.1088/1742-6596/898/5/052019

Title: Experience in using commercial clouds in CMS

Abstract

Historically high energy physics computing has been performed on large purpose-built computing systems. In the beginning there were single site computing facilities, which evolved into the Worldwide LHC Computing Grid (WLCG) used today. The vast majority of the WLCG resources are used for LHC computing and the resources are scheduled to be continuously used throughout the year. In the last several years there has been an explosion in capacity and capability of commercial and academic computing clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest amongst the cloud providers to demonstrate the capability to perform large scale scientific computing. In this presentation we will discuss results from the CMS experiment using the Fermilab HEPCloud Facility, which utilized both local Fermilab resources and Amazon Web Services (AWS). The goal was to work with AWS through a matching grant to demonstrate a sustained scale approximately equal to half of the worldwide processing resources available to CMS. We will discuss the planning and technical challenges involved in organizing the most IO intensive CMS workflows on a large-scale set of virtualized resource provisioned by the Fermilab HEPCloud.more »« less

Authors:: Bauerdick, L.; Bockelman, B.; Dykstra, D.; Fuess, S.; Garzoglio, G.; Girone, M.; Gutsche, O.; Holzman, B.; Hufnagel, D.; Kim, H.; Kennedy, R.; Mason, D.; Spentzouris, P.; Timm, S.; Tiradani, A.; Vaandering, E.

Publication Date:: Wed Nov 22 00:00:00 EST 2017

Research Org.:: Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC), High Energy Physics (HEP)

Contributing Org.:: CMS Collaboration

OSTI Identifier:: 1402473

Report Number(s):: FERMILAB-CONF-17-405-CD
Journal ID: ISSN 1742-6588; 1638477

Grant/Contract Number:: AC02-07CH11359

Resource Type:: Accepted Manuscript

Journal Name:: Journal of Physics. Conference Series

Additional Journal Information:: Journal Volume: 898; Journal Issue: 5; Journal ID: ISSN 1742-6588

Publisher:: IOP Publishing

Country of Publication:: United States

Language:: English

Subject:: 72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS; 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Bauerdick, L., Bockelman, B., Dykstra, D., Fuess, S., Garzoglio, G., Girone, M., Gutsche, O., Holzman, B., Hufnagel, D., Kim, H., Kennedy, R., Mason, D., Spentzouris, P., Timm, S., Tiradani, A., and Vaandering, E. Experience in using commercial clouds in CMS.  United States: N. p., 2017. 
Web.  doi:10.1088/1742-6596/898/5/052019.

Copy to clipboard


                    Bauerdick, L., Bockelman, B., Dykstra, D., Fuess, S., Garzoglio, G., Girone, M., Gutsche, O., Holzman, B., Hufnagel, D., Kim, H., Kennedy, R., Mason, D., Spentzouris, P., Timm, S., Tiradani, A., & Vaandering, E. Experience in using commercial clouds in CMS.  United States.  https://doi.org/10.1088/1742-6596/898/5/052019

Copy to clipboard


                    Bauerdick, L., Bockelman, B., Dykstra, D., Fuess, S., Garzoglio, G., Girone, M., Gutsche, O., Holzman, B., Hufnagel, D., Kim, H., Kennedy, R., Mason, D., Spentzouris, P., Timm, S., Tiradani, A., and Vaandering, E. Wed .  
"Experience in using commercial clouds in CMS".  United States.  https://doi.org/10.1088/1742-6596/898/5/052019.  https://www.osti.gov/servlets/purl/1402473.

Copy to clipboard


                    
@article{osti_1402473,

  title        = {Experience in using commercial clouds in CMS},

  author       = {Bauerdick, L. and Bockelman, B. and Dykstra, D. and Fuess, S. and Garzoglio, G. and Girone, M. and Gutsche, O. and Holzman, B. and Hufnagel, D. and Kim, H. and Kennedy, R. and Mason, D. and Spentzouris, P. and Timm, S. and Tiradani, A. and Vaandering, E.},

  abstractNote = {Historically high energy physics computing has been performed on large purpose-built computing systems. In the beginning there were single site computing facilities, which evolved into the Worldwide LHC Computing Grid (WLCG) used today. The vast majority of the WLCG resources are used for LHC computing and the resources are scheduled to be continuously used throughout the year. In the last several years there has been an explosion in capacity and capability of commercial and academic computing clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest amongst the cloud providers to demonstrate the capability to perform large scale scientific computing. In this presentation we will discuss results from the CMS experiment using the Fermilab HEPCloud Facility, which utilized both local Fermilab resources and Amazon Web Services (AWS). The goal was to work with AWS through a matching grant to demonstrate a sustained scale approximately equal to half of the worldwide processing resources available to CMS. We will discuss the planning and technical challenges involved in organizing the most IO intensive CMS workflows on a large-scale set of virtualized resource provisioned by the Fermilab HEPCloud. We will describe the data handling and data management challenges. Also, we will discuss the economic issues and cost and operational efficiency comparison to our dedicated resources. At the end we will consider the changes in the working model of HEP computing in a domain with the availability of large scale resources scheduled at peak times.},

  doi          = {10.1088/1742-6596/898/5/052019},

  journal      = {Journal of Physics. Conference Series},

  number       = 5,

  volume       = 898,

  place        = {United States},

  year         = {Wed Nov 22 00:00:00 EST 2017},

  month        = {Wed Nov 22 00:00:00 EST 2017}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1088/1742-6596/898/5/052019

Other availability

Search WorldCat to find libraries that may hold this journal

Figures / Tables:

Figure 1: An illustration of provisioning for average vs. provisioning for peak.

All figures and tables (6 total)

Save / Share:

Export Metadata

Save to My Library

Figures / Tables found in this record:

Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

Similar Records in DOE PAGES and OSTI.GOV collections:

The HEPCloud Facility: elastic computing for High Energy Physics – The NOvA Use Case

Conference Fuess, S. ; Garzoglio, G. ; Holzman, B. ; ...

The need for computing in the HEP community follows cycles of peaks and valleys mainly driven by conference dates, accelerator shutdown, holiday schedules, and other factors. Because of this, the classical method of provisioning these resources at providing facilities has drawbacks such as potential overprovisioning. As the appetite for computing increases, however, so does the need to maximize cost efficiency by developing a model for dynamically provisioning resources only when needed. To address this issue, the HEPCloud project was launched by the Fermilab Scientific Computing Division in June 2015. Its goal is to develop a facility that provides a commonmore »« less
Full Text Available
Virtual machine provisioning, code management, and data movement design for the Fermilab HEPCloud Facility

Journal Article Timm, S. ; Cooper, G. ; Fuess, S. ; ... - Journal of Physics. Conference Series

The Fermilab HEPCloud Facility Project has as its goal to extend the current Fermilab facility interface to provide transparent access to disparate resources including commercial and community clouds, grid federations, and HPC centers. This facility enables experiments to perform the full spectrum of computing tasks, including data-intensive simulation and reconstruction. We have evaluated the use of the commercial cloud to provide elasticity to respond to peaks of demand without overprovisioning local resources. Full scale data-intensive workflows have been successfully completed on Amazon Web Services for two High Energy Physics Experiments, CMS and NOνA, at the scale of 58000 simultaneous cores.more »« less
https://doi.org/10.1088/1742-6596/898/5/052041

Full Text Available
HEPCloud, a New Paradigm for HEP Facilities: CMS Amazon Web Services Investigation

Journal Article Holzman, Burt ; Bauerdick, Lothar A. T. ; Bockelman, Brian ; ... - Computing and Software for Big Science

Historically, high energy physics computing has been performed on large purpose-built computing systems. These began as single-site compute facilities, but have evolved into the distributed computing grids used today. Recently, there has been an exponential increase in the capacity and capability of commercial clouds. Cloud resources are highly virtualized and intended to be able to be flexibly deployed for a variety of computing tasks. There is a growing interest among the cloud providers to demonstrate the capability to perform large-scale scientific computing. In this paper, we discuss results from the CMS experiment using the Fermilab HEPCloud facility, which utilized bothmore »« less
https://doi.org/10.1007/s41781-017-0001-9

Full Text Available
HPC resource integration into CMS Computing via HEPCloud

Journal Article Hufnagel, Dirk ; Holzman, Burt ; Mason, David ; ... - EPJ Web of Conferences

The higher energy and luminosity from the LHC in Run 2 have put increased pressure on CMS computing resources. Extrapolating to even higher luminosities (and thus higher event complexities and trigger rates) beyond Run 3, it becomes clear that simply scaling up the the current model of CMS computing alone will become economically unfeasible. High Performance Computing (HPC) facilities, widely used in scientific computing outside of HEP, have the potential to help fill the gap. Here we describe the U.S.CMS efforts to integrate US HPC resources into CMS Computing via the HEPCloud project at Fermilab. We present advancements in ourmore »« less
https://doi.org/10.1051/epjconf/201921403031

Full Text Available
Integration of end-user Cloud storage for CMS analysis

Journal Article Riahi, Hassen ; Aimar, Alberto ; Ayllon, Alejandro Alvarez ; ... - Future Generations Computer Systems

End-user Cloud storage is increasing rapidly in popularity in research communities thanks to the collaboration capabilities it offers, namely synchronisation and sharing. CERN IT has implemented a model of such storage named, CERNBox, integrated with the CERN AuthN and AuthZ services. To exploit the use of the end-user Cloud storage for the distributed data analysis activity, the CMS experiment has started the integration of CERNBox as a Grid resource. This will allow CMS users to make use of their own storage in the Cloud for their analysis activities as well as to benefit from synchronisation and sharing capabilities to achievemore »« less
Cited by 1
https://doi.org/10.1016/j.future.2017.04.021

Full Text Available

Similar Records