skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: The Open High Throughput Computing Content Delivery Network

Abstract

LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases whereexperiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Open HTCmore » CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.« less

Authors:
ORCiD logo [1];  [2];  [3];  [3]
  1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  2. Univ. of Nebraska, Lincoln, NE (United States)
  3. European Organization for Nuclear Research (CERN), Geneva (Switzerland)
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
OSTI Identifier:
1490833
Report Number(s):
FERMILAB-CONF-18-582-CD; CMS-CR-2018-287
Journal ID: ISSN 2100-014X; oai:inspirehep.net:1712632
Grant/Contract Number:  
AC02-07CH11359
Resource Type:
Accepted Manuscript
Journal Name:
EPJ Web of Conferences
Additional Journal Information:
Journal Volume: 214; Conference: 23rd International Conference on Computing in High Energy and Nuclear Physics, Sofia, Bulgaria, 07/09-07/13/2018; Journal ID: ISSN 2100-014X
Publisher:
EDP Sciences
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Dykstra, Dave, Bockelman, Brian, Blomer, Jakob, and Field, Laurence. The Open High Throughput Computing Content Delivery Network. United States: N. p., 2019. Web. doi:10.1051/epjconf/201921404023.
Dykstra, Dave, Bockelman, Brian, Blomer, Jakob, & Field, Laurence. The Open High Throughput Computing Content Delivery Network. United States. doi:10.1051/epjconf/201921404023.
Dykstra, Dave, Bockelman, Brian, Blomer, Jakob, and Field, Laurence. Tue . "The Open High Throughput Computing Content Delivery Network". United States. doi:10.1051/epjconf/201921404023. https://www.osti.gov/servlets/purl/1490833.
@article{osti_1490833,
title = {The Open High Throughput Computing Content Delivery Network},
author = {Dykstra, Dave and Bockelman, Brian and Blomer, Jakob and Field, Laurence},
abstractNote = {LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases whereexperiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Open HTC CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.},
doi = {10.1051/epjconf/201921404023},
journal = {EPJ Web of Conferences},
number = ,
volume = 214,
place = {United States},
year = {2019},
month = {9}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Scaling HEP to Web Size with RESTful Protocols: The Frontier Example
journal, December 2011


Status and future perspectives of CernVM-FS
journal, December 2012


Web Proxy Auto Discovery for the WLCG
journal, October 2017


Greatly improved cache update times for conditions data with Frontier/Squid
journal, April 2010