skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: The Open High Throughput Computing Content Delivery Network

Abstract

LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases where experiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Openmore » HTC CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.« less

Authors:
ORCiD logo [1];  [2];  [3];  [3]
  1. Fermilab
  2. U. Nebraska, Lincoln
  3. CERN
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
Contributing Org.:
CERN-LHC-CMS
OSTI Identifier:
1490833
Report Number(s):
FERMILAB-CONF-18-582-CD
1712632
DOE Contract Number:  
AC02-07CH11359
Resource Type:
Conference
Resource Relation:
Conference: 23rd International Conference on Computing in High Energy and Nuclear Physics, Sofia, Bulgaria, 07/09-07/13/2018
Country of Publication:
United States
Language:
English

Citation Formats

Dykstra, Dave, Bockelman, Brian, Blomer, Jakob, and Field, Laurence. The Open High Throughput Computing Content Delivery Network. United States: N. p., 2018. Web.
Dykstra, Dave, Bockelman, Brian, Blomer, Jakob, & Field, Laurence. The Open High Throughput Computing Content Delivery Network. United States.
Dykstra, Dave, Bockelman, Brian, Blomer, Jakob, and Field, Laurence. Wed . "The Open High Throughput Computing Content Delivery Network". United States. https://www.osti.gov/servlets/purl/1490833.
@article{osti_1490833,
title = {The Open High Throughput Computing Content Delivery Network},
author = {Dykstra, Dave and Bockelman, Brian and Blomer, Jakob and Field, Laurence},
abstractNote = {LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases where experiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Open HTC CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {11}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: