skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Web Proxy Auto Discovery for the WLCG

Abstract

All four of the LHC experiments depend on web proxies (that is, squids) at each grid site to support software distribution by the CernVM FileSystem (CVMFS). CMS and ATLAS also use web proxies for conditions data distributed through the Frontier Distributed Database caching system. ATLAS & CMS each have their own methods for their grid jobs to find out which web proxies to use for Frontier at each site, and CVMFS has a third method. Those diverse methods limit usability and flexibility, particularly for opportunistic use cases, where an experiment’s jobs are run at sites that do not primarily support that experiment. This paper describes a new Worldwide LHC Computing Grid (WLCG) system for discovering the addresses of web proxies. The system is based on an internet standard called Web Proxy Auto Discovery (WPAD). WPAD is in turn based on another standard called Proxy Auto Configuration (PAC). Both the Frontier and CVMFS clients support this standard. The input into the WLCG system comes from squids registered in the ATLAS Grid Information System (AGIS) and CMS SITECONF files, cross-checked with squids registered by sites in the Grid Configuration Database (GOCDB) and the OSG Information Management (OIM) system, and combined with somemore » exceptions manually configured by people from ATLAS and CMS who operate WLCG Squid monitoring. WPAD servers at CERN respond to http requests from grid nodes all over the world with a PAC file that lists available web proxies, based on IP addresses matched from a database that contains the IP address ranges registered to organizations. Large grid sites are encouraged to supply their own WPAD web servers for more flexibility, to avoid being affected by short term long distance network outages, and to offload the WLCG WPAD servers at CERN. The CERN WPAD servers additionally support requests from jobs running at non-grid sites (particularly for LHC@Home) which it directs to the nearest publicly accessible web proxy servers. Furthermore, the responses to those requests are geographically ordered based on a separate database that maps IP addresses to longitude and latitude.« less

Authors:
 [1];  [2];  [3];  [4];  [5];  [2]
  1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  2. European Organization for Nuclear Research (CERN), Geneva (Switzerland)
  3. Johns Hopkins Univ., Baltimore, MD (United States)
  4. Istituto Nazionale di Fisica Nucleare, Rome (Italy)
  5. Rutherford Appleton Lab., Oxfordshire (England)
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Sponsoring Org.:
USDOE Office of Science (SC), High Energy Physics (HEP)
OSTI Identifier:
1346930
Report Number(s):
FERMILAB-CONF-17-024-CD
Journal ID: ISSN 1742-6588; 1517405
Grant/Contract Number:  
AC02-07CH11359
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Journal of Physics. Conference Series
Additional Journal Information:
Journal Volume: 898; Conference: 22nd International Conference on Computing in High Energy and Nuclear Physics, San Francisco, CA (United States), 14-16 Oct 2016; Journal ID: ISSN 1742-6588
Publisher:
IOP Publishing
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Dykstra, D., Blomer, J., Blumenfeld, B., De Salvo, Alessandro, Dewhurst, Alastair, and Verguilov, Vassil. Web Proxy Auto Discovery for the WLCG. United States: N. p., 2017. Web. doi:10.1088/1742-6596/898/5/052043.
Dykstra, D., Blomer, J., Blumenfeld, B., De Salvo, Alessandro, Dewhurst, Alastair, & Verguilov, Vassil. Web Proxy Auto Discovery for the WLCG. United States. https://doi.org/10.1088/1742-6596/898/5/052043
Dykstra, D., Blomer, J., Blumenfeld, B., De Salvo, Alessandro, Dewhurst, Alastair, and Verguilov, Vassil. 2017. "Web Proxy Auto Discovery for the WLCG". United States. https://doi.org/10.1088/1742-6596/898/5/052043. https://www.osti.gov/servlets/purl/1346930.
@article{osti_1346930,
title = {Web Proxy Auto Discovery for the WLCG},
author = {Dykstra, D. and Blomer, J. and Blumenfeld, B. and De Salvo, Alessandro and Dewhurst, Alastair and Verguilov, Vassil},
abstractNote = {All four of the LHC experiments depend on web proxies (that is, squids) at each grid site to support software distribution by the CernVM FileSystem (CVMFS). CMS and ATLAS also use web proxies for conditions data distributed through the Frontier Distributed Database caching system. ATLAS & CMS each have their own methods for their grid jobs to find out which web proxies to use for Frontier at each site, and CVMFS has a third method. Those diverse methods limit usability and flexibility, particularly for opportunistic use cases, where an experiment’s jobs are run at sites that do not primarily support that experiment. This paper describes a new Worldwide LHC Computing Grid (WLCG) system for discovering the addresses of web proxies. The system is based on an internet standard called Web Proxy Auto Discovery (WPAD). WPAD is in turn based on another standard called Proxy Auto Configuration (PAC). Both the Frontier and CVMFS clients support this standard. The input into the WLCG system comes from squids registered in the ATLAS Grid Information System (AGIS) and CMS SITECONF files, cross-checked with squids registered by sites in the Grid Configuration Database (GOCDB) and the OSG Information Management (OIM) system, and combined with some exceptions manually configured by people from ATLAS and CMS who operate WLCG Squid monitoring. WPAD servers at CERN respond to http requests from grid nodes all over the world with a PAC file that lists available web proxies, based on IP addresses matched from a database that contains the IP address ranges registered to organizations. Large grid sites are encouraged to supply their own WPAD web servers for more flexibility, to avoid being affected by short term long distance network outages, and to offload the WLCG WPAD servers at CERN. The CERN WPAD servers additionally support requests from jobs running at non-grid sites (particularly for LHC@Home) which it directs to the nearest publicly accessible web proxy servers. Furthermore, the responses to those requests are geographically ordered based on a separate database that maps IP addresses to longitude and latitude.},
doi = {10.1088/1742-6596/898/5/052043},
url = {https://www.osti.gov/biblio/1346930}, journal = {Journal of Physics. Conference Series},
issn = {1742-6588},
number = ,
volume = 898,
place = {United States},
year = {Thu Nov 23 00:00:00 EST 2017},
month = {Thu Nov 23 00:00:00 EST 2017}
}