Big Data over a 100G network at Fermilab
Abstract
As the need for Big Data in science becomes ever more relevant, networks around the world are upgrading their infrastructure to support high-speed interconnections. To support its mission, the high-energy physics community as a pioneer in Big Data has always been relying on the Fermi National Accelerator Laboratory to be at the forefront of storage and data movement. This need was reiterated in recent years with the data-taking rate of the major LHC experiments reaching tens of petabytes per year. At Fermilab, this resulted regularly in peaks of data movement on the Wide area network (WAN) in and out of the laboratory of about 30 Gbit/s and on the Local are network (LAN) between storage and computational farms of 160 Gbit/s. To address these ever increasing needs, as of this year Fermilab is connected to the Energy Sciences Network (ESnet) through a 100 Gb/s link. To understand the optimal system-and application-level configuration to interface computational systems with the new highspeed interconnect, Fermilab has deployed a Network Research & Development facility connected to the ESnet 100G Testbed. For the past two years, the High Throughput Data Program (HTDP) has been using the Testbed to identify gaps in data movement middleware [5]more »
- Authors:
-
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Publication Date:
- Research Org.:
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), High Energy Physics (HEP)
- OSTI Identifier:
- 1296591
- Report Number(s):
- FERMILAB-PUB-13-486-CD
Journal ID: ISSN 1742-6588; 1302177
- Grant/Contract Number:
- AC02-07CH11359
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Journal of Physics. Conference Series
- Additional Journal Information:
- Journal Volume: 513; Journal Issue: 6; Journal ID: ISSN 1742-6588
- Publisher:
- IOP Publishing
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
Citation Formats
Garzoglio, Gabriele, Mhashilkar, Parag, Kim, Hyunwoo, Dykstra, Dave, and Slyz, Marko. Big Data over a 100G network at Fermilab. United States: N. p., 2014.
Web. doi:10.1088/1742-6596/513/6/062017.
Garzoglio, Gabriele, Mhashilkar, Parag, Kim, Hyunwoo, Dykstra, Dave, & Slyz, Marko. Big Data over a 100G network at Fermilab. United States. https://doi.org/10.1088/1742-6596/513/6/062017
Garzoglio, Gabriele, Mhashilkar, Parag, Kim, Hyunwoo, Dykstra, Dave, and Slyz, Marko. Wed .
"Big Data over a 100G network at Fermilab". United States. https://doi.org/10.1088/1742-6596/513/6/062017. https://www.osti.gov/servlets/purl/1296591.
@article{osti_1296591,
title = {Big Data over a 100G network at Fermilab},
author = {Garzoglio, Gabriele and Mhashilkar, Parag and Kim, Hyunwoo and Dykstra, Dave and Slyz, Marko},
abstractNote = {As the need for Big Data in science becomes ever more relevant, networks around the world are upgrading their infrastructure to support high-speed interconnections. To support its mission, the high-energy physics community as a pioneer in Big Data has always been relying on the Fermi National Accelerator Laboratory to be at the forefront of storage and data movement. This need was reiterated in recent years with the data-taking rate of the major LHC experiments reaching tens of petabytes per year. At Fermilab, this resulted regularly in peaks of data movement on the Wide area network (WAN) in and out of the laboratory of about 30 Gbit/s and on the Local are network (LAN) between storage and computational farms of 160 Gbit/s. To address these ever increasing needs, as of this year Fermilab is connected to the Energy Sciences Network (ESnet) through a 100 Gb/s link. To understand the optimal system-and application-level configuration to interface computational systems with the new highspeed interconnect, Fermilab has deployed a Network Research & Development facility connected to the ESnet 100G Testbed. For the past two years, the High Throughput Data Program (HTDP) has been using the Testbed to identify gaps in data movement middleware [5] when transferring data at these high-speeds. The program has published evaluations of technologies typically used in High Energy Physics, such as GridFTP [4], XrootD [9], and Squid [8]. Furthermore, this work presents the new R&D facility and the continuation of the evaluation program.},
doi = {10.1088/1742-6596/513/6/062017},
journal = {Journal of Physics. Conference Series},
number = 6,
volume = 513,
place = {United States},
year = {Wed Jun 11 00:00:00 EDT 2014},
month = {Wed Jun 11 00:00:00 EDT 2014}
}
Search WorldCat to find libraries that may hold this journalWeb of Science