High Throughput WAN Data Transfer with Hadoop-Based Storage
- Caltech
- Nebraska U.
- UC, San Diego
- Fermilab
Hadoop distributed file system (HDFS) is becoming more popular in recent years as a key building block of integrated grid storage solution in the field of scientific computing. Wide Area Network (WAN) data transfer is one of the important data operations for large high energy physics experiments to manage, share and process datasets of PetaBytes scale in a highly distributed grid computing environment. In this paper, we present the experience of high throughput WAN data transfer with HDFS-based Storage Element. Two protocols, GridFTP and fast data transfer (FDT), are used to characterize the network performance of WAN data transfer.
- Research Organization:
- Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
- DOE Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1433336
- Report Number(s):
- FERMILAB-CONF-10-715-CD; 1111550
- Conference Information:
- Journal Name: J.Phys.Conf.Ser. Journal Volume: 331
- Country of Publication:
- United States
- Language:
- English
Similar Records
XRootD popularity on hadoop clusters
Throughput Analytics of Data Transfer Infrastructures
Applied techniques for high bandwidth data transfers across wide area networks
Journal Article
·
Tue Nov 21 19:00:00 EST 2017
· Journal of Physics. Conference Series
·
OSTI ID:1831862
Throughput Analytics of Data Transfer Infrastructures
Conference
·
Thu Jan 31 23:00:00 EST 2019
·
OSTI ID:1509526
Applied techniques for high bandwidth data transfers across wide area networks
Conference
·
Mon Apr 30 00:00:00 EDT 2001
·
OSTI ID:789141