skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: High Performance Data Transfer for Distributed Data Intensive Sciences

Abstract

We report on the development of ZX software providing high performance data transfer and encryption. The design scales in: computation power, network interfaces, and IOPS while carefully balancing the available resources. Two U.S. patent-pending algorithms help tackle data sets containing lots of small files and very large files, and provide insensitivity to network latency. It has a cluster-oriented architecture, using peer-to-peer technologies to ease deployment, operation, usage, and resource discovery. Its unique optimizations enable effective use of flash memory. Using a pair of existing data transfer nodes at SLAC and NERSC, we compared its performance to that of bbcp and GridFTP and determined that they were comparable. With a proof of concept created using two four-node clusters with multiple distributed multi-core CPUs, network interfaces and flash memory, we achieved 155Gbps memory-to-memory over a 2x100Gbps link aggregated channel and 70Gbps file-to-file with encryption over a 5000 mile 100Gbps link.

Authors:
 [1];  [2];  [2];  [2];  [2]
  1. Zettar Inc., Mountain View, CA (United States)
  2. SLAC National Accelerator Lab., Menlo Park, CA (United States)
Publication Date:
Research Org.:
SLAC National Accelerator Lab., Menlo Park, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1346534
Report Number(s):
SLAC-TN-16-001
DOE Contract Number:  
AC02-76SF00515
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Fang, Chin, Cottrell, R 'Les' A., Hanushevsky, Andrew B., Kroeger, Wilko, and Yang, Wei. High Performance Data Transfer for Distributed Data Intensive Sciences. United States: N. p., 2017. Web. doi:10.2172/1346534.
Fang, Chin, Cottrell, R 'Les' A., Hanushevsky, Andrew B., Kroeger, Wilko, & Yang, Wei. High Performance Data Transfer for Distributed Data Intensive Sciences. United States. doi:10.2172/1346534.
Fang, Chin, Cottrell, R 'Les' A., Hanushevsky, Andrew B., Kroeger, Wilko, and Yang, Wei. Mon . "High Performance Data Transfer for Distributed Data Intensive Sciences". United States. doi:10.2172/1346534. https://www.osti.gov/servlets/purl/1346534.
@article{osti_1346534,
title = {High Performance Data Transfer for Distributed Data Intensive Sciences},
author = {Fang, Chin and Cottrell, R 'Les' A. and Hanushevsky, Andrew B. and Kroeger, Wilko and Yang, Wei},
abstractNote = {We report on the development of ZX software providing high performance data transfer and encryption. The design scales in: computation power, network interfaces, and IOPS while carefully balancing the available resources. Two U.S. patent-pending algorithms help tackle data sets containing lots of small files and very large files, and provide insensitivity to network latency. It has a cluster-oriented architecture, using peer-to-peer technologies to ease deployment, operation, usage, and resource discovery. Its unique optimizations enable effective use of flash memory. Using a pair of existing data transfer nodes at SLAC and NERSC, we compared its performance to that of bbcp and GridFTP and determined that they were comparable. With a proof of concept created using two four-node clusters with multiple distributed multi-core CPUs, network interfaces and flash memory, we achieved 155Gbps memory-to-memory over a 2x100Gbps link aggregated channel and 70Gbps file-to-file with encryption over a 5000 mile 100Gbps link.},
doi = {10.2172/1346534},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Mon Mar 06 00:00:00 EST 2017},
month = {Mon Mar 06 00:00:00 EST 2017}
}

Technical Report:

Save / Share: