High Performance Data Transfer for Distributed Data Intensive Sciences
- Zettar Inc., Mountain View, CA (United States)
- SLAC National Accelerator Lab., Menlo Park, CA (United States)
We report on the development of ZX software providing high performance data transfer and encryption. The design scales in: computation power, network interfaces, and IOPS while carefully balancing the available resources. Two U.S. patent-pending algorithms help tackle data sets containing lots of small files and very large files, and provide insensitivity to network latency. It has a cluster-oriented architecture, using peer-to-peer technologies to ease deployment, operation, usage, and resource discovery. Its unique optimizations enable effective use of flash memory. Using a pair of existing data transfer nodes at SLAC and NERSC, we compared its performance to that of bbcp and GridFTP and determined that they were comparable. With a proof of concept created using two four-node clusters with multiple distributed multi-core CPUs, network interfaces and flash memory, we achieved 155Gbps memory-to-memory over a 2x100Gbps link aggregated channel and 70Gbps file-to-file with encryption over a 5000 mile 100Gbps link.
- Research Organization:
- SLAC National Accelerator Lab., Menlo Park, CA (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- AC02-76SF00515
- OSTI ID:
- 1346534
- Report Number(s):
- SLAC-TN-16-001
- Country of Publication:
- United States
- Language:
- English
Similar Records
Harnessing multicore processors for high-speed secure transfer.
NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure