A Technique for Moving Large Data Sets over High-Performance Long Distance Networks
- ORNL
In this paper we look at the performance characteristics of three tools used to move large data sets over dedicated long distance networking infrastructure. Although performance studies of wide area networks have been a frequent topic of interest, performance analyses have tended to focus on network latency characteristics and peak throughput using network traffic generators. In this study we instead perform an end-to-end long distance networking analysis that includes reading large data sets from a source file system and committing the data to a remote destination file system. An evaluation of end-to-end data movement is also an evaluation of the system configurations employed and the tools used to move the data. For this paper, we have built several storage platforms and connected them with a high performance long distance network configuration. We use these systems to analyze the capabilities of three data movement tools: BBcp, GridFTP, and XDD. Our studies demonstrate that existing data movement tools do not provide efficient performance levels or exercise the storage devices in their highest performance modes.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- Work for Others (WFO)
- DOE Contract Number:
- DE-AC05-00OR22725
- OSTI ID:
- 1015103
- Resource Relation:
- Conference: MSST 2011: Mass Storage Systems and Technology Conference, Denver, CO, USA, 20110523, 20110527
- Country of Publication:
- United States
- Language:
- English
Similar Records
Comparative Performance Evaluation of High-performance Data Transfer Tools
Comparative Performance Evaluation of High-performance Data Transfer Tools