FCP: A Fast and Scalable Data Copy Tool for High Performance Parallel File Systems
The design of HPC file and storage systems has largely been driven by the requirements on capability, reliability, and capacity. However, the convergence of large-scale simulations with big data analytics have put the data, its usability, and management back on the front and center position. One of the most common and time consuming data management tasks is the transfer of very large datasets within and across file systems.In this paper, we are introducing the FCP tool, a file system agnostic copy tool designed at the OLCF for scalable and high-performance data transfers between two file system endpoints. It provides an array of interesting features such as adaptive chunking, checksumming on the fly, checkpoint and resume capabilities to handle failures, and preserving stripe information for Lustre file system among others. It is currently available on the Titan supercomputer at the OLCF. Initial tests have shown that FCP has much better and scalable performance than traditional data copy tools and it was capable of transferring petabyte-scale datasets between two Lustre file systems.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1648987
- Resource Relation:
- Conference: Cray User Group (CUG) Conference Proceedings - London, , United Kingdom - 5/9/2016 12:00:00 PM-5/12/2016 4:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
Lessons Learned in Deploying the World s Largest Scale Lustre File System
Diving into petascale production file systems through large scale profiling and analysis