mdtmFTP and its evaluation on ESNET SDN testbed
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- ESnet, Berkeley, CA (United States)
In this paper, to address the high-performance challenges of data transfer in the big data era, we are developing and implementing mdtmFTP: a high-performance data transfer tool for big data. mdtmFTP has four salient features. First, it adopts an I/O centric architecture to execute data transfer tasks. Second, it more efficiently utilizes the underlying multicore platform through optimized thread scheduling. Third, it implements a large virtual file mechanism to address the lots-of-small-files (LOSF) problem. In conclusion, mdtmFTP integrates multiple optimization mechanisms, including–zero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer pools–to enhance performance. mdtmFTP has been extensively tested and evaluated within the ESNET 100G testbed. Evaluations show that mdtmFTP can achieve higher performance than existing data transfer tools, such as GridFTP, FDT, and BBCP.
- Research Organization:
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); ESnet, Berkeley, CA (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
- Grant/Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1358099
- Report Number(s):
- FERMILAB-CONF--17-123-CD; 1599032
- Journal Information:
- Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Journal Issue: 1 Vol. 79; ISSN 0167-739X
- Publisher:
- ElsevierCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Similar Records
Comparative Performance Evaluation of High-performance Data Transfer Tools
Comparative Performance Evaluation of High-performance Data Transfer Tools