mdtmFTP and its evaluation on ESNET SDN testbed
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
- ESnet, Berkeley, CA (United States)
In this paper, to address the high-performance challenges of data transfer in the big data era, we are developing and implementing mdtmFTP: a high-performance data transfer tool for big data. mdtmFTP has four salient features. First, it adopts an I/O centric architecture to execute data transfer tasks. Second, it more efficiently utilizes the underlying multicore platform through optimized thread scheduling. Third, it implements a large virtual file mechanism to address the lots-of-small-files (LOSF) problem. In conclusion, mdtmFTP integrates multiple optimization mechanisms, including–zero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer pools–to enhance performance. mdtmFTP has been extensively tested and evaluated within the ESNET 100G testbed. Evaluations show that mdtmFTP can achieve higher performance than existing data transfer tools, such as GridFTP, FDT, and BBCP.
- Research Organization:
- Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); ESnet, Berkeley, CA (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- Grant/Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1358099
- Report Number(s):
- FERMILAB-CONF-17-123-CD; 1599032
- Journal Information:
- Future Generations Computer Systems, Vol. 79, Issue 1; ISSN 0167-739X
- Publisher:
- ElsevierCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Web of Science
Similar Records
Comparative Performance Evaluation of High-performance Data Transfer Tools
Continuing Efforts on Efficiency and Performance Improvement of mdtmFTP