mdtmFTP and its evaluation on ESNET SDN testbed

Journal Article · Fri Apr 21 00:00:00 EDT 2017 · Future Generations Computer Systems

DOI:https://doi.org/10.1016/j.future.2017.04.024· OSTI ID:1358099

Zhang, Liang ^[1]; Wu, Wenji ^[1]; DeMar, Phil ^[1]; Pouyoul, Eric ^[2]

---

1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
2. ESnet, Berkeley, CA (United States)

In this paper, to address the high-performance challenges of data transfer in the big data era, we are developing and implementing mdtmFTP: a high-performance data transfer tool for big data. mdtmFTP has four salient features. First, it adopts an I/O centric architecture to execute data transfer tasks. Second, it more efficiently utilizes the underlying multicore platform through optimized thread scheduling. Third, it implements a large virtual file mechanism to address the lots-of-small-files (LOSF) problem. In conclusion, mdtmFTP integrates multiple optimization mechanisms, including–zero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer pools–to enhance performance. mdtmFTP has been extensively tested and evaluated within the ESNET 100G testbed. Evaluations show that mdtmFTP can achieve higher performance than existing data transfer tools, such as GridFTP, FDT, and BBCP.

• Research Organization: ESnet, Berkeley, CA (United States); Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
• Grant/Contract Number: AC02-07CH11359
• OSTI ID: 1358099
• Report Number(s): FERMILAB-CONF--17-123-CD; 1599032
• Journal Information: Future Generations Computer Systems, Journal Name: Future Generations Computer Systems Journal Issue: 1 Vol. 79; ISSN 0167-739X
• Publisher: Elsevier
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Data transfer
High-speed networking
Multicore