skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: mdtmFTP and its evaluation on ESNET SDN testbed

Abstract

In this paper, to address the high-performance challenges of data transfer in the big data era, we are developing and implementing mdtmFTP: a high-performance data transfer tool for big data. mdtmFTP has four salient features. First, it adopts an I/O centric architecture to execute data transfer tasks. Second, it more efficiently utilizes the underlying multicore platform through optimized thread scheduling. Third, it implements a large virtual file mechanism to address the lots-of-small-files (LOSF) problem. In conclusion, mdtmFTP integrates multiple optimization mechanisms, including–zero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer pools–to enhance performance. mdtmFTP has been extensively tested and evaluated within the ESNET 100G testbed. Evaluations show that mdtmFTP can achieve higher performance than existing data transfer tools, such as GridFTP, FDT, and BBCP.

Authors:
ORCiD logo [1];  [1];  [1];  [2]
  1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  2. ESnet, Berkeley, CA (United States)
Publication Date:
Research Org.:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); ESnet, Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1358099
Report Number(s):
FERMILAB-CONF-17-123-CD
Journal ID: ISSN 0167-739X; 1599032
Grant/Contract Number:  
AC02-07CH11359
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Future Generations Computer Systems
Additional Journal Information:
Journal Volume: 79; Journal Issue: 1; Journal ID: ISSN 0167-739X
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Multicore; Data transfer; High-speed networking

Citation Formats

Zhang, Liang, Wu, Wenji, DeMar, Phil, and Pouyoul, Eric. mdtmFTP and its evaluation on ESNET SDN testbed. United States: N. p., 2017. Web. doi:10.1016/j.future.2017.04.024.
Zhang, Liang, Wu, Wenji, DeMar, Phil, & Pouyoul, Eric. mdtmFTP and its evaluation on ESNET SDN testbed. United States. doi:10.1016/j.future.2017.04.024.
Zhang, Liang, Wu, Wenji, DeMar, Phil, and Pouyoul, Eric. Fri . "mdtmFTP and its evaluation on ESNET SDN testbed". United States. doi:10.1016/j.future.2017.04.024. https://www.osti.gov/servlets/purl/1358099.
@article{osti_1358099,
title = {mdtmFTP and its evaluation on ESNET SDN testbed},
author = {Zhang, Liang and Wu, Wenji and DeMar, Phil and Pouyoul, Eric},
abstractNote = {In this paper, to address the high-performance challenges of data transfer in the big data era, we are developing and implementing mdtmFTP: a high-performance data transfer tool for big data. mdtmFTP has four salient features. First, it adopts an I/O centric architecture to execute data transfer tasks. Second, it more efficiently utilizes the underlying multicore platform through optimized thread scheduling. Third, it implements a large virtual file mechanism to address the lots-of-small-files (LOSF) problem. In conclusion, mdtmFTP integrates multiple optimization mechanisms, including–zero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer pools–to enhance performance. mdtmFTP has been extensively tested and evaluated within the ESNET 100G testbed. Evaluations show that mdtmFTP can achieve higher performance than existing data transfer tools, such as GridFTP, FDT, and BBCP.},
doi = {10.1016/j.future.2017.04.024},
journal = {Future Generations Computer Systems},
number = 1,
volume = 79,
place = {United States},
year = {Fri Apr 21 00:00:00 EDT 2017},
month = {Fri Apr 21 00:00:00 EDT 2017}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 2 works
Citation information provided by
Web of Science

Save / Share: