skip to main content

DOE PAGESDOE PAGES

Title: mdtmFTP and its evaluation on ESNET SDN testbed

In this paper, to address the high-performance challenges of data transfer in the big data era, we are developing and implementing mdtmFTP: a high-performance data transfer tool for big data. mdtmFTP has four salient features. First, it adopts an I/O centric architecture to execute data transfer tasks. Second, it more efficiently utilizes the underlying multicore platform through optimized thread scheduling. Third, it implements a large virtual file mechanism to address the lots-of-small-files (LOSF) problem. In conclusion, mdtmFTP integrates multiple optimization mechanisms, including–zero copy, asynchronous I/O, pipelining, batch processing, and pre-allocated buffer pools–to enhance performance. mdtmFTP has been extensively tested and evaluated within the ESNET 100G testbed. Evaluations show that mdtmFTP can achieve higher performance than existing data transfer tools, such as GridFTP, FDT, and BBCP.
Authors:
ORCiD logo [1] ;  [1] ;  [1] ;  [2]
  1. Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
  2. ESnet, Berkeley, CA (United States)
Publication Date:
Report Number(s):
FERMILAB-CONF-17-123-CD
Journal ID: ISSN 0167-739X; 1599032
Grant/Contract Number:
AC02-07CH11359
Type:
Accepted Manuscript
Journal Name:
Future Generations Computer Systems
Additional Journal Information:
Journal Volume: 79; Journal Issue: 1; Journal ID: ISSN 0167-739X
Publisher:
Elsevier
Research Org:
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); ESnet, Berkeley, CA (United States)
Sponsoring Org:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Multicore; Data transfer; High-speed networking
OSTI Identifier:
1358099