MDTM: Optimizing Data Transfer Using Multicore-Aware I/O Scheduling

Journal Article · Sun Oct 01 00:00:00 EDT 2017 · 2017 IEEE 42nd Conference on Local Computer Networks (LCN)

DOI:https://doi.org/10.1109/LCN.2017.64· OSTI ID:1394792

Zhang, Liang; Demar, Phil; Kim, Bockjoo; Wu, Wenji

Bulk data transfer is facing significant challenges in the coming era of big data. There are multiple performance bottlenecks along the end-to-end path from the source to destination storage system. The limitations of current generation data transfer tools themselves can have a significant impact on end-to-end data transfer rates. In this paper, we identify the issues that lead to underperformance of these tools, and present a new data transfer tool with an innovative I/O scheduler called MDTM. The MDTM scheduler exploits underlying multicore layouts to optimize throughput by reducing delay and contention for I/O reading and writing operations. With our evaluations, we show how MDTM successfully avoids NUMA-based congestion and significantly improves end-to-end data transfer rates across high-speed wide area networks.

• Research Organization: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
• Sponsoring Organization: USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
• DOE Contract Number: AC02-07CH11359
• OSTI ID: 1394792
• Report Number(s): FERMILAB-PUB-17-151-CD; 1624636
• Journal Information: 2017 IEEE 42nd Conference on Local Computer Networks (LCN), Journal Name: 2017 IEEE 42nd Conference on Local Computer Networks (LCN)
• Country of Publication: United States
• Language: English