Arra: Tas::89 0227::Tas Recovery Act 100g Ftp: An Ultra-High Speed Data Transfer Service Over Next Generation 100 Gigabit Per Second Network
- Brookhaven National Lab/Stony Brook University
- Stony Brook University
Data-intensive applications, including high energy and nuclear physics, astrophysics, climate modeling, nano-scale materials science, genomics, and financing, are expected to generate exabytes of data over the coming years, which must be transferred, visualized, and analyzed by geographically distributed teams of users. High-performance network capabilities must be available to these users at the application level in a transparent, virtualized manner. Moreover, the application users must have the capability to move large datasets from local and remote locations across network environments to their home institutions. To solve these challenges, the main goal of our project is to design and evaluate high-performance data transfer software to support various data-intensive applications. First, we have designed a middleware software that provides access to Remote Direct Memory Access (RDMA) functionalities. This middleware integrates network access, memory management and multitasking in its core design. We address a number of issues related to its efficient implementation, for instance, explicit buffer management and memory registration, and parallelization of RDMA operations, which are vital to delivering the benefit of RDMA to the applications. Built on top of this middleware, an implementation and experimental evaluation of the RDMA-based FTP software, RFTP, is described and evaluated. This application has been implemented by our team to exploit the full capabilities of advanced RDMA mechanisms for ultra-high speed bulk data transfer applications on Energy Sciences Network (ESnet). Second, we designed our data transfer software to optimize TCP/IP based data transfer performance such that RFTP can be fully compatible with today’s Internet. Our kernel optimization techniques with Linux system calls sendfile and splice, can reduce data copy cost. In this report, we summarize the technical challenges of our project, the primary software design methods, the major project milestones achieved, as well as the testbed evaluation work and demonstrations during our project life time.
- Research Organization:
- Stony Brook Univ., NY (United States)
- Sponsoring Organization:
- USDOE; USDOE SC Office of Advanced Scientific Computing Research (SC-21)
- Contributing Organization:
- Brookhaven National Laboratory
- DOE Contract Number:
- SC0003361
- OSTI ID:
- 1121761
- Report Number(s):
- DOE-STONYBROOK-3361
- Country of Publication:
- United States
- Language:
- English
Similar Records
A Path to Operating System and Runtime Support for Extreme Scale Tools
Final report