skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Wide-area lustre file system using LNet routers

Abstract

Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]
  1. ORNL
Publication Date:
Research Org.:
Oak Ridge National Laboratory, Oak Ridge Leadership Computing Facility (OLCF); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1468050
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: 12th Annual IEEE International Systems Conference (SYSCON2018) - Vancouver, , Canada - 4/23/2018 4:00:00 AM-4/26/2018 4:00:00 AM
Country of Publication:
United States
Language:
English

Citation Formats

Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., and Oral, H Sarp. Wide-area lustre file system using LNet routers. United States: N. p., 2018. Web. doi:10.1109/SYSCON.2018.8369541.
Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., & Oral, H Sarp. Wide-area lustre file system using LNet routers. United States. doi:10.1109/SYSCON.2018.8369541.
Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., and Oral, H Sarp. Sun . "Wide-area lustre file system using LNet routers". United States. doi:10.1109/SYSCON.2018.8369541. https://www.osti.gov/servlets/purl/1468050.
@article{osti_1468050,
title = {Wide-area lustre file system using LNet routers},
author = {Rao, Nageswara S. and Imam, Neena and Hanley, Jesse A. and Oral, H Sarp},
abstractNote = {Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.},
doi = {10.1109/SYSCON.2018.8369541},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {4}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: