Wide-area lustre file system using LNet routers
Abstract
Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.
- Authors:
-
- ORNL
- Publication Date:
- Research Org.:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
- Sponsoring Org.:
- USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
- OSTI Identifier:
- 1468050
- DOE Contract Number:
- AC05-00OR22725
- Resource Type:
- Conference
- Resource Relation:
- Conference: 12th Annual IEEE International Systems Conference (SYSCON2018) - Vancouver, , Canada - 4/23/2018 4:00:00 AM-4/26/2018 4:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Citation Formats
Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., and Oral, H Sarp. Wide-area lustre file system using LNet routers. United States: N. p., 2018.
Web. doi:10.1109/SYSCON.2018.8369541.
Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., & Oral, H Sarp. Wide-area lustre file system using LNet routers. United States. https://doi.org/10.1109/SYSCON.2018.8369541
Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., and Oral, H Sarp. Sun .
"Wide-area lustre file system using LNet routers". United States. https://doi.org/10.1109/SYSCON.2018.8369541. https://www.osti.gov/servlets/purl/1468050.
@article{osti_1468050,
title = {Wide-area lustre file system using LNet routers},
author = {Rao, Nageswara S. and Imam, Neena and Hanley, Jesse A. and Oral, H Sarp},
abstractNote = {Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.},
doi = {10.1109/SYSCON.2018.8369541},
url = {https://www.osti.gov/biblio/1468050},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {4}
}