Wide-area lustre file system using LNet routers

Conference · Sun Apr 01 00:00:00 EDT 2018

DOI:https://doi.org/10.1109/SYSCON.2018.8369541· OSTI ID:1468050

Rao, Nageswara  ^[1]; Imam, Neena  ^[1]; Hanley, Jesse  ^[1]; Oral, Sarp  ^[1]

---

1. ORNL

Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1468050
• Resource Relation: Conference: 12th Annual IEEE International Systems Conference (SYSCON2018) - Vancouver, , Canada - 4/23/2018 4:00:00 AM-4/26/2018 4:00:00 AM
• Country of Publication: United States
• Language: English