Wide-area lustre file system using LNet routers

doi:https://doi.org/10.1109/SYSCON.2018.8369541

Title: Wide-area lustre file system using LNet routers

Full Record
Other Related Research

Abstract

Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.

Authors:

^[1];

^[1]

ORNL

Publication Date:: 2018-04-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

OSTI Identifier:: 1468050

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: 12th Annual IEEE International Systems Conference (SYSCON2018) - Vancouver, , Canada - 4/23/2018 4:00:00 AM-4/26/2018 4:00:00 AM

Country of Publication:: United States

Language:: English

Citation Formats


                    Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., and Oral, H Sarp. Wide-area lustre file system using LNet routers.  United States: N. p., 2018. 
        Web.  doi:10.1109/SYSCON.2018.8369541.

Copy to clipboard


                    Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., & Oral, H Sarp. Wide-area lustre file system using LNet routers.  United States.  https://doi.org/10.1109/SYSCON.2018.8369541

Copy to clipboard


                    Rao, Nageswara S., Imam, Neena, Hanley, Jesse A., and Oral, H Sarp. Sun .  
        "Wide-area lustre file system using LNet routers".  United States.  https://doi.org/10.1109/SYSCON.2018.8369541.  https://www.osti.gov/servlets/purl/1468050.

Copy to clipboard


                    
@article{osti_1468050,

  title        = {Wide-area lustre file system using LNet routers},

  author       = {Rao, Nageswara S. and Imam, Neena and Hanley, Jesse A. and Oral, H Sarp},

  abstractNote = {Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.},

  doi          = {10.1109/SYSCON.2018.8369541},

  url          = {https://www.osti.gov/biblio/1468050},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {4}

}

Copy to clipboard

Conference:

View Conference (0.68 MB)

https://doi.org/10.1109/SYSCON.2018.8369541

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Measurements and analytics of wide-area file transfers over dedicated connections

Conference Rao, Nageswara S. ; Liu, Qiang ; Sen, Satyabrata ; ...

Distributed scientific and big-data computations are becoming increasingly dependent on access to remote files. Wide-area file transfers are supported by two basic schemes: (i) application-level tools, such as GridFTP, that provide transport services between file systems housed at geographically separated sites, and (ii) file systems mounted over wide-area networks, using mechanisms such as LNet routers that make them transparently available. In both cases, the file transfer performance critically depends on the configuration consisting of host, file, IO, and disk subsystems, which are complex by themselves, as well as on their complex compositions implemented using buffers and IO-network data transitions. Wemore »« less
https://doi.org/10.1145/3288599.3288641

Full Text Available
Learning concave-convex profiles of data transport over dedicated connections

Conference Rao, Nageswara S. ; Sen, Satyabrata ; Liu, Zhengchun ; ...

Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or creditmore »« less
https://doi.org/10.1007/978-3-030-19945-6_1

Full Text Available
Learning Concave-Convex Profiles of Data Transport Over Dedicated Connections

Conference Rao, Nageswara S. V. ; Sen, Satyabrata ; Liu, Zhengchun ; ...

Dedicated data transport infrastructures are increasingly being deployed to support distributed big-data and high-performance computing scenarios. These infrastructures employ data transfer nodes that use sophisticated software stacks to support network transport among sites, which often house distributed file and storage systems. Throughput measurements collected over such infrastructures for a range of round trip times (RTTs) reflect the underlying complex end-to-end connections, and have revealed dichotomous throughput profiles as functions of RTT. In particular, concave regions of throughput profiles at lower RTTs indicate near-optimal performance, and convex regions at higher RTTs indicate bottlenecks due to factors such as buffer or creditmore »« less
https://doi.org/10.1007/978-3-030-19945-6_1
On Data Transfers Over Wide-Area Dedicated Connections

Book Rao, Nageswara S. ; Liu, Qiang

Dedicated wide-area network connections are employed in big data and high-performance computing scenarios, since the absence of cross-traffic promises to make it easier to analyze and optimize data transfers over them. However, nonlinear transport dynamics and end-system complexity due to multi-core hosts and distributed file systems make these tasks surprisingly challenging. We present an overview of methods to analyze memory and disk file transfers using extensive measurements over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory transfers, we derive performance profiles of TCP and UDT throughput as a function of RTT, which showmore »« less
Experiments and Analyses of Data Transfers Over Wide-Area Dedicated Connections

Conference Rao, Nageswara S. ; Liu, Qiang ; Sen, Satyabrata ; ...

Dedicated wide-area network connections are increasingly employed in high-performance computing and big data scenarios. One might expect the performance and dynamics of data transfers over such connections to be easy to analyze due to the lack of competing traffic. However, non-linear transport dynamics and end-system complexities (e.g., multi-core hosts and distributed filesystems) can in fact make analysis surprisingly challenging. We present extensive measurements of memory-to-memory and disk-to-disk file transfers over 10 Gbps physical and emulated connections with 0–366 ms round trip times (RTTs). For memory-to-memory transfers, profiles of both TCP and UDT throughput as a function of RTT show concavemore »« less
https://doi.org/10.1109/ICCCN.2017.8038432

Full Text Available

Similar Records