RoCE: Promising Technology for Ethernet as a High Performance Networking Fabric

Technical Report · Thu Nov 07 04:00:00 EST 2019

DOI:https://doi.org/10.2172/1573446· OSTI ID:1573446

Kenny, Joseph P. ^[1]; Ulmer, Craig D. ^[1]

---

1. Sandia National Laboratories (SNL-CA), Livermore, CA (United States)

Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) has the potential to provide performance that rivals traditional high performance fabrics. If this potential proves out, significant impacts on system procurement decisions could follow. This work provides a series of small scale performance results which are used to compare and contrast the performance of RoCE-enabled Ethernet with TCP-based Ethernet and an HPC network. Additionally, a discussion of the maturity of RoCE firmware/software stacks and documentation is provided along with useful approaches for probing performance. A detailed description of two experimental setups known to have good RoCE performance is given, including step-by-step configuration and the exact hardware and software revisions employed. At small scales, RoCE is found to have significant performance advantages over "out-of-the-box" TCP protocols and is competitive with state-of-the-art high performance networks. Further examination of RoCE using a wider array of benchmarks and at greater scale is warranted.

• Research Organization: Sandia National Laboratories (SNL-CA), Livermore, CA (United States)
• Sponsoring Organization: USDOE National Nuclear Security Administration (NNSA)
• DOE Contract Number: AC04-94AL85000
• OSTI ID: 1573446
• Report Number(s): SAND--2019-13444; 681101
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING