skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: InfiniBand Performance Characterization in Virtual Machines and Containers

Abstract

This presentation reports on the results of InfiniBand performance and characterization. This work came about because of a demand for user-defined software stacks within supercomputing centers.

Authors:
 [1];  [1];  [1]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1296652
Report Number(s):
LA-UR-16-26124
DOE Contract Number:
AC52-06NA25396
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Computer Hardware; Computer Science

Citation Formats

Smith, Adam Nathaniel, Ogas, Jordan Andrew, and Easterday, Hunter Patrick. InfiniBand Performance Characterization in Virtual Machines and Containers. United States: N. p., 2016. Web. doi:10.2172/1296652.
Smith, Adam Nathaniel, Ogas, Jordan Andrew, & Easterday, Hunter Patrick. InfiniBand Performance Characterization in Virtual Machines and Containers. United States. doi:10.2172/1296652.
Smith, Adam Nathaniel, Ogas, Jordan Andrew, and Easterday, Hunter Patrick. Mon . "InfiniBand Performance Characterization in Virtual Machines and Containers". United States. doi:10.2172/1296652. https://www.osti.gov/servlets/purl/1296652.
@article{osti_1296652,
title = {InfiniBand Performance Characterization in Virtual Machines and Containers},
author = {Smith, Adam Nathaniel and Ogas, Jordan Andrew and Easterday, Hunter Patrick},
abstractNote = {This presentation reports on the results of InfiniBand performance and characterization. This work came about because of a demand for user-defined software stacks within supercomputing centers.},
doi = {10.2172/1296652},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Mon Aug 08 00:00:00 EDT 2016},
month = {Mon Aug 08 00:00:00 EDT 2016}
}

Technical Report:

Save / Share:
  • A look at the performance of the infiniband interconnect using the Voltaire host stack. This will attempt to compare not only infiniband to other high-performance interconnects, but will also take a look at comparing some of the different hardware choices available at the time of writing (e.g. Opteron, EM64T, pci-express and pci-x).
  • This technical report will be comparing the performance between the most common infiniband-related technologies currently available. Included will be TCP-based, MPI-based and low-level performance tests to see what performance can be expected from Mellanox's SDR and DDR as well as PathScale's Infinipath. Also, we will be performing comparisons of the Infinipath on both OpenIB as well as PathScale's ipath stack. Infiniband promises to bring high performance interconnects for I/O (filesystem and networking) to a new cost performance level. Thus, LLNL has been evaluating Infiniband for use as a cluster interconnect. Various issues impact the decision of which interconnect to usemore » in a cluster. This technical report will be looking more closely at the actual performance of the major infiniband technologies present today. Performance testing will focus on latency, and bandwidth (both uni and bi-directional) using both TCP and MPI. In addition, we will be looking at an even lower-level (removing most of the upper-level protocols) and seeing what the connection can really do if the TCP or MPI layers were perfectly written.« less
  • This preliminary study considers the scaling and performance of a finite element (FE) semiconductor device simulator on a capacity cluster with 272 compute nodes based on a homogeneous multicore node architecture utilizing 16 cores. The inter-node communication backbone for this Tri-Lab Linux Capacity Cluster (TLCC) machine is comprised of an InfiniBand interconnect. The nonuniform memory access (NUMA) nodes consist of 2.2 GHz quad socket/quad core AMD Opteron processors. The performance results for this study are obtained with a FE semiconductor device simulation code (Charon) that is based on a fully-coupled Newton-Krylov solver with domain decomposition and multilevel preconditioners. Scaling andmore » multicore performance results are presented for large-scale problems of 100+ million unknowns on up to 4096 cores. A parallel scaling comparison is also presented with the Cray XT3/4 Red Storm capability platform. The results indicate that an MPI-only programming model for utilizing the multicore nodes is reasonably efficient on all 16 cores per compute node. However, the results also indicated that the multilevel preconditioner, which is critical for large-scale capability type simulations, scales better on the Red Storm machine than the TLCC machine.« less
  • These are the slides for a presentation at the HPC Mini Showcase. This is a comparison of two different high performance network options: EDR InfiniBand and 100Gb RDMA capable ethernet. The conclusion of this comparison is the following: there is good potential, as shown with the direct results; 100Gb technology is too new and not standardized, thus deployment effort is complex for both options; different companies are not necessarily compatible; if you want 100Gb/s, you must get it all from one place.
  • Checkpoint-Restart is one of the most used software approaches to achieve fault-tolerance in high-end clusters. While standard techniques typically focus on user-level solutions, the advent of virtualization software has enabled efficient and transparent system-level approaches. In this paper, we present a scalable transparent system-level solution to address fault-tolerance for applications based on global address space (GAS) programming models on Infiniband clusters. In addition to handling communication, the solution addresses transparent checkpoint of user-generated files. We exploit the support for the Infiniband network in the Xen virtual machine environment. We have developed a version of the Aggregate Remote Memory Copy Interfacemore » (ARMCI) one-sided communication library capable of suspending and resuming applications. We present efficient and scalable mechanisms to distribute checkpoint requests and to backup virtual machines memory images and file systems. We tested our approach in the context of NWChem, a popular computational chemistry suite. We demonstrated that NWChem can be executed, without any modification to the source code, on a virtualized 8-node cluster with very little overhead (below 3%). We observe that the total checkpoint time is limited by disk I/O. Finally, we measured system-size depended components of the checkpoint time on up to 1024 cores (128 nodes), demonstrating the scalability of our approach in medium/large-scale systems.« less