skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Investigations on InfiniBand: Efficient Network Buffer Utilization at Scale.


Abstract not provided.

; ; ; ;
Publication Date:
Research Org.:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
Report Number(s):
DOE Contract Number:
Resource Type:
Resource Relation:
Conference: Proposed for presentation at the 14th European PVM/MPI Users' Group Conference held September 30 - October 3, 2007 in Paris, France.
Country of Publication:
United States

Citation Formats

Brightwell, Ronald B., Shipman, Galen M., Barrett, Brian, Squyreus, Jeffrey M., and Bloch, Gil. Investigations on InfiniBand: Efficient Network Buffer Utilization at Scale.. United States: N. p., 2007. Web.
Brightwell, Ronald B., Shipman, Galen M., Barrett, Brian, Squyreus, Jeffrey M., & Bloch, Gil. Investigations on InfiniBand: Efficient Network Buffer Utilization at Scale.. United States.
Brightwell, Ronald B., Shipman, Galen M., Barrett, Brian, Squyreus, Jeffrey M., and Bloch, Gil. Tue . "Investigations on InfiniBand: Efficient Network Buffer Utilization at Scale.". United States. doi:.
title = {Investigations on InfiniBand: Efficient Network Buffer Utilization at Scale.},
author = {Brightwell, Ronald B. and Shipman, Galen M. and Barrett, Brian and Squyreus, Jeffrey M. and Bloch, Gil},
abstractNote = {Abstract not provided.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Tue May 01 00:00:00 EDT 2007},
month = {Tue May 01 00:00:00 EDT 2007}

Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:
  • In the last decade or so, clusters have observed a tremendous rise in popularity due to excellent price to performance ratio. A variety of Interconnects have been proposed during this period, with InfiniBand leading the way due to its high performance and open standard. Increasing size of the InfiniBand clusters has reduced the mean time between failures of various components of these clusters tremendously. In this paper, we specifically focus on the network component failure and propose a hybrid hardware-software approach to handling network faults. The hybrid approach leverages the user-transparent network fault detection and recovery using Automatic Path Migrationmore » (APM), and the software approach is used in the wake of APM failure. Using Global Arrays as the programming model, we implement this approach with Aggregate Remote Memory Copy Interface (ARMCI), the runtime system of Global Arrays. We evaluate our approach using various benchmarks (siosi7, pentane, h2o7 and siosi3) with NWChem, a very popular {\em ab initio} quantum chemistry application. Using the proposed approach, the applications run to completion without restart on emulated network faults and acceptable overhead for benchmarks executing for a longer period of time.« less
  • Real-time applications demand fast computation, this paper proposes an efficient algorithm for real-time network reconfiguration on large unbalanced distribution networks. A novel formulation of the network reconfiguration to achieve loss minimization and load balancing is given. To reduce computational requirements for the solution algorithm, well justified power flow and loss reduction formulas in terms of the on/off status of network switches are proposed for efficient system updating. The algorithm relies only a few full power flow studies based on system states attained by explicit expressions using backward-forward sweeps for efficient computation of system`s states at the critical system operating points.more » The solution algorithm runs in an amount of time linearly proportional to the number of tie switches and the number of sectionalizing switches in the system. The solution algorithm has been implemented into a software package and tested on unbalanced distribution systems including a system with 292-buses, 76-laterals, 7 transformers, 45 switches and 255 lines sections under diverse system conditions.« less
  • Abstract not provided.
  • This paper introduces the newly developed Infini-Band (IB) Management Queue capability, used by the Host Channel Adapter (HCA) to manage network task data flow dependancies, and progress the communications associated with such flows. These tasks include sends, receives, and the newly supported wait task, and are scheduled by the HCA based on a data dependency description provided by the user. This functionality is supported by the ConnectX-2 HCA, and provides the means for delegating collective communication management and progress to the HCA, also known as collective communication offload. This provides a means for overlapping collective communications managed by the HCAmore » and computation on the Central Processing Unit (CPU), thus making it possible to reduce the impact of system noise on parallel applications using collective operations. This paper further describes how this new capability can be used to implement scalable Message Passing Interface (MPI) collective operations, describing the high level details of how this new capability is used to implement the MPI Barrier collective operation, focusing on the latency sensitive performance aspects of this new capability. This paper concludes with small scale benchmark experiments comparing implementations of the barrier collective operation, using the new network offload capabilities, with established point-to-point based implementations of these same algorithms, which manage the data flow using the central processing unit. These early results demonstrate the promise this new capability provides to improve the scalability of high performance applications using collective communications. The latency of the HCA based implementation of the barrier is similar to that of the best performing point-to-point based implementation managed by the central processing unit, starting to outperform these as the number of processes involved in the collective operation increases.« less