Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System

Kumar, Mohit; Gupta, Saurabh; Patel, Tirthak; Wilder, Michael; Shi, Weisong; Fu, Song; Engelmann, Christian; Tiwari, Devesh

doi:10.1109/DSN.2018.00023

Title: Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System

Full Record
Other Related Research

Abstract

Today's High Performance Computing (HPC) systems are capable of delivering performance in the order of petaflops due to the fast computing devices, network interconnect, and back-end storage systems. In particular, interconnect resilience and congestion resolution methods have a major impact on the overall interconnect and application performance. This is especially true for scientific applications running multiple processes on different compute nodes as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks state-of-practice experience reports that detail how different interconnect errors and congestion events occur on large-scale HPC systems. Therefore, in this paper, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors and congestion events. We also study the interaction between interconnect, errors, network congestion and application characteristics.

Authors:

Kumar, Mohit ^[1]; Gupta, Saurabh ^[2]; Patel, Tirthak ^[3]; Wilder, Michael ^[4]; Shi, Weisong ^[1]; Fu, Song ^[5];

^[6]; Tiwari, Devesh ^[3]

Wayne State University, Detroit
Intel Corporation
Northeastern University, Boston
University of Tennessee, Knoxville (UTK)
University of North Texas
ORNL

Publication Date:: Fri Jun 01 00:00:00 EDT 2018

Research Org.:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1465034

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: 48th IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) 2018 - Luxembourg City, , Luxembourg - 6/25/2018 12:00:00 PM-6/28/2018 12:00:00 PM

Country of Publication:: United States

Language:: English

Citation Formats


                    Kumar, Mohit, Gupta, Saurabh, Patel, Tirthak, Wilder, Michael, Shi, Weisong, Fu, Song, Engelmann, Christian, and Tiwari, Devesh. Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System.  United States: N. p., 2018. 
        Web.  doi:10.1109/DSN.2018.00023.

Copy to clipboard


                    Kumar, Mohit, Gupta, Saurabh, Patel, Tirthak, Wilder, Michael, Shi, Weisong, Fu, Song, Engelmann, Christian, & Tiwari, Devesh. Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System.  United States.  https://doi.org/10.1109/DSN.2018.00023

Copy to clipboard


                    Kumar, Mohit, Gupta, Saurabh, Patel, Tirthak, Wilder, Michael, Shi, Weisong, Fu, Song, Engelmann, Christian, and Tiwari, Devesh. 2018.  
        "Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System".  United States.  https://doi.org/10.1109/DSN.2018.00023.  https://www.osti.gov/servlets/purl/1465034.

Copy to clipboard


                    
@article{osti_1465034,

  title        = {Understanding and Analyzing Interconnect Errors and Network Congestion on a Large Scale HPC System},

  author       = {Kumar, Mohit and Gupta, Saurabh and Patel, Tirthak and Wilder, Michael and Shi, Weisong and Fu, Song and Engelmann, Christian and Tiwari, Devesh},

  abstractNote = {Today's High Performance Computing (HPC) systems are capable of delivering performance in the order of petaflops due to the fast computing devices, network interconnect, and back-end storage systems. In particular, interconnect resilience and congestion resolution methods have a major impact on the overall interconnect and application performance. This is especially true for scientific applications running multiple processes on different compute nodes as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks state-of-practice experience reports that detail how different interconnect errors and congestion events occur on large-scale HPC systems. Therefore, in this paper, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors and congestion events. We also study the interaction between interconnect, errors, network congestion and application characteristics.},

  doi          = {10.1109/DSN.2018.00023},

  url          = {https://www.osti.gov/biblio/1465034},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {Fri Jun 01 00:00:00 EDT 2018},

  month        = {Fri Jun 01 00:00:00 EDT 2018}

}

Copy to clipboard

Conference:

View Conference (0.34 MB)

https://doi.org/10.1109/DSN.2018.00023

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Study of interconnect errors, network congestion, and applications characteristics for throttle prediction on a large scale HPC system

Journal Article Kumar, Mohit; Gupta, Saurabh; Patel, Tirthak; ... - Journal of Parallel and Distributed Computing

Today’s High Performance Computing (HPC) systems contain thousand of nodes which work together to provide performance in the order of petaflops. The performance of these systems depends on various components like processors, memory, and interconnect. Among all, interconnect plays a major role as it glues together all the hardware components in an HPC system. A slow interconnect can impact a scientific application running on multiple processes severely as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks a study that explores different interconnect errors, congestion events and applications characteristics on a large-scale HPCmore »« less
https://doi.org/10.1016/j.jpdc.2021.03.001

Full Text Available
Understanding GPU errors on large-scale HPC systems and the implications for system design and operation

Conference Tiwari, Devesh; Gupta, Saurabh; Rogers, James; ... - 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA); 7-11 Feb. 2015; Burlingame, CA, USA

Increase in graphics hardware performance and improvements in programmability has enabled GPUs to evolve from a graphics-specific accelerator to a general-purpose computing device. Titan, the world's second fastest supercomputer for open science in 2014, consists of more dum 18,000 GPUs that scientists from various domains such as astrophysics, fusion, climate, and combustion use routinely to run large-scale simulations. Unfortunately, while the performance efficiency of GPUs is well understood, their resilience characteristics in a large-scale computing system have not been fully evaluated. We present a detailed study to provide a thorough understanding of GPU errors on a large-scale GPU-enabled system. Ourmore »« less
https://doi.org/10.1109/HPCA.2015.7056044
Designing Scalable PGAS Communication Subsystems on Cray Gemini Interconnect

Conference Vishnu, Abhinav; Daily, Jeffrey; Palmer, Bruce

The Cray Gemini Interconnect has been recently introduced as a next generation network architecture for building multi-petaflop supercomputers. Cray XE6 systems including LANL Cielo, NERSC Hopper, ORNL Titan and proposed NCSA BlueWaters leverage the Gemini Interconnect as their primary Interconnection network. At the same time, programming models such as the Message Passing Interface (MPI) and Partitioned Global Address Space (PGAS) models such as Unified Parallel C (UPC) and Co-Array Fortran (CAF) have become available on these systems. Global Arrays is a popular PGAS model used in a variety of application domains including hydrodynamics, chemistry and visualization. Global Arrays uses Aggregatemore »« less
https://doi.org/10.1109/HiPC.2012.6507506
A case for Virtual Machine based Fault Injection in a High-Performance Computing Environment

Conference Vallee, Geoffroy; Engelmann, Christian; Scott, Stephen

Large-scale computing platforms provide tremendous capabilities for scientific discovery. These systems have hundreds of thousands of computing cores, hundreds of terabytes of memory, and enormous high-performance interconnection networks. These systems are facing enormous challenges to achieve performance at such scale. Failures are an Achilles heel of these enormous systems. As applications and system software scale up to multi-petaflop and beyond to exascale platforms, the occurrence of failure will be much more common. This has given rise to a push in fault-tolerance and resilience research for HPC systems. This includes work on log analysis to identify types of failures, enhancements tomore »« less
Characterization and identification of HPC applications at leadership computing facility

Conference Liu, Zhengchun; Rao, Nageswara; Kettimuthu, Rajkumar; ...

High Performance Computing (HPC) is an important method for scientific discovery via large-scale simulation, data analysis, or artificial intelligence. Leadership-class supercomputers are expensive, but essential to run large HPC applications. The Petascale era of supercomputers began in 2008, with the first machines achieving performance in excess of one petaflops, and with the advent of new supercomputers in 2021 (e.g., Aurora, Frontier), the Exascale era will soon begin. However, the high theoretical computing capability (i.e., peak FLOPS) of a machine is not the only meaningful target when designing a supercomputer, as the resources demand of applications varies. A deep understanding ofmore »« less
https://doi.org/10.1145/3392717.3392774

Full Text Available

Similar Records