Statistical Fault Detection for Parallel Applications with AutomaDeD

Bronevetsky, G; Laguna, I; Bagchi, S; de Supinski, B R; Ahn, D; Schulz, M

Title: Statistical Fault Detection for Parallel Applications with AutomaDeD

Full Record
Other Related Research

Abstract

Today's largest systems have over 100,000 cores, with million-core systems expected over the next few years. The large component count means that these systems fail frequently and often in very complex ways, making them difficult to use and maintain. While prior work on fault detection and diagnosis has focused on faults that significantly reduce system functionality, the wide variety of failure modes in modern systems makes them likely to fail in complex ways that impair system performance but are difficult to detect and diagnose. This paper presents AutomaDeD, a statistical tool that models the timing behavior of each application task and tracks its behavior to identify any abnormalities. If any are observed, AutomaDeD can immediately detect them and report to the system administrator the task where the problem began. This identification of the fault's initial manifestation can provide administrators with valuable insight into the fault's root causes, making it significantly easier and cheaper for them to understand and repair it. Our experimental evaluation shows that AutomaDeD detects a wide range of faults immediately after they occur 80% of the time, with a low false-positive rate. Further, it identifies weaknesses of the current approach that motivate future research.

Authors:: Bronevetsky, G; Laguna, I; Bagchi, S; de Supinski, B R; Ahn, D; Schulz, M

Publication Date:: 2010-03-23

Research Org.:: Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 974392

Report Number(s):: LLNL-CONF-426254
TRN: US201007%%695

DOE Contract Number:: W-7405-ENG-48

Resource Type:: Conference

Resource Relation:: Conference: Presented at: IEEE Workshop on Silicon Errors in Logic - System Effects, Stanford, CA, United States, Mar 23 - Mar 24, 2010

Country of Publication:: United States

Language:: English

Subject:: 99 GENERAL AND MISCELLANEOUS; DETECTION; DIAGNOSIS; EVALUATION; PERFORMANCE; REPAIR; SILICON

Citation Formats


                    Bronevetsky, G, Laguna, I, Bagchi, S, de Supinski, B R, Ahn, D, and Schulz, M. Statistical Fault Detection for Parallel Applications with AutomaDeD.  United States: N. p., 2010. 
        Web.

Copy to clipboard


                    Bronevetsky, G, Laguna, I, Bagchi, S, de Supinski, B R, Ahn, D, & Schulz, M. Statistical Fault Detection for Parallel Applications with AutomaDeD.  United States.

Copy to clipboard


                    Bronevetsky, G, Laguna, I, Bagchi, S, de Supinski, B R, Ahn, D, and Schulz, M. 2010.  
        "Statistical Fault Detection for Parallel Applications with AutomaDeD".  United States.  https://www.osti.gov/servlets/purl/974392.

Copy to clipboard


                    
@article{osti_974392,

  title        = {Statistical Fault Detection for Parallel Applications with AutomaDeD},

  author       = {Bronevetsky, G and Laguna, I and Bagchi, S and de Supinski, B R and Ahn, D and Schulz, M},

  abstractNote = {Today's largest systems have over 100,000 cores, with million-core systems expected over the next few years. The large component count means that these systems fail frequently and often in very complex ways, making them difficult to use and maintain. While prior work on fault detection and diagnosis has focused on faults that significantly reduce system functionality, the wide variety of failure modes in modern systems makes them likely to fail in complex ways that impair system performance but are difficult to detect and diagnose. This paper presents AutomaDeD, a statistical tool that models the timing behavior of each application task and tracks its behavior to identify any abnormalities. If any are observed, AutomaDeD can immediately detect them and report to the system administrator the task where the problem began. This identification of the fault's initial manifestation can provide administrators with valuable insight into the fault's root causes, making it significantly easier and cheaper for them to understand and repair it. Our experimental evaluation shows that AutomaDeD detects a wide range of faults immediately after they occur 80% of the time, with a low false-positive rate. Further, it identifies weaknesses of the current approach that motivate future research.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/974392},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2010},

  month        = {3}

}

Copy to clipboard

Conference:

View Conference (0.13 MB)

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

AutomaDeD: Automata-Based Debugging for Dissimilar Parallel Tasks

Conference Bronevetsky, G; Laguna, I; Bagchi, S; ...

Today's largest systems have over 100,000 cores, with million-core systems expected over the next few years. This growing scale makes debugging the applications that run on them a daunting challenge. Few debugging tools perform well at this scale and most provide an overload of information about the entire job. Developers need tools that quickly direct them to the root cause of the problem. This paper presents AutomaDeD, a tool that identifies which tasks of a large-scale application first manifest a bug at a specific code region at a specific point during program execution. AutomaDeD creates a statistical model of themore »« less
https://doi.org/10.1109/DSN.2010.5544927

Full Text Available
Automatic Fault Characterization via Abnormality-Enhanced Classification

Conference Bronevetsky, G; Laguna, I; de Supinski, B

Enterprise and high-performance computing systems are growing extremely large and complex, employing hundreds to hundreds of thousands of processors and software/hardware stacks built by many people across many organizations. As the growing scale of these machines increases the frequency of faults, system complexity makes these faults difficult to detect and to diagnose. Current system management techniques, which focus primarily on efficient data access and query mechanisms, require system administrators to examine the behavior of various system services manually. Growing system complexity is making this manual process unmanageable: administrators require more effective management tools that can detect faults and help tomore »« less
https://doi.org/10.1109/DSN.2012.6263926

Full Text Available
Petascale Computing Enabling Technologies Project Final Report

Technical Report de Supinski, B

The Petascale Computing Enabling Technologies (PCET) project addressed challenges arising from current trends in computer architecture that will lead to large-scale systems with many more nodes, each of which uses multicore chips. These factors will soon lead to systems that have over one million processors. Also, the use of multicore chips will lead to less memory and less memory bandwidth per core. We need fundamentally new algorithmic approaches to cope with these memory constraints and the huge number of processors. Further, correct, efficient code development is difficult even with the number of processors in current systems; more processors will onlymore »« less
https://doi.org/10.2172/972422

Full Text Available
Holistic Measurement Driven Resilience: Combining Operational Fault and Failure Measurements and Fault Injection for Quantifying Fault Detection, Propagation and Impact. Final report

Technical Report Kramer, William; Jha, Saurabh; Brandt, James; ...

For HPC systems to date, application resilience to faults and failures has been accomplished by the brute- force method of checkpoint/restart, which allows an application to make forward progress in the face of system and application faults, errors, and failures independent of root cause or end result. It has remained the primary resilience mechanism because we lack a way to identify faults and anticipate consequences early enough to take meaningful mitigating action. However, checkpoint/restart implementations put a tremendous burden on system resources and on the applications themselves and is becoming less feasible at scale. Because we have not yet operatedmore »« less
https://doi.org/10.2172/1615150

Full Text Available
FY2018 Discovery Through Situational Awareness: Anomalies, Oscillations, and Classification

Technical Report Amidan, Brett; Follum, James; Yin, Tianzhixi; ...

This technical report is a summary of work performed under the DOE GMLC program under the task titled “Discovery through Situational Awareness”. The focus of this work was to apply big data, statistical techniques to PMU data to do the following: 1) baseline PMU data and find anomalies with a focus on phase angle pairs, 2) identify the frequencies of dominant modal oscillations between different parts of a power system, and 3) identify and classify frequency events in PMU data. This work was performed on PMU data from the Western Interconnect. Many of these algorithms will be applied to widemore »« less
https://doi.org/10.2172/1846595

Full Text Available

Similar Records