SpotSDC: Revealing the Silent Data Corruption Propagation in High-Performance Computing Systems

Li, Zhimin; Menon, Harshitha; Maljovec, Dan; Livnat, Yarden; Liu, Shusen; Mohror, Kathryn; Bremer, Peer-Timo; Pascucci, Valerio

doi:10.1109/tvcg.2020.2994954

Title: SpotSDC: Revealing the Silent Data Corruption Propagation in High-Performance Computing Systems

Journal Article · Fri May 15 00:00:00 EDT 2020 · IEEE Transactions on Visualization and Computer Graphics

DOI:https://doi.org/10.1109/tvcg.2020.2994954· OSTI ID:1868154

^[1]; Menon, Harshitha ^[2]; Maljovec, Dan ^[1];

^[1];

^[2]; Mohror, Kathryn ^[2];

^[2];

^[1]

Univ. of Utah, Salt Lake City, UT (United States)
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

We report the trend of rapid technology scaling is expected to make the hardware of high-performance computing (HPC) systems more susceptible to computational errors due to random bit flips. Some bit flips may cause a program to crash or have a minimal effect on the output, but others may lead to silent data corruption (SDC), i.e., undetected yet significant output errors. Classical fault injection analysis methods employ uniform sampling of random bit flips during program execution to derive a statistical resiliency profile. However, summarizing such fault injection result with sufficient detail is difficult, and understanding the behavior of the fault-corrupted program is still a challenge. In this article, we introduce SpotSDC, a visualization system to facilitate the analysis of a program's resilience to SDC. SpotSDC provides multiple perspectives at various levels of detail of the impact on the output relative to where in the source code the flipped bit occurs, which bit is flipped, and when during the execution it happens. SpotSDC also enables users to study the code protection and provide new insights to understand the behavior of a fault-injected program. Based on lessons learned, we demonstrate how what we found can improve the fault injection campaign method.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC52-07NA27344

OSTI ID:: 1868154

Report Number(s):: LLNL-JRNL-764021; 953990

Journal Information:: IEEE Transactions on Visualization and Computer Graphics, Vol. 27, Issue 10; ISSN 1077-2626

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (48)

One Bit is (Not) Enough: An Empirical Study of the Impact of Single and Multiple Bit-Flip Errors Sangchoolie, Behrooz; Pattabiraman, Karthik; Karlsson, Johan 2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) https://doi.org/10.1109/DSN.2017.30	conference	June 2017
Visualizing Network Traffic to Understand the Performance of Massively Parallel Simulations Landge, A. G.; Levine, J. A.; Bhatele, A. IEEE Transactions on Visualization and Computer Graphics, Vol. 18, Issue 12 https://doi.org/10.1109/TVCG.2012.286	journal	December 2012
Relyzer: exploiting application-level fault equivalence to analyze application resiliency to transient faults Hari, Siva Kumar Sastry; Adve, Sarita V.; Naeimi, Helia ACM SIGPLAN Notices, Vol. 47, Issue 4 https://doi.org/10.1145/2248487.2150990	journal	June 2012
Statistical fault injection: Quantified error and confidence Leveugle, R.; Calvez, A.; Maistri, P. 2009 Design, Automation & Test in Europe Conference & Exhibition (DATE'09) https://doi.org/10.1109/DATE.2009.5090716	conference	April 2009
Self-stabilizing iterative solvers Sao, Piyush; Vuduc, Richard Proceedings of the Workshop on Latest Advances in Scalable Algorithms for Large-Scale Systems - ScalA '13 https://doi.org/10.1145/2530268.2530272	conference	January 2013
Hierarchical Aggregation for Information Visualization: Overview, Techniques, and Design Guidelines Elmqvist, N.; Fekete, J. -D. IEEE Transactions on Visualization and Computer Graphics, Vol. 16, Issue 3 https://doi.org/10.1109/TVCG.2009.84	journal	May 2010
Understanding the propagation of transient errors in HPC applications Ashraf, Rizwan A.; Gioiosa, Roberto; Kestor, Gokcen Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807670	conference	January 2015
Ensemble-Vis: A Framework for the Statistical Visualization of Ensemble Data Potter, Kristin; Wilson, Andrew; Bremer, Peer-Timo 2009 IEEE International Conference on Data Mining Workshops (ICDMW) https://doi.org/10.1109/ICDMW.2009.55	conference	December 2009
ADAPT: Algorithmic Differentiation Applied to Floating-Point Precision Tuning Menon, Harshitha; Lam, Michael O.; Osei-Kuffuor, Daniel SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00051	conference	November 2018
Tree visualization with tree-maps: 2-d space-filling approach Shneiderman, Ben ACM Transactions on Graphics, Vol. 11, Issue 1 https://doi.org/10.1145/102377.115768	journal	January 1992
Shoestring: probabilistic soft error reliability on the cheap Feng, Shuguang; Gupta, Shantanu; Ansari, Amin ACM SIGARCH Computer Architecture News, Vol. 38, Issue 1 https://doi.org/10.1145/1735970.1736063	journal	March 2010
Quantifying the Accuracy of High-Level Fault Injection Techniques for Hardware Faults Wei, Jiesheng; Thomas, Anna; Li, Guanpeng 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) https://doi.org/10.1109/DSN.2014.2	conference	June 2014
Supercomputing's monster in the closet Geist, Al IEEE Spectrum, Vol. 53, Issue 3 https://doi.org/10.1109/MSPEC.2016.7420396	journal	March 2016
Low-cost program-level detectors for reducing silent data corruptions Hari, Siva Kumar Sastry; Adve, Sarita V.; Naeimi, Helia 2012 42nd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2012) https://doi.org/10.1109/DSN.2012.6263960	conference	June 2012
An algorithm for the machine calculation of complex Fourier series Cooley, James W.; Tukey, John W. Mathematics of Computation, Vol. 19, Issue 90 https://doi.org/10.1090/S0025-5718-1965-0178586-1	journal	May 1965
Treevis.net: A Tree Visualization Reference Schulz, Hans-Jorg IEEE Computer Graphics and Applications, Vol. 31, Issue 6 https://doi.org/10.1109/MCG.2011.103	journal	November 2011
The eyes have it: a task by data type taxonomy for information visualizations Shneiderman, B. Proceedings 1996 IEEE Symposium on Visual Languages https://doi.org/10.1109/VL.1996.545307	conference	January 1996
Evaluating the viability of process replication reliability for exascale systems Ferreira, Kurt; Stearley, Jon; Laros, James H. Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063443	conference	January 2011
Software-controlled fault tolerance Reis, George A.; Chang, Jonathan; Vachharajani, Neil ACM Transactions on Architecture and Code Optimization, Vol. 2, Issue 4 https://doi.org/10.1145/1113841.1113843	journal	December 2005
Visualization of large hierarchical data by circle packing Wang, Weixin; Wang, Hui; Dai, Guozhong CHI06: CHI 2006 Conference on Human Factors in Computing Systems, Proceedings of the SIGCHI Conference on Human Factors in Computing Systems https://doi.org/10.1145/1124772.1124851	conference	April 2006
Evaluating the Impact of SDC on the GMRES Iterative Solver Elliott, James; Hoemmen, Mark; Mueller, Frank 2014 IEEE International Parallel & Distributed Processing Symposium (IPDPS), 2014 IEEE 28th International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2014.123	conference	May 2014
Addressing failures in exascale computing Snir, Marc; Wisniewski, Robert W.; Abraham, Jacob A. The International Journal of High Performance Computing Applications, Vol. 28, Issue 2 https://doi.org/10.1177/1094342014522573	journal	March 2014
Hauberk: Lightweight Silent Data Corruption Error Detector for GPGPU Yim, Keun Soo; Pham, Cuong; Saleheen, Mushfiq Distributed Processing Symposium (IPDPS), 2011 IEEE International Parallel & Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2011.36	conference	May 2011
Modeling Soft-Error Propagation in Programs Li, Guanpeng; Pattabiraman, Karthik; Hari, Siva Kumar Sastry 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) https://doi.org/10.1109/DSN.2018.00016	conference	June 2018
CFGExplorer: Designing a Visual Control Flow Analytics System around Basic Program Analysis Operations Devkota, Sabin; Isaacs, Katherine E. Computer Graphics Forum, Vol. 37, Issue 3 https://doi.org/10.1111/cgf.13433	journal	June 2018
Understanding GPU errors on large-scale HPC systems and the implications for system design and operation Tiwari, Devesh; Gupta, Saurabh; Rogers, James 2015 IEEE 21st International Symposium on High Performance Computer Architecture (HPCA) https://doi.org/10.1109/HPCA.2015.7056044	conference	February 2015
Fault injection acceleration by architectural importance sampling Ebrahimi, Mojtaba; Sayed, Nour; Rashvand, Maryam 2015 International Conference on Hardware/Software Codesign and System Synthesis (CODES+ISSS) https://doi.org/10.1109/CODESISSS.2015.7331384	conference	October 2015
Analysis of Fault Tolerance in Artificial Neural Networks Piuri, Vincenzo Journal of Parallel and Distributed Computing, Vol. 61, Issue 1 https://doi.org/10.1006/jpdc.2000.1663	journal	January 2001
A Visual Analytics Framework for the Detection of Anomalous Call Stack Trees in High Performance Computing Applications Xie, Cong; Xu, Wei; Mueller, Klaus IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 1 https://doi.org/10.1109/TVCG.2018.2865026	journal	January 2019
Contour Boxplots: A Method for Characterizing Uncertainty in Feature Sets from Simulation Ensembles Whitaker, Ross T.; Mirzargar, Mahsa; Kirby, Robert M. IEEE Transactions on Visualization and Computer Graphics, Vol. 19, Issue 12 https://doi.org/10.1109/TVCG.2013.143	journal	December 2013
SWIFT: Software Implemented Fault Tolerance Reis, G. A.; Chang, J.; Vachharajani, N. International Symposium on Code Generation and Optimization https://doi.org/10.1109/CGO.2005.34	conference	January 2005
Juniper: A Tree+Table Approach to Multivariate Graph Visualization Nobre, Carolina; Streit, Marc; Lex, Alexander IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 1 https://doi.org/10.1109/TVCG.2018.2865149	journal	January 2019
The Soft Error Problem: An Architectural Perspective Mukherjee, S. S.; Emer, J.; Reinhardt, S. K. 11th International Symposium on High-Performance Computer Architecture https://doi.org/10.1109/HPCA.2005.37	conference	January 2005
Error Detecting and Error Correcting Codes Hamming, R. W. Bell System Technical Journal, Vol. 29, Issue 2 https://doi.org/10.1002/j.1538-7305.1950.tb00463.x	journal	April 1950
A system for graph-based visualization of the evolution of software Collberg, Christian; Kobourov, Stephen; Nagra, Jasvir Proceedings of the 2003 ACM symposium on Software visualization - SoftVis '03 https://doi.org/10.1145/774841.774844	conference	January 2003
D is CV ar: discovering critical variables using algorithmic differentiation for transient faults Menon, Harshitha; Mohror, Kathryn PPoPP '18: 23nd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, Proceedings of the 23rd ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming https://doi.org/10.1145/3178487.3178502	conference	February 2018
IPAS: intelligent protection against silent output corruption in scientific applications Laguna, Ignacio; Schulz, Martin; Richards, David F. CGO '16: 14th Annual IEEE/ACM International Symposium on Code Generation and Optimization, Proceedings of the 2016 International Symposium on Code Generation and Optimization https://doi.org/10.1145/2854038.2854059	conference	February 2016
Combing the Communication Hairball: Visualizing Parallel Execution Traces using Logical Time Isaacs, Katherine E.; Bremer, Peer-Timo; Jusufi, Ilir IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346456	journal	December 2014
FlipTracker: Understanding Natural Error Resilience in HPC Applications Guo, Luanzheng; Li, Dong; Laguna, Ignacio SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2018.00011	conference	November 2018
Curve Boxplot: Generalization of Boxplot for Ensembles of Curves Mirzargar, Mahsa; Whitaker, Ross T.; Kirby, Robert M. IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346455	journal	December 2014
Fixed-Rate Compressed Floating-Point Arrays Lindstrom, Peter IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346458	journal	December 2014
Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers Hohman, Fred; Kahng, Minsuk; Pienta, Robert IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 8 https://doi.org/10.1109/TVCG.2018.2843369	journal	August 2019
SDCTune: a model for predicting the SDC proneness of an application for configurable protection Lu, Qining; Pattabiraman, Karthik; Gupta, Meeta S. ESWEEK'14: TENTH EMBEDDED SYSTEM WEEK, Proceedings of the 2014 International Conference on Compilers, Architecture and Synthesis for Embedded Systems https://doi.org/10.1145/2656106.2656127	conference	October 2014
Lineage: Visualizing Multivariate Clinical Data in Genealogy Graphs Nobre, Carolina; Gehlenborg, Nils; Coon, Hilary IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 3 https://doi.org/10.1109/TVCG.2018.2811488	journal	March 2019
Fault resilience of the algebraic multi-grid solver Casas, Marc; de Supinski, Bronis R.; Bronevetsky, Greg Proceedings of the 26th ACM international conference on Supercomputing - ICS '12 https://doi.org/10.1145/2304576.2304590	conference	January 2012
Understanding Error Propagation in GPGPU Applications Li, Guanpeng; Pattabiraman, Karthik; Cher, Chen-Yang SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.20	conference	November 2016
ColorBrewer.org: An Online Tool for Selecting Colour Schemes for Maps Harrower, Mark; Brewer, Cynthia A. The Cartographic Journal, Vol. 40, Issue 1 https://doi.org/10.1179/000870403235002042	journal	June 2003
Visualization and Visual Analysis of Ensemble Data: A Survey Wang, Junpeng; Hazarika, Subhashis; Li, Cheng IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 9 https://doi.org/10.1109/TVCG.2018.2853721	journal	September 2019

Similar Records

An Efficient Silent Data Corruption Detection Method with Error-Feedback Control and Even Sampling for HPC Applications

Conference · Thu Jan 01 00:00:00 EST 2015 · OSTI ID:1868154

Di, Sheng; Berrocal, Eduardo; Cappello, Franck

LADR: low-cost application-level detector for reducing silent output corruptions

Conference · Fri Jun 01 00:00:00 EDT 2018 · OSTI ID:1868154

Chen, Chao; Eisenhauer, Greg; Wolf, Matthew D.; +1 more

Exploring the capabilities of support vector machines in detecting silent data corruptions

Journal Article · Thu Feb 01 00:00:00 EST 2018 · Sustainable Computing · OSTI ID:1868154

Subasi, Omer; Di, Sheng; Bautista-Gomez, Leonardo; +6 more

Related Subjects

97 MATHEMATICS AND COMPUTING
fault injection sampling
error propagation
information visualization
silent data corruption

Title: SpotSDC: Revealing the Silent Data Corruption Propagation in High-Performance Computing Systems

Citation Formats

References (48)

Similar Records

Related Subjects