Fault Injection for TensorFlow Applications

Narayanan, Niranjhana; Chen, Zitao; Fang, Bo; Li, Guanpeng; Pattabiraman, Karthik; DeBardeleben, Nathan A.

doi:10.1109/tdsc.2022.3175930

Fault Injection for TensorFlow Applications

Journal Article · Mon Jul 18 00:00:00 EDT 2022 · IEEE Transactions on Dependable and Secure Computing

DOI:https://doi.org/10.1109/tdsc.2022.3175930· OSTI ID:1994506

^[1]; Chen, Zitao ^[1]; Fang, Bo ^[2]; Li, Guanpeng ^[3]; ^[1]; ^[4]

University of British Columbia, Vancouver, BC (Canada)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
University of Iowa, Iowa City, IA (United States)
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

As machine learning (ML) has seen increasing adoption in safety-critical domains (e.g., autonomous vehicles), the reliability of ML systems has also grown in importance. While prior studies have proposed techniques to enable efficient error-resilience (e.g., selective instruction duplication), a fundamental requirement for realizing these techniques is a detailed understanding of the application’s resilience. In this work, we present TensorFI 1 and TensorFI 2, high-level fault injection (FI) frameworks for TensorFlow-based applications. TensorFI 1 and 2 are able to inject both hardware and software faults in any general TensorFlow 1 and 2 program respectively. Both are configurable FI tools that are flexible, easy to use, and portable. They can be integrated into existing TensorFlow programs to assess their resilience for different fault types (e.g., bit-flips in particular operations or layers). We use the TensorFI 1 and TensorFI 2 to evaluate the resilience of 12 and 10 ML programs written in TensorFlow, including DNNs used in the autonomous vehicle domain. The results give us insights into why some of the models are more resilient. We also measure the performance overheads of the two injectors, and present 4 case studies, two for each tool, to demonstrate their utility.

View Accepted Manuscript (DOE)

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: 89233218CNA000001; AC05-76RL01830

OSTI ID:: 1994506

Report Number(s):: LA-UR-21-22618; PNNL-SA-161122

Journal Information:: IEEE Transactions on Dependable and Secure Computing, Journal Name: IEEE Transactions on Dependable and Secure Computing Journal Issue: 4 Vol. 20; ISSN 1545-5971

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (24)

Fault injection techniques and tools No authors listed Computer, Vol. 30, Issue 4 https://doi.org/10.1109/2.585157	journal	April 1997
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey Akhtar, Naveed; Mian, Ajmal IEEE Access, Vol. 6 https://doi.org/10.1109/ACCESS.2018.2807385	journal	January 2018
DeepMutation++: A Mutation Testing Framework for Deep Learning Systems Hu, Qiang; Ma, Lei; Xie, Xiaofei 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) https://doi.org/10.1109/ASE.2019.00126	conference	November 2019
ImageNet: A large-scale hierarchical image database Deng, Jia; Dong, Wei; Socher, Richard 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), 2009 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2009.5206848	conference	June 2009
Cats and dogs Parkhi, O. M.; Vedaldi, A.; Zisserman, A. 2012 IEEE Conference on Computer Vision and Pattern Recognition https://doi.org/10.1109/CVPR.2012.6248092	conference	June 2012
Policy compression for aircraft collision avoidance systems Julian, Kyle D.; Lopez, Jessica; Brush, Jeffrey S. 2016 IEEE/AIAA 35th Digital Avionics Systems Conference (DASC) https://doi.org/10.1109/DASC.2016.7778091	conference	September 2016
PyTorchFI: A Runtime Perturbation Tool for DNNs Mahmoud, Abdulrahman; Aggarwal, Neeraj; Nobbe, Alex 2020 50th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W) https://doi.org/10.1109/DSN-W50199.2020.00014	conference	June 2020
LFI: A practical and general library-level fault injector Marinescu, Paul D.; Candea, George 2009 IEEE/IFIP International Conference on Dependable Systems & Networks https://doi.org/10.1109/DSN.2009.5270313	conference	June 2009
Quantifying the Accuracy of High-Level Fault Injection Techniques for Hardware Faults Wei, Jiesheng; Thomas, Anna; Li, Guanpeng 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) https://doi.org/10.1109/DSN.2014.2	conference	June 2014
Hands Off the Wheel in Autonomous Vehicles?: A Systems Perspective on over a Million Miles of Field Data Banerjee, Subho S.; Jha, Saurabh; Cyriac, James 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) https://doi.org/10.1109/DSN.2018.00066	conference	June 2018
Automated robustness testing of off-the-shelf software components Kropp, N. P.; Koopman, P. J.; Siewiorek, D. P. Digest of Papers. Twenty-Eighth Annual International Symposium on Fault-Tolerant Computing (Cat. No.98CB36224) https://doi.org/10.1109/FTCS.1998.689474	conference	January 1998
Evaluating Fault Resiliency of Compressed Deep Neural Networks Sabbagh, Majid; Gongye, Cheng; Fei, Yunsi 2019 IEEE International Conference on Embedded Software and Systems (ICESS) https://doi.org/10.1109/ICESS.2019.8782505	conference	June 2019
Detection of traffic signs in real-world images: The German traffic sign detection benchmark Houben, Sebastian; Stallkamp, Johannes; Salmen, Jan The 2013 International Joint Conference on Neural Networks (IJCNN) https://doi.org/10.1109/IJCNN.2013.6706807	conference	August 2013
NFTAPE: a framework for assessing dependability in distributed systems with lightweight fault injectors Stott, D. T.; Floering, B.; Burke, D. IPDS 2K: IEEE International Computer Performance and Dependability Symposium, Proceedings IEEE International Computer Performance and Dependability Symposium. IPDS 2000 https://doi.org/10.1109/IPDS.2000.839467	conference	January 2000
DeepMutation: Mutation Testing of Deep Learning Systems Ma, Lei; Zhang, Fuyuan; Sun, Jiyuan 2018 IEEE 29th International Symposium on Software Reliability Engineering (ISSRE) https://doi.org/10.1109/ISSRE.2018.00021	conference	October 2018
TensorFI: A Flexible Fault Injection Framework for TensorFlow Applications Chen, Zitao; Narayanan, Niranjhana; Fang, Bo 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE) https://doi.org/10.1109/ISSRE5003.2020.00047	conference	October 2020
LLFI: An Intermediate Code-Level Fault Injection Tool for Hardware Faults Lu, Qining; Farahani, Mostafa; Wei, Jiesheng 2015 IEEE International Conference on Software Quality, Reliability and Security https://doi.org/10.1109/QRS.2015.13	conference	August 2015
An empirical study of injected versus actual interface errors Lanzaro, Anna; Natella, Roberto; Winter, Stefan Proceedings of the 2014 International Symposium on Software Testing and Analysis https://doi.org/10.1145/2610384.2610418	conference	July 2014
Understanding error propagation in deep learning neural network (DNN) accelerators and applications Li, Guanpeng; Hari, Siva Kumar Sastry; Sullivan, Michael Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3126908.3126964	conference	November 2017
DeepXplore Pei, Kexin; Cao, Yinzhi; Yang, Junfeng Proceedings of the 26th Symposium on Operating Systems Principles https://doi.org/10.1145/3132747.3132785	conference	October 2017
Ares Reagen, Brandon; Gupta, Udit; Pentecost, Lillian Proceedings of the 55th Annual Design Automation Conference https://doi.org/10.1145/3195970.3195997	conference	June 2018
BinFI Chen, Zitao; Li, Guanpeng; Pattabiraman, Karthik Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356177	conference	November 2019
Audee Guo, Qianyu; Xie, Xiaofei; Li, Yi Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering https://doi.org/10.1145/3324884.3416571	conference	December 2020
Deep learning library testing via effective model generation Wang, Zan; Yan, Ming; Chen, Junjie Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering https://doi.org/10.1145/3368089.3409761	conference	November 2020

Similar Records

Supporting the Development of Soft-Error Resilient Message Passing Applications using Simulation

Conference · Thu Dec 31 23:00:00 EST 2015 · OSTI ID:1241477

SpotSDC: Revealing the Silent Data Corruption Propagation in High-Performance Computing Systems

Journal Article · Thu May 14 20:00:00 EDT 2020 · IEEE Transactions on Visualization and Computer Graphics · OSTI ID:1868154

Exploring the Interplay of Resilience and Energy Consumption for a Task-Based Partial Differential Equations Preconditioner

Technical Report · Mon Feb 29 23:00:00 EST 2016 · OSTI ID:1561016

Related Subjects

97 MATHEMATICS AND COMPUTING
Deep Neural Networks
Error Resilience
Fault Injection
Machine Learning
TensorFlow

Fault Injection for TensorFlow Applications

Citation Formats

References (24)

Similar Records

Related Subjects