Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Topological Interpretability for Deep Learning

Conference ·
 [1];  [1];  [1];  [2]
  1. ORNL
  2. National Cancer Institute, Bethesda, MD
+ Show Author Affiliations

With the growing adoption of AI-based systems across everyday life, the need to understand their decision-making mechanisms is correspondingly increasing. The level at which we can trust the statistical inferences made from AI-based decision systems is an increasing concern, especially in high-risk systems such as criminal justice or medical diagnosis, where incorrect inferences may have tragic consequences. Despite their successes in providing solutions to problems involving real-world data, deep learning (DL) models cannot quantify the certainty of their predictions. These models are frequently quite confident, even when their solutions are incorrect.This work presents a method to infer prominent features in two DL classification models trained on clinical and non-clinical text by employing techniques from topological and geometric data analysis. We create a graph of a model's feature space and cluster the inputs into the graph's vertices by the similarity of features and prediction statistics. We then extract subgraphs demonstrating high-predictive accuracy for a given label. These subgraphs contain a wealth of information about features that the DL model has recognized as relevant to its decisions. We infer these features for a given label using a distance metric between probability measures, and demonstrate the stability of our method compared to the LIME and SHAP interpretability methods. This work establishes that we may gain insights into the decision mechanism of a DL model. This method allows us to ascertain if the model is making its decisions based on information germane to the problem or identifies extraneous patterns within the data.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Organization:
USDOE
DOE Contract Number:
AC05-00OR22725
OSTI ID:
2371068
Country of Publication:
United States
Language:
English

References (15)

Topological data analysis for discovery in preclinical spinal cord injury and traumatic brain injury journal October 2015
Fooling LIME and SHAP conference February 2020
TopoBERT: Exploring the topology of fine-tuned word representations journal May 2023
Automatic extraction of cancer registry reportable information from free-text pathology reports using multitask convolutional neural networks journal November 2019
An Efficient Algorithm for Calculating the Exact Hausdorff Distance journal November 2015
The directed Hausdorff distance between imprecise point sets journal July 2011
Classifying cancer pathology reports with hierarchical self-attention networks journal November 2019
Topology based data analysis identifies a subgroup of breast cancers with a unique mutational profile and excellent survival journal April 2011
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
  • Ribeiro, Marco Tulio; Singh, Sameer; Guestrin, Carlos
  • Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16 https://doi.org/10.1145/2939672.2939778
conference January 2016
Topology and data journal January 2009
Efficient k-nearest neighbor graph construction for generic similarity measures conference January 2011
NF-κB activation in human breast cancer specimens and its role in cell proliferation and apoptosis journal June 2004
Fibers of Failure: Classifying Errors in Predictive Processes journal June 2020
A weighted k-nearest neighbor density estimate for geometric inference journal January 2011
The need for uncertainty quantification in machine-assisted medical decision making journal January 2019

Similar Records

Related Subjects