skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Reliability analysis framework for computer-assisted medical decision systems

Abstract

We present a technique that enhances computer-assisted decision (CAD) systems with the ability to assess the reliability of each individual decision they make. Reliability assessment is achieved by measuring the accuracy of a CAD system with known cases similar to the one in question. The proposed technique analyzes the feature space neighborhood of the query case to dynamically select an input-dependent set of known cases relevant to the query. This set is used to assess the local (query-specific) accuracy of the CAD system. The estimated local accuracy is utilized as a reliability measure of the CAD response to the query case. The underlying hypothesis of the study is that CAD decisions with higher reliability are more accurate. The above hypothesis was tested using a mammographic database of 1337 regions of interest (ROIs) with biopsy-proven ground truth (681 with masses, 656 with normal parenchyma). Three types of decision models, (i) a back-propagation neural network (BPNN), (ii) a generalized regression neural network (GRNN), and (iii) a support vector machine (SVM), were developed to detect masses based on eight morphological features automatically extracted from each ROI. The performance of all decision models was evaluated using the Receiver Operating Characteristic (ROC) analysis. The studymore » showed that the proposed reliability measure is a strong predictor of the CAD system's case-specific accuracy. Specifically, the ROC area index for CAD predictions with high reliability was significantly better than for those with low reliability values. This result was consistent across all decision models investigated in the study. The proposed case-specific reliability analysis technique could be used to alert the CAD user when an opinion that is unlikely to be reliable is offered. The technique can be easily deployed in the clinical environment because it is applicable with a wide range of classifiers regardless of their structure and it requires neither additional training nor building multiple decision models to assess the case-specific CAD accuracy.« less

Authors:
; ; ;  [1];  [2];  [2]
  1. Computational Intelligence Laboratory, Department of Electrical and Computer Engineering, University of Louisville, Louisville, Kentucky 40292 (United States)
  2. (United States)
Publication Date:
OSTI Identifier:
20951065
Resource Type:
Journal Article
Resource Relation:
Journal Name: Medical Physics; Journal Volume: 34; Journal Issue: 2; Other Information: DOI: 10.1118/1.2432409; (c) 2007 American Association of Physicists in Medicine; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
62 RADIOLOGY AND NUCLEAR MEDICINE; ACCURACY; ARTIFICIAL INTELLIGENCE; BIOMEDICAL RADIOGRAPHY; BIOPSY; DECISION MAKING; GROUND TRUTH MEASUREMENTS; MAMMARY GLANDS; NEURAL NETWORKS; REGRESSION ANALYSIS; RELIABILITY; SENSITIVITY ANALYSIS; TRAINING

Citation Formats

Habas, Piotr A., Zurada, Jacek M., Elmaghraby, Adel S., Tourassi, Georgia D., Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292, and Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705. Reliability analysis framework for computer-assisted medical decision systems. United States: N. p., 2007. Web. doi:10.1118/1.2432409.
Habas, Piotr A., Zurada, Jacek M., Elmaghraby, Adel S., Tourassi, Georgia D., Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292, & Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705. Reliability analysis framework for computer-assisted medical decision systems. United States. doi:10.1118/1.2432409.
Habas, Piotr A., Zurada, Jacek M., Elmaghraby, Adel S., Tourassi, Georgia D., Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292, and Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705. Thu . "Reliability analysis framework for computer-assisted medical decision systems". United States. doi:10.1118/1.2432409.
@article{osti_20951065,
title = {Reliability analysis framework for computer-assisted medical decision systems},
author = {Habas, Piotr A. and Zurada, Jacek M. and Elmaghraby, Adel S. and Tourassi, Georgia D. and Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292 and Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705},
abstractNote = {We present a technique that enhances computer-assisted decision (CAD) systems with the ability to assess the reliability of each individual decision they make. Reliability assessment is achieved by measuring the accuracy of a CAD system with known cases similar to the one in question. The proposed technique analyzes the feature space neighborhood of the query case to dynamically select an input-dependent set of known cases relevant to the query. This set is used to assess the local (query-specific) accuracy of the CAD system. The estimated local accuracy is utilized as a reliability measure of the CAD response to the query case. The underlying hypothesis of the study is that CAD decisions with higher reliability are more accurate. The above hypothesis was tested using a mammographic database of 1337 regions of interest (ROIs) with biopsy-proven ground truth (681 with masses, 656 with normal parenchyma). Three types of decision models, (i) a back-propagation neural network (BPNN), (ii) a generalized regression neural network (GRNN), and (iii) a support vector machine (SVM), were developed to detect masses based on eight morphological features automatically extracted from each ROI. The performance of all decision models was evaluated using the Receiver Operating Characteristic (ROC) analysis. The study showed that the proposed reliability measure is a strong predictor of the CAD system's case-specific accuracy. Specifically, the ROC area index for CAD predictions with high reliability was significantly better than for those with low reliability values. This result was consistent across all decision models investigated in the study. The proposed case-specific reliability analysis technique could be used to alert the CAD user when an opinion that is unlikely to be reliable is offered. The technique can be easily deployed in the clinical environment because it is applicable with a wide range of classifiers regardless of their structure and it requires neither additional training nor building multiple decision models to assess the case-specific CAD accuracy.},
doi = {10.1118/1.2432409},
journal = {Medical Physics},
number = 2,
volume = 34,
place = {United States},
year = {Thu Feb 15 00:00:00 EST 2007},
month = {Thu Feb 15 00:00:00 EST 2007}
}