skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Reliability analysis framework for computer-assisted medical decision systems

Abstract

We present a technique that enhances computer-assisted decision (CAD) systems with the ability to assess the reliability of each individual decision they make. Reliability assessment is achieved by measuring the accuracy of a CAD system with known cases similar to the one in question. The proposed technique analyzes the feature space neighborhood of the query case to dynamically select an input-dependent set of known cases relevant to the query. This set is used to assess the local (query-specific) accuracy of the CAD system. The estimated local accuracy is utilized as a reliability measure of the CAD response to the query case. The underlying hypothesis of the study is that CAD decisions with higher reliability are more accurate. The above hypothesis was tested using a mammographic database of 1337 regions of interest (ROIs) with biopsy-proven ground truth (681 with masses, 656 with normal parenchyma). Three types of decision models, (i) a back-propagation neural network (BPNN), (ii) a generalized regression neural network (GRNN), and (iii) a support vector machine (SVM), were developed to detect masses based on eight morphological features automatically extracted from each ROI. The performance of all decision models was evaluated using the Receiver Operating Characteristic (ROC) analysis. The studymore » showed that the proposed reliability measure is a strong predictor of the CAD system's case-specific accuracy. Specifically, the ROC area index for CAD predictions with high reliability was significantly better than for those with low reliability values. This result was consistent across all decision models investigated in the study. The proposed case-specific reliability analysis technique could be used to alert the CAD user when an opinion that is unlikely to be reliable is offered. The technique can be easily deployed in the clinical environment because it is applicable with a wide range of classifiers regardless of their structure and it requires neither additional training nor building multiple decision models to assess the case-specific CAD accuracy.« less

Authors:
; ; ;  [1];  [2];  [2]
  1. Computational Intelligence Laboratory, Department of Electrical and Computer Engineering, University of Louisville, Louisville, Kentucky 40292 (United States)
  2. (United States)
Publication Date:
OSTI Identifier:
20951065
Resource Type:
Journal Article
Resource Relation:
Journal Name: Medical Physics; Journal Volume: 34; Journal Issue: 2; Other Information: DOI: 10.1118/1.2432409; (c) 2007 American Association of Physicists in Medicine; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
62 RADIOLOGY AND NUCLEAR MEDICINE; ACCURACY; ARTIFICIAL INTELLIGENCE; BIOMEDICAL RADIOGRAPHY; BIOPSY; DECISION MAKING; GROUND TRUTH MEASUREMENTS; MAMMARY GLANDS; NEURAL NETWORKS; REGRESSION ANALYSIS; RELIABILITY; SENSITIVITY ANALYSIS; TRAINING

Citation Formats

Habas, Piotr A., Zurada, Jacek M., Elmaghraby, Adel S., Tourassi, Georgia D., Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292, and Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705. Reliability analysis framework for computer-assisted medical decision systems. United States: N. p., 2007. Web. doi:10.1118/1.2432409.
Habas, Piotr A., Zurada, Jacek M., Elmaghraby, Adel S., Tourassi, Georgia D., Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292, & Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705. Reliability analysis framework for computer-assisted medical decision systems. United States. doi:10.1118/1.2432409.
Habas, Piotr A., Zurada, Jacek M., Elmaghraby, Adel S., Tourassi, Georgia D., Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292, and Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705. Thu . "Reliability analysis framework for computer-assisted medical decision systems". United States. doi:10.1118/1.2432409.
@article{osti_20951065,
title = {Reliability analysis framework for computer-assisted medical decision systems},
author = {Habas, Piotr A. and Zurada, Jacek M. and Elmaghraby, Adel S. and Tourassi, Georgia D. and Department of Computer Engineering and Computer Science, University of Louisville, Louisville, Kentucky 40292 and Digital Advanced Imaging Laboratories, Department of Radiology, Duke University Medical Center, Durham, North Carolina 27705},
abstractNote = {We present a technique that enhances computer-assisted decision (CAD) systems with the ability to assess the reliability of each individual decision they make. Reliability assessment is achieved by measuring the accuracy of a CAD system with known cases similar to the one in question. The proposed technique analyzes the feature space neighborhood of the query case to dynamically select an input-dependent set of known cases relevant to the query. This set is used to assess the local (query-specific) accuracy of the CAD system. The estimated local accuracy is utilized as a reliability measure of the CAD response to the query case. The underlying hypothesis of the study is that CAD decisions with higher reliability are more accurate. The above hypothesis was tested using a mammographic database of 1337 regions of interest (ROIs) with biopsy-proven ground truth (681 with masses, 656 with normal parenchyma). Three types of decision models, (i) a back-propagation neural network (BPNN), (ii) a generalized regression neural network (GRNN), and (iii) a support vector machine (SVM), were developed to detect masses based on eight morphological features automatically extracted from each ROI. The performance of all decision models was evaluated using the Receiver Operating Characteristic (ROC) analysis. The study showed that the proposed reliability measure is a strong predictor of the CAD system's case-specific accuracy. Specifically, the ROC area index for CAD predictions with high reliability was significantly better than for those with low reliability values. This result was consistent across all decision models investigated in the study. The proposed case-specific reliability analysis technique could be used to alert the CAD user when an opinion that is unlikely to be reliable is offered. The technique can be easily deployed in the clinical environment because it is applicable with a wide range of classifiers regardless of their structure and it requires neither additional training nor building multiple decision models to assess the case-specific CAD accuracy.},
doi = {10.1118/1.2432409},
journal = {Medical Physics},
number = 2,
volume = 34,
place = {United States},
year = {Thu Feb 15 00:00:00 EST 2007},
month = {Thu Feb 15 00:00:00 EST 2007}
}
  • A new framework, Decerns, for multicriteria decision analysis (MCDA) of a wide range of practical problems on risk management is introduced. Decerns framework contains a library of modules that are the basis for two scalable systems: DecernsMCDA for analysis of multicriteria problems, and DecernsSDSS for multicriteria analysis of spatial options. DecernsMCDA includes well known MCDA methods and original methods for uncertainty treatment based on probabilistic approaches and fuzzy numbers. As a result, these MCDA methods are described along with a case study on analysis of multicriteria location problem.
  • A computer program was developed to aid decision makers in choosing among alternatives. It facilitiates the implementation of the decision analysis approach to multiobjective decision-making problems. The program's main functions are to store the information and perform all the necessary computations required by the approach. The program is designed so that only a few basic commands need to be understood in order to use it effectively. The style of input can be both batch and interactively oriented. Detailed specification of preferences and alternatives is usually done in batch mode while sensitivity analysis can be performed interactively. The output consists ofmore » ranking, preference and alternative information displays. The program is quite general and should be applicable to a wide variety of problems. The code allows for an interface to user supplied models when that is desirable. It is designed to run on most computer systems without or with very minor system-specific modifications. This report presents a user's manual for the program that includes a simple illustrative example.« less
  • This paper reviews major developments in continuous haulage technology for underground coal mines. Haulage systems in use and development are investigated through a comprehensive literature search and visits to mines and manufacturers. The systems include flexible conveyor train, mobile conveyor, multiple-unit continuous haulage, extensible and belt turning systems. Strengths and weaknesses of the systems are assessed in light of their operating under remote control and their application to the computer-assisted continuous mining machine being developed by the US Bureau of Mines. Through mining scenarios, candidate haulage systems are conceptualized to be consistent with the capabilities of the mining machine.
  • The lower voltage part of the power distribution system (primary and secondary sub-systems) does not have the provision for real-time status feedback, and as a result evaluation of outages is an extremely difficult task, especially during system emergencies caused by tornadoes and ice-storms. In this paper, a knowledge based approach is proposed for evaluation of storm related outages in the distribution systems. At the outset, binary voltage sensors capable of transmitting the real-time voltage on/off symptoms are recommended to be installed at strategic locations in the distribution system.