skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Nonparametric ROC and LROC analysis

Abstract

In this paper we review several results of the nonparametric receiver operating characteristic (ROC) analysis and present an extension to the nonparametric localization ROC (LROC) analysis. Equations for the estimation of the area under the characteristic curve and for the variance calculations are derived. Expressions for the choice of the optimal ratio between the number of signal-absent and signal-present image samples are also presented. The results can be applied both with continuous or discrete scoring scales. The simulation studies carried out validate the theoretical derivations and show that the LROC analysis is considerably more sensitive than the ROC analysis.

Authors:
 [1]
  1. University of Pennsylvania, Department of Radiology, 423 Guardian Drive, 4th floor Blockley Hall, Philadelphia, PA 19104-6021 (United States)
Publication Date:
OSTI Identifier:
20951286
Resource Type:
Journal Article
Resource Relation:
Journal Name: Medical Physics; Journal Volume: 34; Journal Issue: 5; Other Information: DOI: 10.1118/1.2717407; (c) 2007 American Association of Physicists in Medicine; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
62 RADIOLOGY AND NUCLEAR MEDICINE; COMPUTERIZED SIMULATION; IMAGE PROCESSING; IMAGES; REVIEWS; SENSITIVITY ANALYSIS; SIGNALS

Citation Formats

Popescu, Lucretiu M. Nonparametric ROC and LROC analysis. United States: N. p., 2007. Web. doi:10.1118/1.2717407.
Popescu, Lucretiu M. Nonparametric ROC and LROC analysis. United States. doi:10.1118/1.2717407.
Popescu, Lucretiu M. Tue . "Nonparametric ROC and LROC analysis". United States. doi:10.1118/1.2717407.
@article{osti_20951286,
title = {Nonparametric ROC and LROC analysis},
author = {Popescu, Lucretiu M.},
abstractNote = {In this paper we review several results of the nonparametric receiver operating characteristic (ROC) analysis and present an extension to the nonparametric localization ROC (LROC) analysis. Equations for the estimation of the area under the characteristic curve and for the variance calculations are derived. Expressions for the choice of the optimal ratio between the number of signal-absent and signal-present image samples are also presented. The results can be applied both with continuous or discrete scoring scales. The simulation studies carried out validate the theoretical derivations and show that the LROC analysis is considerably more sensitive than the ROC analysis.},
doi = {10.1118/1.2717407},
journal = {Medical Physics},
number = 5,
volume = 34,
place = {United States},
year = {Tue May 15 00:00:00 EDT 2007},
month = {Tue May 15 00:00:00 EDT 2007}
}
  • The limitations of diagnostic accuracy as a measure of decision performance require introduction of the concepts of the sensitivity and specificity of a diagnostic test. These measures and the related indices, true positive fraction and false positive fraction, are more meaningful than accuracy, yet do not provide a unique description of diagnostic performance because they depend on the arbitrary selection of a decision threshold. The receiver operating characteristic (ROC) curve is shown to be a simple yet complete empirical description of this decision threshold effect, indicating all possible combinations of the relative frequencies of the various kinds of correct andmore » incorrect decisions. Practical experimental techniques for measuring ROC curves are described, and the issues of case selection and curve-fitting are discussed briefly. Possible generalizations of conventional ROC analysis to account for decision performance in complex diagnostic tasks are indicated. ROC analysis is shown to be related in a direct and natural way to cost/benefit analysis of diagnostic decision making. The concepts of average diagnostic cost and average net benefit are developed and used to identify the optimal compromise among various kinds of diagnostic error. Finally, the way in which ROC analysis can be employed to optimize diagnostic strategies is suggested.« less
  • Studies on the accuracy of liver scintigraphy for the detection of metastases were assembled from 38 sources in the medical literature. An ROC curve was fitted to the observed values of sensitivity and specificity using an algorithm developed by Ogilvie and Creelman. This ROC curve fitted the data better than average sensitivity and specificity values in each of four subsets of the data. For the subset dealing with Tc-99m sulfur colloid scintigraphy, performed for detection of suspected metastases and containing data on 2800 scans from 17 independent series, it was not possible to reject the hypothesis that interobserver variation wasmore » entirely due to the use of different decision thresholds by the reporting clinicians. Thus the ROC curve obtained is a reasonable baseline estimate of the performance potentially achievable in today's clinical setting. Comparison of new reports with these data is impossible, but is limited by the small sample sizes in most reported series.« less
  • The value of hTg-measurements are still discussed controversially. Also, there are no tables with sensitivity/specificity pairs for different cutoffs, since valuable information is lost by presenting only one sensitivity/specificity pair. Therefore, we wanted to determine these values for hTg by a ROC-analysis. Inclusion criteria were: (a) differentiated thyroid carcinoma (papillary or follicular), (b) no thyroid-rest after primary therapy (e.g. no detectable hTg and nc iodine accumulating foci in iodine-whole body scan), (c) no detectable hTg- antibodies, (d) an observation period of at least 1 year, hTg was determined by TG-RIA and hTg-antibodies by IMMUtest anti-Tg. Recurrence was proven by iodine-whole-bodymore » scans under maximal endogenous TSH-stimulation or by histology. Sensitivity and specificity were calculated for different hTg-cutoffs. During 10/80 and 10/89 93 patients met these criteria. There were 71 female and 22 male with a mean age of 48.6 {plus_minus} 16.3 years. 51 patients had papillary carcinomas, 42 patients follicular carcinomas. The observation period was 78.4 {plus_minus} 40.9 months. In these 93 patients there were 16 recurrences during the observation period. Sensitivity/specificity pairs for different cutoffs are given. These sensitivity/specificity pairs avoid the reduction of the original continuous data to a Yes/No decision. The knowledge of these data and of the ROC-curve could improve the intuitive rating of a laboratory result by the physician.« less
  • Receiver operator characteristic (ROC) analysis, the preferred method of evaluating diagnostic imaging tests, requires an independent assessment of the true state of disease, which can be difficult to obtain and is often of questionable accuracy. A new method of analysis is described which does not require independent truth data and which can be used when several accurate tests are being compared. This method uses correlative information to estimate the underlying model of multivariate normal distributions of disease-positive and disease-negative patients. The method is shown to give results equivalent to conventional ROC analysis in a comparison of computed tomography, radionuclide scintigraphy,more » and magnetic resonance imaging for liver metastasis. When independent truth is available, the method can be extended to incorporate truth data or to evaluate the consistency of the truth data with the imaging data.« less
  • Purpose: This work investigates the use of receiver operating characteristic (ROC) methods in patient specific IMRT quality assurance (QA) in order to determine unbiased methods to set threshold criteria for {gamma}-distance to agreement measurements. Methods: A group of 17 prostate plans was delivered as planned while a second group of 17 prostate plans was modified with the introduction of random multileaf collimator (MLC) position errors that are normally distributed with {sigma}{approx}{+-}0.5, {+-}1.0, {+-}2.0, and {+-}3.0 mm (a total of 68 modified plans were created). All plans were evaluated using five different {gamma}-criteria. ROC methodology was applied by quantifying the fractionmore » of modified plans reported as 'fail' and unmodified plans reported as 'pass.'Results: {gamma}-based criteria were able to attain nearly 100% sensitivity/specificity in the detection of large random errors ({sigma} > 3 mm). Sensitivity and specificity decrease rapidly for all {gamma}-criteria as the size of error to be detected decreases below 2 mm. Predictive power is null with all criteria used in the detection of small MLC errors ({sigma} < 0.5 mm). Optimal threshold values were established by determining which criteria maximized sensitivity and specificity. For 3%/3 mm {gamma}-criteria, optimal threshold values range from 92% to 99%, whereas for 2%/2 mm, the range was from 77% to 94%. Conclusions: The optimal threshold values that were determined represent a maximized test sensitivity and specificity and are not subject to any user bias. When applied to the datasets that we studied, our results suggest the use of patient specific QA as a safety tool that can effectively prevent large errors (e.g., {sigma} > 3 mm) as opposed to a tool to improve the quality of IMRT delivery.« less