Comparative analysis of data collection methods for individualized modeling of radiologists' visual similarity judgments

Tourassi, Georgia; Xu, Songhua; Yoon, Hong-Jun; Morin-Ducote, Garnetta; Hudson, Kathy

doi:10.1016/j.acra.2013.08.002

Title: Comparative analysis of data collection methods for individualized modeling of radiologists' visual similarity judgments

Full Record
Other Related Research

Abstract

Rationale and Objectives: We conducted an observer study to investigate how the data collection method affects the efficacy of modeling individual radiologists judgments regarding the perceptual similarity of breast masses on mammograms. Materials and Methods: Institutional review board approval was obtained prior to the study. Six observers of variable experience levels in breast imaging were recruited to assess the perceptual similarity of mammographic masses. The observers subjective judgments were collected using: (i) a rating method, (ii) a preference method, and (iii) a hybrid method combining rating and ranking. Personalized user models were developed with the collected data to predict observers opinions. The relative efficacy of each data collection method was assessed based on the classification accuracy of the resulting user models. Results: The hybrid data collection method produced significantly more accurate individualized user models of perceptual opinions with comparable and sometimes better time efficiency than the other two data collection methods. The user models derived from hybrid data were clearly superior even when developed with a dramatically smaller number of training cases. Conclusions: A hybrid method combining rating and ranking is an intuitive and efficient way for collecting subjective similarity judgments to model human perceptual opinions with a higher accuracymore »« less

Authors:

Tourassi, Georgia ^[1]; Xu, Songhua ^[1]; Yoon, Hong-Jun ^[1]; Morin-Ducote, Garnetta ^[2]; Hudson, Kathy ^[2]

ORNL
University of Tennessee, Knoxville (UTK)

Publication Date:: Tue Jan 01 00:00:00 EST 2013

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Laboratory Directed Research and Development (LDRD) Program

OSTI Identifier:: 1097477

DOE Contract Number:: DE-AC05-00OR22725

Resource Type:: Journal Article

Journal Name:: Academic Radiology

Additional Journal Information:: Journal Volume: 20; Journal Issue: 11; Journal ID: ISSN 1076-6332

Country of Publication:: United States

Language:: English

Subject:: breast imaging; mammography; observer variability; perception; visual similarity user modeling

Citation Formats


                    Tourassi, Georgia, Xu, Songhua, Yoon, Hong-Jun, Morin-Ducote, Garnetta, and Hudson, Kathy. Comparative analysis of data collection methods for individualized modeling of radiologists' visual similarity judgments.  United States: N. p., 2013. 
        Web.  doi:10.1016/j.acra.2013.08.002.

Copy to clipboard


                    Tourassi, Georgia, Xu, Songhua, Yoon, Hong-Jun, Morin-Ducote, Garnetta, & Hudson, Kathy. Comparative analysis of data collection methods for individualized modeling of radiologists' visual similarity judgments.  United States.  https://doi.org/10.1016/j.acra.2013.08.002

Copy to clipboard


                    Tourassi, Georgia, Xu, Songhua, Yoon, Hong-Jun, Morin-Ducote, Garnetta, and Hudson, Kathy. 2013.  
        "Comparative analysis of data collection methods for individualized modeling of radiologists' visual similarity judgments".  United States.  https://doi.org/10.1016/j.acra.2013.08.002.

Copy to clipboard


                    
@article{osti_1097477,

  title        = {Comparative analysis of data collection methods for individualized modeling of radiologists' visual similarity judgments},

  author       = {Tourassi, Georgia and Xu, Songhua and Yoon, Hong-Jun and Morin-Ducote, Garnetta and Hudson, Kathy},

  abstractNote = {Rationale and Objectives: We conducted an observer study to investigate how the data collection method affects the efficacy of modeling individual radiologists judgments regarding the perceptual similarity of breast masses on mammograms. Materials and Methods: Institutional review board approval was obtained prior to the study. Six observers of variable experience levels in breast imaging were recruited to assess the perceptual similarity of mammographic masses. The observers subjective judgments were collected using: (i) a rating method, (ii) a preference method, and (iii) a hybrid method combining rating and ranking. Personalized user models were developed with the collected data to predict observers opinions. The relative efficacy of each data collection method was assessed based on the classification accuracy of the resulting user models. Results: The hybrid data collection method produced significantly more accurate individualized user models of perceptual opinions with comparable and sometimes better time efficiency than the other two data collection methods. The user models derived from hybrid data were clearly superior even when developed with a dramatically smaller number of training cases. Conclusions: A hybrid method combining rating and ranking is an intuitive and efficient way for collecting subjective similarity judgments to model human perceptual opinions with a higher accuracy than other more commonly used data collection methods.},

  doi          = {10.1016/j.acra.2013.08.002},

  url          = {https://www.osti.gov/biblio/1097477},
  journal      = {Academic Radiology},

  issn         = {1076-6332},

  number       = 11,

  volume       = 20,

  place        = {United States},

  year         = {Tue Jan 01 00:00:00 EST 2013},

  month        = {Tue Jan 01 00:00:00 EST 2013}

}

Copy to clipboard

Journal Article:

https://doi.org/10.1016/j.acra.2013.08.002

Other availability

Find in Google Scholar

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

A Novel Graphical User Interface for High-Efficacy Modeling of Human Perceptual Similarity Opinions

Conference Kress, James; Xu, Songhua; Tourassi, Georgia

We present a novel graphical user interface (GUI) that facilitates high-efficacy collection of perceptual similarity opinions of a user in an effective and intuitive manner. The GUI is based on a hybrid mechanism that combines ranking and rating. Namely, it presents a base image for rating its similarity to seven peripheral images that are displayed simultaneously following a circular layout. The user is asked to report the base image s pairwise similarity to each peripheral image on a fixed scale while preserving the relative ranking among all peripheral images. The collected data are then used to predict the user smore »« less
Investigating the Link Between Radiologists Gaze, Diagnostic Decision, and Image Content

Journal Article Tourassi, Georgia; Voisin, Sophie; Paquit, Vincent; ... - Journal of American Medical Informatics Association

Objective: To investigate machine learning for linking image content, human perception, cognition, and error in the diagnostic interpretation of mammograms. Methods: Gaze data and diagnostic decisions were collected from six radiologists who reviewed 20 screening mammograms while wearing a head-mounted eye-tracker. Texture analysis was performed in mammographic regions that attracted radiologists attention and in all abnormal regions. Machine learning algorithms were investigated to develop predictive models that link: (i) image content with gaze, (ii) image content and gaze with cognition, and (iii) image content, gaze, and cognition with diagnostic error. Both group-based and individualized models were explored. Results: By poolingmore »« less
https://doi.org/10.1136/amiajnl-2012-001503
MO-F-CAMPUS-I-01: Accuracy of Radiologists Interpretation of Mammographic Breast Density

Journal Article Vedantham, S; Shi, L; Karellas, A; ... - Medical Physics

Purpose: Several commercial and non-commercial software and techniques are available for determining breast density from mammograms. However, where mandated by law the breast density information communicated to the subject/patient is based on radiologist’s interpretation of breast density from mammograms. Several studies have reported on the concordance among radiologists in interpreting mammographic breast density. In this work, we investigated the accuracy of radiologist’s interpretation of breast density. Methods: Volumetric breast density (VBD) determined from 134 unilateral dedicated breast CT scans from 134 subjects was considered the truth. An MQSA-qualified study radiologist with more than 20 years of breast imaging experience reviewedmore »« less
https://doi.org/10.1118/1.4925452
Determination of subjective similarity for pairs of masses and pairs of clustered microcalcifications on mammograms: Comparison of similarity ranking scores and absolute similarity ratings

Journal Article Muramatsu, Chisako; Qiang, Li; Schmidt, Robert; ... - Medical Physics

The presentation of images that are similar to that of an unknown lesion seen on a mammogram may be helpful for radiologists to correctly diagnose that lesion. For similar images to be useful, they must be quite similar from the radiologists' point of view. We have been trying to quantify the radiologists' impression of similarity for pairs of lesions and to establish a ''gold standard'' for development and evaluation of a computerized scheme for selecting such similar images. However, it is considered difficult to reliably and accurately determine similarity ratings, because they are subjective. In this study, we compared themore »« less
https://doi.org/10.1118/1.2745937
Investigation of psychophysical similarity measures for selection of similar images in the diagnosis of clustered microcalcifications on mammograms

Journal Article Muramatsu, Chisako; Qiang, Li; Schmidt, Robert; ... - Medical Physics

The presentation of images with lesions of known pathology that are similar to an unknown lesion may be helpful to radiologists in the diagnosis of challenging cases for improving the diagnostic accuracy and also for reducing variation among different radiologists. The authors have been developing a computerized scheme for automatically selecting similar images with clustered microcalcifications on mammograms from a large database. For similar images to be useful, they must be similar from the point of view of the diagnosing radiologists. In order to select such images, subjective similarity ratings were obtained for a number of pairs of clustered microcalcificationsmore »« less
https://doi.org/10.1118/1.3020760

Similar Records