skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems

Abstract

Purpose: Incident learning systems encompass volumes, varieties, values, and velocities of underlying data elements consistent with the V’s of big data. Veracity, the 5th V however exists only if there is high inter-rater reliability (IRR) within the data elements. The purpose of this work was to assess IRR in the nationally deployed RO-ILS: Radiation Oncology-Incident Learning System (R) sponsored by the American Society for Radiation Oncology (ASTRO) and the American Association of Physicists in Medicine (AAPM). Methods: Ten incident reports covering a wide range of scenarios were created in standardized narrative and video formats and disseminated to 67 volunteers of multiple disciplines from 26 institutions along with two published narratives from the International Commission of Radiological Protection to assess IRR on a nationally representative level. The volunteers were instructed to independently enter the associated data elements in a test version of RO-ILS over a 3-week period. All responses were aggregated into a spreadsheet to assess IRR using free-marginal kappa metrics. Results: 48 volunteers from 21 institutions completed all reports in the study period. The average kappa score for all raters across all critical data elements was 0.659 [range 0.326–1.000]. Statistically significant differences (p <0.05) were noted between reporters of differentmore » disciplines and raters with varying levels of experience. Kappa scores were high for event classification (0.781) and contributory factors (0.777) and low for likelihood-of-harm (0.326). IRR was highest among AAPM-ASTRO members (0.672) and lowest among trainees (0.463). Conclusion: A moderate-to-substantial level of IRR in RO-ILS was noted in this study. Although the number of events reviewed in this study was small, opportunities for improving the taxonomy for the lower scoring data elements as well as specific educational targets for training were identified by assessing data veracity quantitatively. This is expected to improve the quality of the data garnered from RO-ILS.« less

Authors:
 [1];  [2];  [3];  [4];  [5];  [6]; ;  [7]
  1. Northwell Health System, New Hyde Park, NY (United States)
  2. Yale University New Haven, CT (United States)
  3. University of California, San Diego, La Jolla, CA (United States)
  4. Mayo Clinic Arizona, Phoenix, AZ (United States)
  5. The University of California San Diego, San Diego, CA (United States)
  6. UC Davis Medical Center, Sacramento, CA (United States)
  7. American Society for Radiation Oncology, Fairfax, VA (United States)
Publication Date:
OSTI Identifier:
22653968
Resource Type:
Journal Article
Resource Relation:
Journal Name: Medical Physics; Journal Volume: 43; Journal Issue: 6; Other Information: (c) 2016 American Association of Physicists in Medicine; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
61 RADIATION PROTECTION AND DOSIMETRY; 60 APPLIED LIFE SCIENCES; ACCIDENTS; CLASSIFICATION; COST; DRUGS; ECONOMICS; FORMATES; LEARNING; MEDICINE; RADIATION PROTECTION; RELIABILITY; REVIEWS; TRAINING

Citation Formats

Kapur, A, Evans, S, Brown, D, Ezzell, G, Hoopes, D, Dieterich, S, Kapetanovic, K, and Tomlinson, C. TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems. United States: N. p., 2016. Web. doi:10.1118/1.4957470.
Kapur, A, Evans, S, Brown, D, Ezzell, G, Hoopes, D, Dieterich, S, Kapetanovic, K, & Tomlinson, C. TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems. United States. doi:10.1118/1.4957470.
Kapur, A, Evans, S, Brown, D, Ezzell, G, Hoopes, D, Dieterich, S, Kapetanovic, K, and Tomlinson, C. Wed . "TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems". United States. doi:10.1118/1.4957470.
@article{osti_22653968,
title = {TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems},
author = {Kapur, A and Evans, S and Brown, D and Ezzell, G and Hoopes, D and Dieterich, S and Kapetanovic, K and Tomlinson, C},
abstractNote = {Purpose: Incident learning systems encompass volumes, varieties, values, and velocities of underlying data elements consistent with the V’s of big data. Veracity, the 5th V however exists only if there is high inter-rater reliability (IRR) within the data elements. The purpose of this work was to assess IRR in the nationally deployed RO-ILS: Radiation Oncology-Incident Learning System (R) sponsored by the American Society for Radiation Oncology (ASTRO) and the American Association of Physicists in Medicine (AAPM). Methods: Ten incident reports covering a wide range of scenarios were created in standardized narrative and video formats and disseminated to 67 volunteers of multiple disciplines from 26 institutions along with two published narratives from the International Commission of Radiological Protection to assess IRR on a nationally representative level. The volunteers were instructed to independently enter the associated data elements in a test version of RO-ILS over a 3-week period. All responses were aggregated into a spreadsheet to assess IRR using free-marginal kappa metrics. Results: 48 volunteers from 21 institutions completed all reports in the study period. The average kappa score for all raters across all critical data elements was 0.659 [range 0.326–1.000]. Statistically significant differences (p <0.05) were noted between reporters of different disciplines and raters with varying levels of experience. Kappa scores were high for event classification (0.781) and contributory factors (0.777) and low for likelihood-of-harm (0.326). IRR was highest among AAPM-ASTRO members (0.672) and lowest among trainees (0.463). Conclusion: A moderate-to-substantial level of IRR in RO-ILS was noted in this study. Although the number of events reviewed in this study was small, opportunities for improving the taxonomy for the lower scoring data elements as well as specific educational targets for training were identified by assessing data veracity quantitatively. This is expected to improve the quality of the data garnered from RO-ILS.},
doi = {10.1118/1.4957470},
journal = {Medical Physics},
number = 6,
volume = 43,
place = {United States},
year = {Wed Jun 15 00:00:00 EDT 2016},
month = {Wed Jun 15 00:00:00 EDT 2016}
}