skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems

Abstract

Purpose: Incident learning systems encompass volumes, varieties, values, and velocities of underlying data elements consistent with the V’s of big data. Veracity, the 5th V however exists only if there is high inter-rater reliability (IRR) within the data elements. The purpose of this work was to assess IRR in the nationally deployed RO-ILS: Radiation Oncology-Incident Learning System (R) sponsored by the American Society for Radiation Oncology (ASTRO) and the American Association of Physicists in Medicine (AAPM). Methods: Ten incident reports covering a wide range of scenarios were created in standardized narrative and video formats and disseminated to 67 volunteers of multiple disciplines from 26 institutions along with two published narratives from the International Commission of Radiological Protection to assess IRR on a nationally representative level. The volunteers were instructed to independently enter the associated data elements in a test version of RO-ILS over a 3-week period. All responses were aggregated into a spreadsheet to assess IRR using free-marginal kappa metrics. Results: 48 volunteers from 21 institutions completed all reports in the study period. The average kappa score for all raters across all critical data elements was 0.659 [range 0.326–1.000]. Statistically significant differences (p <0.05) were noted between reporters of differentmore » disciplines and raters with varying levels of experience. Kappa scores were high for event classification (0.781) and contributory factors (0.777) and low for likelihood-of-harm (0.326). IRR was highest among AAPM-ASTRO members (0.672) and lowest among trainees (0.463). Conclusion: A moderate-to-substantial level of IRR in RO-ILS was noted in this study. Although the number of events reviewed in this study was small, opportunities for improving the taxonomy for the lower scoring data elements as well as specific educational targets for training were identified by assessing data veracity quantitatively. This is expected to improve the quality of the data garnered from RO-ILS.« less

Authors:
 [1];  [2];  [3];  [4];  [5];  [6]; ;  [7]
  1. Northwell Health System, New Hyde Park, NY (United States)
  2. Yale University New Haven, CT (United States)
  3. University of California, San Diego, La Jolla, CA (United States)
  4. Mayo Clinic Arizona, Phoenix, AZ (United States)
  5. The University of California San Diego, San Diego, CA (United States)
  6. UC Davis Medical Center, Sacramento, CA (United States)
  7. American Society for Radiation Oncology, Fairfax, VA (United States)
Publication Date:
OSTI Identifier:
22653968
Resource Type:
Journal Article
Resource Relation:
Journal Name: Medical Physics; Journal Volume: 43; Journal Issue: 6; Other Information: (c) 2016 American Association of Physicists in Medicine; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
61 RADIATION PROTECTION AND DOSIMETRY; 60 APPLIED LIFE SCIENCES; ACCIDENTS; CLASSIFICATION; COST; DRUGS; ECONOMICS; FORMATES; LEARNING; MEDICINE; RADIATION PROTECTION; RELIABILITY; REVIEWS; TRAINING

Citation Formats

Kapur, A, Evans, S, Brown, D, Ezzell, G, Hoopes, D, Dieterich, S, Kapetanovic, K, and Tomlinson, C. TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems. United States: N. p., 2016. Web. doi:10.1118/1.4957470.
Kapur, A, Evans, S, Brown, D, Ezzell, G, Hoopes, D, Dieterich, S, Kapetanovic, K, & Tomlinson, C. TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems. United States. doi:10.1118/1.4957470.
Kapur, A, Evans, S, Brown, D, Ezzell, G, Hoopes, D, Dieterich, S, Kapetanovic, K, and Tomlinson, C. 2016. "TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems". United States. doi:10.1118/1.4957470.
@article{osti_22653968,
title = {TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems},
author = {Kapur, A and Evans, S and Brown, D and Ezzell, G and Hoopes, D and Dieterich, S and Kapetanovic, K and Tomlinson, C},
abstractNote = {Purpose: Incident learning systems encompass volumes, varieties, values, and velocities of underlying data elements consistent with the V’s of big data. Veracity, the 5th V however exists only if there is high inter-rater reliability (IRR) within the data elements. The purpose of this work was to assess IRR in the nationally deployed RO-ILS: Radiation Oncology-Incident Learning System (R) sponsored by the American Society for Radiation Oncology (ASTRO) and the American Association of Physicists in Medicine (AAPM). Methods: Ten incident reports covering a wide range of scenarios were created in standardized narrative and video formats and disseminated to 67 volunteers of multiple disciplines from 26 institutions along with two published narratives from the International Commission of Radiological Protection to assess IRR on a nationally representative level. The volunteers were instructed to independently enter the associated data elements in a test version of RO-ILS over a 3-week period. All responses were aggregated into a spreadsheet to assess IRR using free-marginal kappa metrics. Results: 48 volunteers from 21 institutions completed all reports in the study period. The average kappa score for all raters across all critical data elements was 0.659 [range 0.326–1.000]. Statistically significant differences (p <0.05) were noted between reporters of different disciplines and raters with varying levels of experience. Kappa scores were high for event classification (0.781) and contributory factors (0.777) and low for likelihood-of-harm (0.326). IRR was highest among AAPM-ASTRO members (0.672) and lowest among trainees (0.463). Conclusion: A moderate-to-substantial level of IRR in RO-ILS was noted in this study. Although the number of events reviewed in this study was small, opportunities for improving the taxonomy for the lower scoring data elements as well as specific educational targets for training were identified by assessing data veracity quantitatively. This is expected to improve the quality of the data garnered from RO-ILS.},
doi = {10.1118/1.4957470},
journal = {Medical Physics},
number = 6,
volume = 43,
place = {United States},
year = 2016,
month = 6
}
  • Purpose: Describe a Web-based Radiation Oncology Incident Reporting and Learning system that has the potential to improve quality of care for radiation therapy patients. This system is an important facet of continuing effort by our community to maintain and improve safety of radiotherapy.Material and Methods: The VA National Radiation Oncology Program office has embarked on a program to electronically collect adverse events and near miss data of radiation treatment of over 25,000 veterans treated with radiotherapy annually. Software used for this program is deployed on the VAs intranet as a Website. All data entry forms (adverse event or near missmore » reports, work product reports) utilize standard causal, RT process step taxonomies and data dictionaries defined in AAPM and ASTRO reports on error reporting (AAPM Work Group Report on Prevention of Errors and ASTROs safety is no accident report). All reported incidents are investigated by the radiation oncology domain experts. This system encompasses the entire feedback loop of reporting an incident, analyzing it for salient details, and developing interventions to prevent it from happening again. The operational workflow is similar to that of the Aviation Safety Reporting System. This system is also synergistic with ROSIS and SAFRON. Results: The ROIRLS facilitates the collection of data that help in tracking adverse events and near misses and develop new interventions to prevent such incidents. The ROIRLS electronic infrastructure is fully integrated with each registered facility profile data thus minimizing key strokes and multiple entries by the event reporters. Conclusions: OIRLS is expected to improve the quality and safety of a broad spectrum of radiation therapy patients treated in the VA and fulfills our goal of Effecting Quality While Treating Safely The Radiation Oncology Incident Reporting and Learning System software used for this program has been developed, conceptualized and maintained by TSG Innovations Inc. and is deployed on the VA intranet as a Website. The Radiation Oncology Incident Reporting and Learning System software used for this program has been developed, conceptualized and maintained by TSG Innovations Inc. and is deployed on the VA intranet as a Website.« less
  • Purpose: To transition from an in-house incident reporting system to a ROILS standards system with the intent to develop a safety focused culture in the Department and enroll in ROILS. Methods: Since the AAPM Safety Summit (2010) several safety and reporting systems have been implemented within the Department. Specific checklists and SBAR reporting systems were introduced. However, the active learning component was lost due to reporting being viewed with distrust and possible retribution.To Facilitate introducing ROILS each leader in the Department received a copy of the ROILS participation guide. Four specific tasks were assigned to each leader: develop a reportingmore » tree, begin the ROILS based system, facilitate adopting ROILS Terminology, and educate the staff on expectations of safety culture. Next, the ROILS questions were broken down into area specific questions (10–15) per departmental area. Excel spreadsheets were developed for each area and setup for error reporting entries. The Role of the Process Improvement Committee (PI) has been modified to review and make recommendations based on the ROILS entries. Results: The ROILS based Reporting has been in place for 4 months. To date 64 reports have been entered. Since the adoption of ROILS the reporting of incidents has increased from 2/month to 18/month on average. Three reports had a dosimetric effect on the patient (<5%) dose variance. The large majority of entries have been Characterized as Processes not followed or not sure how to Characterize, and Human Behavior. Conclusion: The majority of errors are typo’s that create confusion. The introduction of the ROILS standards has provided a platform for making changes to policies that increase patient safety. The goal is to develop a culture that sees reporting at a national level as a safe and effective way to improve our safety, and to dynamically learn from other institutions reporting.« less
  • Purpose: Though FMEA (Failure Mode and Effects Analysis) is becoming more widely adopted for risk assessment in radiation therapy, to our knowledge it has never been validated against actual incident learning data. The objective of this study was to perform an FMEA analysis of an SBRT (Stereotactic Body Radiation Therapy) treatment planning process and validate this against data recorded within an incident learning system. Methods: FMEA on the SBRT treatment planning process was carried out by a multidisciplinary group including radiation oncologists, medical physicists, and dosimetrists. Potential failure modes were identified through a systematic review of the workflow process. Failuremore » modes were rated for severity, occurrence, and detectability on a scale of 1 to 10 and RPN (Risk Priority Number) was computed. Failure modes were then compared with historical reports identified as relevant to SBRT planning within a departmental incident learning system that had been active for two years. Differences were identified. Results: FMEA identified 63 failure modes. RPN values for the top 25% of failure modes ranged from 60 to 336. Analysis of the incident learning database identified 33 reported near-miss events related to SBRT planning. FMEA failed to anticipate 13 of these events, among which 3 were registered with severity ratings of severe or critical in the incident learning system. Combining both methods yielded a total of 76 failure modes, and when scored for RPN the 13 events missed by FMEA ranked within the middle half of all failure modes. Conclusion: FMEA, though valuable, is subject to certain limitations, among them the limited ability to anticipate all potential errors for a given process. This FMEA exercise failed to identify a significant number of possible errors (17%). Integration of FMEA with retrospective incident data may be able to render an improved overview of risks within a process.« less
  • Purpose: Radiation oncologists are faced with the challenge of irradiating tumors to a curative dose while limiting toxicity to healthy surrounding tissues. This can be achieved only with superior knowledge of radiologic anatomy and treatment planning. Educational resources designed to meet these specific needs are lacking. A web-based interactive module designed to improve residents' knowledge and application of key anatomy concepts pertinent to radiotherapy treatment planning was developed, and its effectiveness was assessed. Methods and Materials: The module, based on gynecologic malignancies, was developed in collaboration with a multidisciplinary team of subject matter experts. Subsequently, a multi-centre randomized controlled studymore » was conducted to test the module's effectiveness. Thirty-six radiation oncology residents participated in the study; 1920 were granted access to the module (intervention group), and 17 in the control group relied on traditional methods to acquire their knowledge. Pretests and posttests were administered to all participants. Statistical analysis was carried out using paired t test, analysis of variance, and post hoc tests. Results: The randomized control study revealed that the intervention group's pretest and posttest mean scores were 35% and 52%, respectively, and those of the control group were 37% and 42%, respectively. The mean improvement in test scores was 17% (p < 0.05) for the intervention group and 5% (p = not significant) for the control group. Retrospective pretest and posttest surveys showed a statistically significant change on all measured module objectives. Conclusions: The use of an interactive e-learning teaching module for radiation oncology is an effective method to improve the radiologic anatomy knowledge and treatment planning skills of radiation oncology residents.« less
  • Purpose: Waiting times remain one of the most vexing patient satisfaction challenges facing healthcare. Waiting time uncertainty can cause patients, who are already sick or in pain, to worry about when they will receive the care they need. These waiting periods are often difficult for staff to predict and only rough estimates are typically provided based on personal experience. This level of uncertainty leaves most patients unable to plan their calendar, making the waiting experience uncomfortable, even painful. In the present era of electronic health records (EHRs), waiting times need not be so uncertain. Extensive EHRs provide unprecedented amounts ofmore » data that can statistically cluster towards representative values when appropriate patient cohorts are selected. Predictive modelling, such as machine learning, is a powerful approach that benefits from large, potentially complex, datasets. The essence of machine learning is to predict future outcomes by learning from previous experience. The application of a machine learning algorithm to waiting time data has the potential to produce personalized waiting time predictions such that the uncertainty may be removed from the patient’s waiting experience. Methods: In radiation oncology, patients typically experience several types of waiting (eg waiting at home for treatment planning, waiting in the waiting room for oncologist appointments and daily waiting in the waiting room for radiotherapy treatments). A daily treatment wait time model is discussed in this report. To develop a prediction model using our large dataset (with more than 100k sample points) a variety of machine learning algorithms from the Python package sklearn were tested. Results: We found that the Random Forest Regressor model provides the best predictions for daily radiotherapy treatment waiting times. Using this model, we achieved a median residual (actual value minus predicted value) of 0.25 minutes and a standard deviation residual of 6.5 minutes. This means that the majority of our estimates are within 6.5 minutes of the actual wait time. Conclusion: The goal of this project was to define an appropriate machine learning algorithm to estimate waiting times based on the collective knowledge and experience learned from previous patients. Our results offer an opportunity to improve the information that is provided to patients and family members regarding the amount of time they can expect to wait for radiotherapy treatment at our centre. AJ acknowledges support by the CREATE Medical Physics Research Training Network grant of the Natural Sciences and Engineering Research Council (Grant number: 432290) and from the 2014 Q+ Initiative of the McGill University Health Centre.« less