SU-E-T-511: Inter-Rater Variability in Classification of Incidents in a New Incident Reporting System
- Hofstra University, Hempstead, NY (United States)
- Long Island Jewish Medical Center, New Hyde Park, NY (United States)
Purpose To determine how consistent the results of different raters are when reviewing the same cases within the Radiation Oncology Incident Learning System (ROILS). Methods Three second-year medical physics graduate students filled out incident reports in spreadsheets set up to mimic ROILS. All students studied the same 33 cases and independently entered their assessments, for a total of 99 reviewed cases. The narratives for these cases were obtained from a published International Commission on Radiological Protection (ICRP) report which included shorter narratives selected from the Radiation Oncology Safety Information System (ROSIS) database. Each category of questions was reviewed to see how consistent the results were by utilizing free-marginal multirater kappa analysis. The percentage of cases where all raters shared full agreement or full disagreement was recorded to show which questions were answered consistently by multiple raters for a given case. The consistency among the raters was analyzed between ICRP and ROSIS cases to see if either group led to more reliable results. Results The categories where all raters agreed 100 percent in their choices were the event type (93.94 percent of cases 0.946 kappa) and the likelihood of the event being harmful to the patient (42.42 percent of cases 0.409 kappa). The categories where all raters disagreed 100 percent in their choices were the dosimetric severity scale (39.39 percent of cases 0.139 kappa) and the potential future toxicity (48.48 percent of cases 0.205 kappa). ROSIS had more cases where all raters disagreed than ICRP (23.06 percent of cases compared to 15.58 percent, respectively). Conclusion Despite reviewing the same cases, the results among the three raters was widespread. ROSIS narratives were shorter than ICRP, which suggests that longer narratives lead to more consistent results. This study shows that the incident reporting system can be optimized to yield more consistent results.
- OSTI ID:
- 22548549
- Journal Information:
- Medical Physics, Vol. 42, Issue 6; Other Information: (c) 2015 American Association of Physicists in Medicine; Country of input: International Atomic Energy Agency (IAEA); ISSN 0094-2405
- Country of Publication:
- United States
- Language:
- English
Similar Records
TU-D-201-04: Veracity of Data Elements in Radiation Oncology Incident Learning Systems
Targeting safety improvements through identification of incident origination and detection in a near-miss incident learning system