skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications

Abstract

This report reviews existing literature describing forecast accuracy metrics, concentrating on those based on relative errors and percentage errors. We then review how the most common of these metrics, the mean absolute percentage error (MAPE), has been applied in recent radiation belt modeling literature. Finally, we describe metrics based on the ratios of predicted to observed values (the accuracy ratio) that address the drawbacks inherent in using MAPE. Specifically, we define and recommend the median log accuracy ratio as a measure of bias and the median symmetric accuracy as a measure of accuracy.

Authors:
 [1]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1260362
Report Number(s):
LA-UR-16-24592
TRN: US1601539
DOE Contract Number:
AC52-06NA25396
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; 71 CLASSICAL AND QUANTUM MECHANICS, GENERAL PHYSICS; ACCURACY; DATA COVARIANCES; RADIATION BELTS; COMPUTERIZED SIMULATION; SYMMETRY; Heliospheric and Magnetospheric Physics; Mathematics

Citation Formats

Morley, Steven Karl. Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications. United States: N. p., 2016. Web. doi:10.2172/1260362.
Morley, Steven Karl. Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications. United States. doi:10.2172/1260362.
Morley, Steven Karl. 2016. "Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications". United States. doi:10.2172/1260362. https://www.osti.gov/servlets/purl/1260362.
@article{osti_1260362,
title = {Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications},
author = {Morley, Steven Karl},
abstractNote = {This report reviews existing literature describing forecast accuracy metrics, concentrating on those based on relative errors and percentage errors. We then review how the most common of these metrics, the mean absolute percentage error (MAPE), has been applied in recent radiation belt modeling literature. Finally, we describe metrics based on the ratios of predicted to observed values (the accuracy ratio) that address the drawbacks inherent in using MAPE. Specifically, we define and recommend the median log accuracy ratio as a measure of bias and the median symmetric accuracy as a measure of accuracy.},
doi = {10.2172/1260362},
journal = {},
number = ,
volume = ,
place = {United States},
year = 2016,
month = 7
}

Technical Report:

Save / Share:
  • New features of the outer radiation belt dynamic response to geomagnetic storms are characterized using CRRES data. The response is found to have an intermediate (10-20 days) time scale and an intense short time scale (1-2 days). The intermediate time scale changes are characterized as invariant-violating diffusion, as has been developed by us for SCATHA data. The short time scale decreases of flux are characterized as adiabatic (reversible) dynamics, conserving the first variant under trapping volume change.
  • This report summarizes achievements made in the first year of the subject contract. New results on modeling the outer magnetosphere have been derived in order to provide an interpretation of CRRES radiation belt data. New features of the outer radiation belt dynamic response to geomagnetic storms are characterized using CRRES data. Detailed analysis of a benchmark event is presented. (Author)
  • This report summarizes achievements made in the subject contract. New results on modeling the outer magnetosphere have been derived in order to provide an interpretation of CRRES radiation belt data. New features of the outer radiation belt dynamic response to geomagnetic storms are characterized by using CRRES data. A CRRES outer-belt electron data base was chosen and quantitative determination of adiabatic and diffusive elements of the outer electron belt dynamic response to distinct geomagnetic disturbances were delineated. Scientific results of the subject contract were published in six articles in referred journals and books.
  • Purpose: Detection of treatment delivery errors is important in radiation therapy. However, accurate quantification of delivery errors is also of great importance. This study aims to evaluate the 3DVH software’s ability to accurately quantify delivery errors. Methods: Three VMAT plans (prostate, H&N and brain) were randomly chosen for this study. First, we evaluated whether delivery errors could be detected by gamma evaluation. Conventional per-beam IMRT QA was performed with the ArcCHECK diode detector for the original plans and for the following modified plans: (1) induced dose difference error up to ±4.0% and (2) control point (CP) deletion (3 to 10more » CPs were deleted) (3) gantry angle shift error (3 degree uniformly shift). 2D and 3D gamma evaluation were performed for all plans through SNC Patient and 3DVH, respectively. Subsequently, we investigated the accuracy of 3DVH analysis for all cases. This part evaluated, using the Eclipse TPS plans as standard, whether 3DVH accurately can model the changes in clinically relevant metrics caused by the delivery errors. Results: 2D evaluation seemed to be more sensitive to delivery errors. The average differences between ECLIPSE predicted and 3DVH results for each pair of specific DVH constraints were within 2% for all three types of error-induced treatment plans, illustrating the fact that 3DVH is fairly accurate in quantifying the delivery errors. Another interesting observation was that even though the gamma pass rates for the error plans are high, the DVHs showed significant differences between original plan and error-induced plans in both Eclipse and 3DVH analysis. Conclusion: The 3DVH software is shown to accurately quantify the error in delivered dose based on clinically relevant DVH metrics, where a conventional gamma based pre-treatment QA might not necessarily detect.« less
  • This paper seeks to evaluate the accuracy of the regression technique used in 2006 by using the same least-squares regression analysis on data created from hourly energy simulation software.