skip to main content

DOE PAGESDOE PAGES

This content will become publicly available on January 3, 2019

Title: Measures of model performance based on the log accuracy ratio

Quantitative assessment of modeling and forecasting of continuous quantities uses a variety of approaches. We review existing literature describing metrics for forecast accuracy and bias, concentrating on those based on relative errors and percentage errors. Of these accuracy metrics, the mean absolute percentage error (MAPE) is one of the most common across many fields and has been widely applied in recent space science literature and we highlight the benefits and drawbacks of MAPE and proposed alternatives. We then introduce the log accuracy ratio, and derive from it two metrics: the median symmetric accuracy; and the symmetric signed percentage bias. Robust methods for estimating the spread of a multiplicative linear model using the log accuracy ratio are also presented. The developed metrics are shown to be easy to interpret, robust, and to mitigate the key drawbacks of their more widely-used counterparts based on relative errors and percentage errors. Their use is illustrated with radiation belt electron flux modeling examples.
Authors:
ORCiD logo [1] ; ORCiD logo [1] ; ORCiD logo [2]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  2. Univ. of Michigan, Ann Arbor, MI (United States). Climate and Space Sciences and Engineering Dept.
Publication Date:
Report Number(s):
LA-UR-17-24737
Journal ID: ISSN 1542-7390
Grant/Contract Number:
AC52-06NA25396; 20150033DR
Type:
Accepted Manuscript
Journal Name:
Space Weather
Additional Journal Information:
Journal Volume: 16; Journal Issue: 1; Journal ID: ISSN 1542-7390
Publisher:
American Geophysical Union
Research Org:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org:
USDOE Laboratory Directed Research and Development (LDRD) Program
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; Heliospheric and Magnetospheric Physics; Mathematics; Model validation; Forecasting
OSTI Identifier:
1416283
Alternate Identifier(s):
OSTI ID: 1417763