skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Measures of model performance based on the log accuracy ratio

Abstract

Quantitative assessment of modeling and forecasting of continuous quantities uses a variety of approaches. We review existing literature describing metrics for forecast accuracy and bias, concentrating on those based on relative errors and percentage errors. Of these accuracy metrics, the mean absolute percentage error (MAPE) is one of the most common across many fields and has been widely applied in recent space science literature and we highlight the benefits and drawbacks of MAPE and proposed alternatives. We then introduce the log accuracy ratio, and derive from it two metrics: the median symmetric accuracy; and the symmetric signed percentage bias. Robust methods for estimating the spread of a multiplicative linear model using the log accuracy ratio are also presented. The developed metrics are shown to be easy to interpret, robust, and to mitigate the key drawbacks of their more widely-used counterparts based on relative errors and percentage errors. Their use is illustrated with radiation belt electron flux modeling examples.

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [2]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  2. Univ. of Michigan, Ann Arbor, MI (United States). Climate and Space Sciences and Engineering Dept.
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Laboratory Directed Research and Development (LDRD) Program
OSTI Identifier:
1416283
Alternate Identifier(s):
OSTI ID: 1417763
Report Number(s):
LA-UR-17-24737
Journal ID: ISSN 1542-7390
Grant/Contract Number:  
AC52-06NA25396; 20150033DR
Resource Type:
Accepted Manuscript
Journal Name:
Space Weather
Additional Journal Information:
Journal Volume: 16; Journal Issue: 1; Journal ID: ISSN 1542-7390
Publisher:
American Geophysical Union
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; Heliospheric and Magnetospheric Physics; Mathematics

Citation Formats

Morley, Steven Karl, Brito, Thiago Vasconcelos, and Welling, Daniel T. Measures of model performance based on the log accuracy ratio. United States: N. p., 2018. Web. doi:10.1002/2017SW001669.
Morley, Steven Karl, Brito, Thiago Vasconcelos, & Welling, Daniel T. Measures of model performance based on the log accuracy ratio. United States. doi:10.1002/2017SW001669.
Morley, Steven Karl, Brito, Thiago Vasconcelos, and Welling, Daniel T. Wed . "Measures of model performance based on the log accuracy ratio". United States. doi:10.1002/2017SW001669. https://www.osti.gov/servlets/purl/1416283.
@article{osti_1416283,
title = {Measures of model performance based on the log accuracy ratio},
author = {Morley, Steven Karl and Brito, Thiago Vasconcelos and Welling, Daniel T.},
abstractNote = {Quantitative assessment of modeling and forecasting of continuous quantities uses a variety of approaches. We review existing literature describing metrics for forecast accuracy and bias, concentrating on those based on relative errors and percentage errors. Of these accuracy metrics, the mean absolute percentage error (MAPE) is one of the most common across many fields and has been widely applied in recent space science literature and we highlight the benefits and drawbacks of MAPE and proposed alternatives. We then introduce the log accuracy ratio, and derive from it two metrics: the median symmetric accuracy; and the symmetric signed percentage bias. Robust methods for estimating the spread of a multiplicative linear model using the log accuracy ratio are also presented. The developed metrics are shown to be easy to interpret, robust, and to mitigate the key drawbacks of their more widely-used counterparts based on relative errors and percentage errors. Their use is illustrated with radiation belt electron flux modeling examples.},
doi = {10.1002/2017SW001669},
journal = {Space Weather},
number = 1,
volume = 16,
place = {United States},
year = {2018},
month = {1}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 11 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Alternatives to the Median Absolute Deviation
journal, December 1993


A model for the Am (Km) planetary geomagnetic activity index and application to prediction
journal, June 1996


Linear prediction filter analysis of relativistic electron properties at 6.6 R E
journal, January 1990

  • Baker, D. N.; McPherron, R. L.; Cayton, T. E.
  • Journal of Geophysical Research, Vol. 95, Issue A9
  • DOI: 10.1029/JA095iA09p15133

Heliocentric Potential (HCP) Prediction Model for Nowscast of Aviation Radiation Dose
journal, March 2015

  • Hwang, Junga; Kim, Kyung-Chan; Dokgo, Kyunghwan
  • Journal of Astronomy and Space Sciences, Vol. 32, Issue 1
  • DOI: 10.5140/JASS.2015.32.1.39

A Note on the Measurement of Accuracy for Subnational Demographic Estimates
journal, May 2000

  • Swanson, David A.; Tayman, Jeff; Barr, Charles F.
  • Demography, Vol. 37, Issue 2
  • DOI: 10.2307/2648121

High- Z energetic particles at geosynchronous orbit during the Great Solar Proton Event Series of October 1989
journal, January 1992

  • Belian, R. D.; Gisler, G. R.; Cayton, T.
  • Journal of Geophysical Research, Vol. 97, Issue A11
  • DOI: 10.1029/92JA01139

A better measure of relative prediction accuracy for model selection and model estimation
journal, August 2015

  • Tofallis, Chris
  • Journal of the Operational Research Society, Vol. 66, Issue 8
  • DOI: 10.1057/jors.2014.103

Forecasting space weather: Can new econometric methods improve accuracy?
journal, June 2011


The Magnetic Electron Ion Spectrometer (MagEIS) Instruments Aboard the Radiation Belt Storm Probes (RBSP) Spacecraft
journal, June 2013

  • Blake, J. B.; Carranza, P. A.; Claudepierre, S. G.
  • Space Science Reviews, Vol. 179, Issue 1-4
  • DOI: 10.1007/s11214-013-9991-8

Accuracy measures: theoretical and practical concerns
journal, December 1993


The Global Positioning System constellation as a space weather monitor: Comparison of electron measurements with Van Allen Probes data: GPS AS SPACE WEATHER MONITOR
journal, February 2016

  • Morley, Steven K.; Sullivan, John P.; Henderson, Michael G.
  • Space Weather, Vol. 14, Issue 2
  • DOI: 10.1002/2015SW001339

What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting
journal, June 1993


How to judge the quality and value of weather forecast products
journal, September 2001


A comparison of artificial neural network and time series models for forecasting commodity prices
journal, March 1996


Dynamical characteristics of magnetospheric energetic ion time series: evidence for low dimensional chaos
journal, January 2003


Multisatellite determination of the relativistic electron phase space density at geosynchronous orbit: An integrated investigation during geomagnetic storm times: STORM
journal, November 2007

  • Chen, Y.; Friedel, R. H. W.; Reeves, G. D.
  • Journal of Geophysical Research: Space Physics, Vol. 112, Issue A11
  • DOI: 10.1029/2007JA012314

Dynamic Radiation Environment Assimilation Model: DREAM: DYNAMIC RADIATION ENVIRONMENT ASSIMILATION MODEL
journal, March 2012

  • Reeves, G. D.; Chen, Y.; Cunningham, G. S.
  • Space Weather, Vol. 10, Issue 3
  • DOI: 10.1029/2011SW000729

Modeling ionospheric outflows and their impact on the magnetosphere, initial results: MODELING IONOSPHERIC OUTFLOWS AND IMPACT
journal, May 2009

  • Glocer, A.; Tóth, G.; Gombosi, T.
  • Journal of Geophysical Research: Space Physics, Vol. 114, Issue A5
  • DOI: 10.1029/2009JA014053

Relativistic radiation belt electron responses to GEM magnetic storms: Comparison of CRRES observations with 3-D VERB simulations: STORM TIME RELATIVISTIC ELECTRON
journal, August 2012

  • Kim, Kyung-Chan; Shprits, Yuri; Subbotin, Dmitriy
  • Journal of Geophysical Research: Space Physics, Vol. 117, Issue A8
  • DOI: 10.1029/2011JA017460

The long-term effects of space weather on satellite operations
journal, January 2010


AE9, AP9 and SPM: New Models for Specifying the Trapped Energetic Particle and Space Plasma Environment
journal, March 2013


Dynamics of the outer radiation belt
journal, June 1997

  • Selesnick, R. S.; Blake, J. B.
  • Geophysical Research Letters, Vol. 24, Issue 11
  • DOI: 10.1029/97GL51409

Another look at measures of forecast accuracy
journal, October 2006


Operational forecasts of the geomagnetic Dst index : OPERATIONAL FORECASTS OF THE GEOMAGNETIC
journal, December 2002

  • Lundstedt, H.; Gleisner, H.; Wintoft, P.
  • Geophysical Research Letters, Vol. 29, Issue 24
  • DOI: 10.1029/2002GL016151

An examination of the Tsyganenko (T89a) field model using a database of two-satellite magnetic conjunctions
journal, March 1997

  • Weiss, L. A.; Thomsen, M. F.; Reeves, G. D.
  • Journal of Geophysical Research: Space Physics, Vol. 102, Issue A3
  • DOI: 10.1029/96JA02876

Electron Acceleration in the Heart of the Van Allen Radiation Belts
journal, July 2013


The Influence Curve and its Role in Robust Estimation
journal, June 1974


MAPE-R: a rescaled measure of accuracy for cross-sectional subnational population forecasts
journal, March 2011

  • Swanson, David A.; Tayman, Jeff; Bryan, T. M.
  • Journal of Population Research, Vol. 28, Issue 2-3
  • DOI: 10.1007/s12546-011-9054-5

An empirical dynamic equation for energetic electrons at geosynchronous orbit
journal, January 2003


Data-based modelling of the Earth's dynamic magnetosphere: a review
journal, January 2013


Energetic Particle Data From the Global Positioning System Constellation
journal, February 2017

  • Morley, S. K.; Sullivan, J. P.; Carver, M. R.
  • Space Weather, Vol. 15, Issue 2
  • DOI: 10.1002/2017SW001604

On the relationship between relativistic electron flux and solar wind velocity: Paulikas and Blake revisited: RELATIVISTIC ELECTRONS AND VSW 1989-2010
journal, February 2011

  • Reeves, Geoffrey D.; Morley, Steven K.; Friedel, Reiner H. W.
  • Journal of Geophysical Research: Space Physics, Vol. 116, Issue A2
  • DOI: 10.1029/2010JA015735

Linear relation between convective cloud base height and updrafts and application to satellite retrievals: SATELLITE RETRIEVAL OF CLOUD UPDRAFTS
journal, August 2015

  • Zheng, Youtong; Rosenfeld, Daniel
  • Geophysical Research Letters, Vol. 42, Issue 15
  • DOI: 10.1002/2015GL064809

A pragmatic view of accuracy measurement in forecasting
journal, January 1986


Relativistic electron dynamics in the inner magnetosphere — a review
journal, January 2002

  • Friedel, R. H. W.; Reeves, G. D.; Obara, T.
  • Journal of Atmospheric and Solar-Terrestrial Physics, Vol. 64, Issue 2
  • DOI: 10.1016/S1364-6826(01)00088-8

On the Theory of Scales of Measurement
journal, June 1946


Science Objectives and Rationale for the Radiation Belt Storm Probes Mission
journal, September 2012


The K -derived planetary indices: Description and availability
journal, January 1991

  • Menvielle, M.; Berthelier, A.
  • Reviews of Geophysics, Vol. 29, Issue 3
  • DOI: 10.1029/91RG00994

On MAPE-R as a measure of cross-sectional estimation and forecast accuracy
journal, December 2007

  • Coleman, Charles D.; Swanson, David A.
  • Journal of Economic and Social Measurement, Vol. 32, Issue 4
  • DOI: 10.3233/JEM-2007-0290

    Works referencing / citing this record:

    A 30-Year Simulation of the Outer Electron Radiation Belt
    journal, October 2018

    • Glauert, S. A.; Horne, R. B.; Meredith, N. P.
    • Space Weather, Vol. 16, Issue 10
    • DOI: 10.1029/2018sw001981

    A 30-Year Simulation of the Outer Electron Radiation Belt
    journal, October 2018

    • Glauert, S. A.; Horne, R. B.; Meredith, N. P.
    • Space Weather, Vol. 16, Issue 10
    • DOI: 10.1029/2018sw001981