skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Measures of model performance based on the log accuracy ratio

Journal Article · · Space Weather
DOI:https://doi.org/10.1002/2017SW001669· OSTI ID:1416283
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [2]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  2. Univ. of Michigan, Ann Arbor, MI (United States). Climate and Space Sciences and Engineering Dept.
+ Show Author Affiliations

Abstract Quantitative assessment of modeling and forecasting of continuous quantities uses a variety of approaches. We review existing literature describing metrics for forecast accuracy and bias, concentrating on those based on relative errors and percentage errors. Of these accuracy metrics, the mean absolute percentage error (MAPE) is one of the most common across many fields and has been widely applied in recent space science literature and we highlight the benefits and drawbacks of MAPE and proposed alternatives. We then introduce the log accuracy ratio and derive from it two metrics: the median symmetric accuracy and the symmetric signed percentage bias. Robust methods for estimating the spread of a multiplicative linear model using the log accuracy ratio are also presented. The developed metrics are shown to be easy to interpret, robust, and to mitigate the key drawbacks of their more widely used counterparts based on relative errors and percentage errors. Their use is illustrated with radiation belt electron flux modeling examples.

Research Organization:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Organization:
USDOE Laboratory Directed Research and Development (LDRD) Program
Grant/Contract Number:
AC52-06NA25396
OSTI ID:
1416283
Alternate ID(s):
OSTI ID: 1417763
Report Number(s):
LA-UR-17-24737
Journal Information:
Space Weather, Vol. 16, Issue 1; ISSN 1542-7390
Publisher:
American Geophysical UnionCopyright Statement
Country of Publication:
United States
Language:
English
Citation Metrics:
Cited by: 140 works
Citation information provided by
Web of Science

References (58)

Three-dimensional modeling of the radiation belts using the Versatile Electron Radiation Belt (VERB) code: MODELING OF THE RADIATION BELTS journal October 2009
A Note on the Measurement of Accuracy for Subnational Demographic Estimates journal May 2000
On MAPE-R as a measure of cross-sectional estimation and forecast accuracy journal December 2007
A better measure of relative prediction accuracy for model selection and model estimation journal August 2015
The Magnetic Electron Ion Spectrometer (MagEIS) Instruments Aboard the Radiation Belt Storm Probes (RBSP) Spacecraft journal June 2013
Accuracy measures: theoretical and practical concerns journal December 1993
The Global Positioning System constellation as a space weather monitor: Comparison of electron measurements with Van Allen Probes data: GPS AS SPACE WEATHER MONITOR journal February 2016
What Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting journal June 1993
Forecast Verification book December 2011
A comparison of artificial neural network and time series models for forecasting commodity prices journal March 1996
Multisatellite determination of the relativistic electron phase space density at geosynchronous orbit: An integrated investigation during geomagnetic storm times: STORM journal November 2007
Dynamic Radiation Environment Assimilation Model: DREAM: DYNAMIC RADIATION ENVIRONMENT ASSIMILATION MODEL journal March 2012
Modeling ionospheric outflows and their impact on the magnetosphere, initial results: MODELING IONOSPHERIC OUTFLOWS AND IMPACT journal May 2009
Relativistic radiation belt electron responses to GEM magnetic storms: Comparison of CRRES observations with 3-D VERB simulations: STORM TIME RELATIVISTIC ELECTRON journal August 2012
Alternatives to accuracy and bias metrics based on percentage errors for radiation belt modeling applications report July 2016
AE9, AP9 and SPM: New Models for Specifying the Trapped Energetic Particle and Space Plasma Environment journal March 2013
Dynamics of the outer radiation belt journal June 1997
Another look at measures of forecast accuracy journal October 2006
Validation of the effect of cross‐calibrated GOES solar proton effective energies on derived integral fluxes by comparison with STEREO observations journal February 2017
An examination of the Tsyganenko (T89a) field model using a database of two-satellite magnetic conjunctions journal March 1997
Electron Acceleration in the Heart of the Van Allen Radiation Belts journal July 2013
The Influence Curve and its Role in Robust Estimation journal June 1974
MAPE-R: a rescaled measure of accuracy for cross-sectional subnational population forecasts journal March 2011
Energetic electrons, 50 keV to 6 MeV, at geosynchronous orbit: Their responses to solar wind variations: ENERGETIC ELECTRONS AT GEOSYNCHRONOUS ORBIT journal April 2005
An empirical dynamic equation for energetic electrons at geosynchronous orbit journal January 2003
Dynamic linear models for forecasting of radiation belt electrons and limitations on physical interpretation of predictive models: OSTHUS ET AL. journal June 2014
Improving Empirical Magnetic Field Models by Fitting to In Situ Data Using an Optimized Parameter Approach: OPTIMIZING MODELS WITH IN SITU DATA journal December 2017
On the relationship between relativistic electron flux and solar wind velocity: Paulikas and Blake revisited: RELATIVISTIC ELECTRONS AND VSW 1989-2010 journal February 2011
Linear relation between convective cloud base height and updrafts and application to satellite retrievals: SATELLITE RETRIEVAL OF CLOUD UPDRAFTS journal August 2015
Relativistic electron dynamics in the inner magnetosphere — a review journal January 2002
Science Objectives and Rationale for the Radiation Belt Storm Probes Mission journal September 2012
Alternatives to the Median Absolute Deviation journal December 1993
A model for the Am (Km) planetary geomagnetic activity index and application to prediction journal June 1996
Linear prediction filter analysis of relativistic electron properties at 6.6 R E journal January 1990
Heliocentric Potential (HCP) Prediction Model for Nowscast of Aviation Radiation Dose journal March 2015
The concepts of bias, precision and accuracy, and their use in testing the performance of species richness estimators, with a literature review of estimator performance journal December 2005
High- Z energetic particles at geosynchronous orbit during the Great Solar Proton Event Series of October 1989 journal January 1992
Forecasting space weather: Can new econometric methods improve accuracy? journal June 2011
How to judge the quality and value of weather forecast products journal September 2001
Dynamical characteristics of magnetospheric energetic ion time series: evidence for low dimensional chaos journal January 2003
The long-term effects of space weather on satellite operations journal January 2010
Operational forecasts of the geomagnetic Dst index : OPERATIONAL FORECASTS OF THE GEOMAGNETIC journal December 2002
Dependence of radiation belt simulations to assumed radial diffusion rates tested for two empirical models of radial transport: VERB CODE SIMULATIONS WITH DIFFERENT DLL journal January 2017
Data-based modelling of the Earth's dynamic magnetosphere: a review journal January 2013
Energetic Particle Data From the Global Positioning System Constellation journal February 2017
A pragmatic view of accuracy measurement in forecasting journal January 1986
On the Theory of Scales of Measurement journal June 1946
The K -derived planetary indices: Description and availability journal January 1991
Multisegment statistical bias correction of daily GCM precipitation output: M-S PRECIPITATION BIAS CORRECTION journal April 2013
What accuracy statistics really measure journal January 2001
Radial diffusion comparing a THEMIS statistical model with geosynchronous measurements as input journal March 2014
Simultaneous event‐specific estimates of transport, loss, and source rates for relativistic outer radiation belt electrons journal March 2017
Modeling radiation belt electron dynamics during GEM challenge intervals with the DREAM3D diffusion model: MODELING RADIATION BELT WITH DREAM3D journal October 2013
Application and testing of the L * neural network with the self-consistent magnetic field model of RAM-SCB journal March 2014
Automated determination of electron density from electric field measurements on the Van Allen Probes spacecraft: AUTOMATED ELECTRON DENSITY DETERMINATION journal May 2016
Systemic lupus Erythematosus and geomagnetic disturbances: a time series analysis journal March 2021
Another look at measures of forecast accuracy text January 2017
A better measure of relative prediction accuracy for model selection and model estimation text January 2021

Cited By (10)

An online‐offline prognosis model for fatigue life prediction under biaxial cyclic loading with overloads journal February 2019
Forecasting the Arrival Time of Coronal Mass Ejections: Analysis of the CCMC CME Scoreboard journal September 2018
Impact of Significant Time‐Integrated Geomagnetic Activity on 2‐MeV Electron Flux journal June 2019
Analytical Representations for Characterizing the Global Aviation Radiation Environment Based on Model and Measurement Databases journal October 2018
Moisture Sorption Models for Fuel Beds of Standing Dead Grass in Alaska journal December 2018
Statistical Modeling of the Coupled F ‐Region Ionosphere‐Thermosphere at High Latitude During Polar Darkness journal February 2019
Reanalysis of Ring Current Electron Phase Space Densities Using Van Allen Probe Observations, Convection Model, and Log‐Normal Kalman Filter journal April 2019
Space Radiation and Plasma Effects on Satellites and Aviation: Quantities and Metrics for Tracking Performance of Space Weather Environment Models journal October 2019
Forecasting the Arrival Time of Coronal Mass Ejections: Analysis of the CCMC CME Scoreboard text January 2018
The Challenge of Machine Learning in Space Weather Nowcasting and Forecasting text January 2019

Similar Records

Related Subjects

54 ENVIRONMENTAL SCIENCES
Heliospheric and Magnetospheric Physics
Mathematics