Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Model evaluation guidelines for geomagnetic index predictions

Journal Article · · Space Weather
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3]; ORCiD logo [4]; ORCiD logo [3]; ORCiD logo [5]; ORCiD logo [6]; ORCiD logo [7]; ORCiD logo [8]; ORCiD logo [9];  [10];  [11]; ORCiD logo [12]; ORCiD logo [3]; ORCiD logo [13]; ORCiD logo [14]; ORCiD logo [15]; ORCiD logo [16]; ORCiD logo [17]; ORCiD logo [18] more »;  [19] « less
  1. Univ. of Michigan, Ann Arbor, MI (United States)
  2. Air Force Research Lab., Kirtland AFB, NM (United States)
  3. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  4. The Catholic Univ. of America, Washington, D.C. (United States); NASA Goddard Space Flight Center (GSFC), Greenbelt, MD (United States)
  5. Univ. of Alcala, Madrid (Spain)
  6. Space Environment Technologies, Pacific Palisades, CA (United States)
  7. Swedish Institute of Space Physics, Lund (Sweden)
  8. Univ. of Michigan, Ann Arbor, MI (United States); Finnish Meteorological Institute, Helsinki (Finland)
  9. Univ. of Michigan, Ann Arbor, MI (United States); Univ. of Texas, Arlington, TX (United States)
  10. UK Met Office, Devon (United Kingdom)
  11. Univ. of Sheffield, South Yorkshire (United Kingdom)
  12. Swedish Institute of Space Physics, Uppsala (Sweden)
  13. George Mason Univ., Fairfax, VA (United States)
  14. National Oceanic and Atmospheric Administration, Boulder, CO (United States)
  15. CAS, Prague (Czech Republic)
  16. Department of Space Geodesy CNES, Toulouse (France)
  17. GFZ German Research Centre for Geosciences, Telegrafenberg, Potsdam (Germany); Univ. of Potsdam, Potsdam (Germany)
  18. GFZ German Research Centre for Geosciences, Telegrafenberg, Potsdam (Germany); Univ. of Potsdam, Potsdam (Germany); Univ. of California, Los Angeles, CA (United States)
  19. GFZ German Research Centre for Geosciences, Telegrafenberg, Potsdam (Germany)
+ Show Author Affiliations

Geomagnetic indices are convenient quantities that distill the complicated physics of some region or aspect of near–Earth space into a single parameter. Most of the best–known indices are calculated from ground–based magnetometer data sets, such as Dst, SYM–H, Kp, AE, AL, and PC. Many models have been created that predict the values of these indices, often using solar wind measurements upstream from Earth as the input variables to the calculation. This document reviews the current state of models that predict geomagnetic indices and the methods used to assess their ability to reproduce the target index time series. These existing methods are synthesized into a baseline collection of metrics for benchmarking a new or updated geomagnetic index prediction model. These methods fall into two categories: (1) fit performance metrics such as root mean square error (RMSE) and mean absolute error (MAE) that are applied to a time–series comparison of model output and observations; and (2) event detection performance metrics such as Heidke Skill Score and probability of detection (POD) that are derived from a contingency table that compares model and observation values exceeding (or not) a threshold value. Furthermore, a few examples of codes being used with this set of metrics are presented, and other aspects of metrics assessment best practices, limitations, and uncertainties are discussed, including several caveats to consider when using geomagnetic indices.

Research Organization:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Organization:
LDRD; USDOE
Grant/Contract Number:
AC52-06NA25396
OSTI ID:
1482013
Report Number(s):
LA-UR--18-28370
Journal Information:
Space Weather, Journal Name: Space Weather Journal Issue: 12 Vol. 16; ISSN 1542-7390
Publisher:
American Geophysical UnionCopyright Statement
Country of Publication:
United States
Language:
English

References (155)

Prediction of the AU , AL , and AE indices using solar wind parameters : PREDICTION OF THE journal December 2013
Analytic expressions for ULF wave radiation belt radial diffusion coefficients: Analytic Radial Diffusion Coefficients journal March 2014
Canonical correlation analysis of the combined solar wind and geomagnetic index data sets: Canonical Correlation Analysis journal July 2014
Activity-dependent global model of electron loss inside the plasmasphere: ELECTRON LIFETIMES DUE TO HISS journal June 2014
The Comprehensive Inner Magnetosphere-Ionosphere Model: THE CIMI MODEL journal September 2014
Dynamic linear models for forecasting of radiation belt electrons and limitations on physical interpretation of predictive models: OSTHUS ET AL. journal June 2014
Validating the Rice neural network and the Wing K p real-time models journal June 2014
Probabilistic forecasting analysis of geomagnetic indices for southward IMF events journal March 2015
Global empirical models of plasmaspheric hiss using Van Allen Probes: HISS MODEL journal December 2015
Empirical model of lower band chorus wave distribution in the outer radiation belt: EMPIRICAL MODEL OF CHORUS WAVES journal December 2015
New global loss model of energetic and relativistic electrons based on Van Allen Probes measurements journal February 2016
An empirical model of electron and ion fluxes derived from observations at geosynchronous orbit: Empirical model of GEO fluxes journal April 2015
Effects of electric field methods on modeling the midlatitude ionospheric electrodynamics and inner magnetosphere dynamics: SELF-CONSISTENT ELECTRIC FIELD journal May 2017
A scheme for forecasting severe space weather journal March 2017
Community-wide validation of geospace model local K-index predictions to support model transition to operations: GEOSPACE MODEL VALIDATION: REGIONAL K journal July 2016
An improved empirical model of electron and ion fluxes at geosynchronous orbit based on upstream solar wind conditions: MODEL OF ELECTRON AND ION FLUXES AT GEO journal July 2016
Predicting the magnetic vectors within coronal mass ejections arriving at Earth: 2. Geomagnetic response journal February 2017
Modeling and prediction of global magnetic disturbance in near-Earth space: A case study for K p index using NARX models : MODELING AND PREDICTION OF journal October 2016
Is the Dst Index Sufficient to Define All Geospace Storms? : Is journal November 2017
Measures of Model Performance Based On the Log Accuracy Ratio journal January 2018
SWMF Global Magnetosphere Simulations of January 2005: Geomagnetic Indices and Cross-Polar Cap Potential: SWMF JANUARY 2005: INDICES AND CPCP journal December 2017
The Polar Cap index: A critical review of methods and a new approach: THE POLAR CAP INDEX-A CRITICAL REVIEW journal August 2013
Similarities and differences in low- to middle-latitude geomagnetic indices: DIFFERENCES IN GEOMAGNETIC INDICES journal August 2013
Geospace environment modeling 2008-2009 challenge: D st index : GEM 2008-2009 CHALLENGE: DST INDEX journal April 2013
Community-wide validation of geospace model ground magnetic field perturbation predictions to support model transition to operations: GEOSPACE MODEL TRANSITION journal June 2013
The Anemomilos prediction methodology for Dst : ANEMOMILOS DST journal September 2013
An Introduction to the Bootstrap book January 1993
Modelling of the magnetic field of magnetospheric ring current as a function of interplanetary medium parameters journal January 1992
Prediction of geomagnetic storms from solar wind data with the use of a neural network journal January 1994
Derivation, Meaning and Use of Geomagnetic Indices journal October 1981
Extension of convection modeling into the high-latitude ionosphere: some theoretical difficulties journal September 1991
Magnetic activity in the polar cap—A new index journal November 1988
Forecasting the ring current index Dst in real time journal September 2000
Forecasting the geomagnetic activity of the Dst index using multiscale radial basis function networks journal January 2007
The Lyon–Fedder–Mobarry (LFM) global MHD magnetospheric simulation code journal October 2004
Initial results from the coupled magnetosphere ionosphere thermosphere model: magnetospheric and ionospheric responses journal October 2004
AE index forecast at different time scales through an ANN algorithm based on L1 IMF and plasma measurements journal February 2008
Self-consistent geomagnetic storm simulation: The role of the induced electric fields journal February 2008
A neural network Dst index model driven by input time histories of the solar wind–magnetosphere interaction journal April 2014
Specification of the near-Earth space environment with SHIELDS journal October 2018
Adaptive numerical algorithms in space weather modeling journal February 2012
ROC-ing along: Evaluation and interpretation of receiver operating characteristic curves journal June 2016
A skill score based on economic value for probability forecasts journal June 2001
Inner magnetospheric modeling with the Rice Convection Model journal January 2003
An empirical phase space analysis of ring current dynamics: Solar wind control of injection and decay journal April 2000
Analysis of early phase ring current recovery mechanisms during geomagnetic storms journal September 1999
Predictability of Dst index based upon solar wind conditions monitored inside 1 AU journal May 1999
Radial diffusion analysis of outer radiation belt electrons during the October 9, 1990, magnetic storm journal January 2000
Formation of intense nose structures journal February 2001
An automated procedure for near-real-time Kp estimates journal October 2001
Dominant role of the asymmetric ring current in producing the stormtime Dst * journal June 2001
Ring current decay rates of magnetic storms: A statistical study from 1957 to 1998 journal January 2002
Global simulation of the Geospace Environment Modeling substorm challenge event journal January 2001
Identification of nonlinear processes in the magnetospheric dynamics and forecasting of Dst index journal December 2001
A new model for the prediction of Dst on the basis of the solar wind journal December 2002
A new model of the location of the plasmapause: CRRES results journal January 2002
Auroral electrojet predictions with dynamic neural networks journal November 2001
A physics-based study of the Dst - AL relationship journal January 2002
Empirical plasmapause models from magnetic indices: EMPIRICAL PLASMAPAUSE MODELS journal February 2003
Operational forecasts of the geomagnetic Dst index : OPERATIONAL FORECASTS OF THE GEOMAGNETIC journal December 2002
Self-consistent magnetosphere-ionosphere coupling: Theoretical studies journal January 2003
An empirical formula relating the geomagnetic storm's intensity to the interplanetary parameters: − and Δ t : EMPIRICAL FORMULA OF STORM'S INTENSITY journal October 2003
A statistical study of the global structure of the ring current journal January 2004
Dependence of plasmaspheric morphology on the electric field description during the recovery phase of the 17 April 2002 magnetic storm journal January 2004
Prediction of the Dst index using multiresolution wavelet models journal January 2004
What is the physical meaning of the AE index? journal January 2004
Kp forecast models: Kp FORECAST MODELS journal April 2005
CRRES electric field power spectra and radial diffusion coefficients journal January 2005
Why Kp is such a good measure of magnetospheric convection journal November 2004
High-resolution global storm index: Dst versus SYM-H journal January 2006
Space Weather Modeling Framework: A new tool for the space science community journal January 2005
Dst model for 1995–2002 journal January 2006
Differences between CME-driven storms and CIR-driven storms journal January 2006
Decision theory and the analysis of rare event space weather forecasts: DECISION THEORY AND RARE EVENTS journal May 2006
Limits on the complexity of empirical models of magnetic storm phenomena: LIMITS ON EMPIRICAL MODELS journal April 2006
Self-consistent modeling of magnetic fields and plasmas in the inner magnetosphere: Application to a geomagnetic storm journal January 2006
Prediction of the AL index using solar wind parameters : PREDICTION OF THE journal June 2007
Analysis of the 3-7 October 2000 and 15-24 April 2002 geomagnetic storms with an optimized nonlinear dynamical model: OPTIMIZED DYNAMICAL MODEL journal April 2007
Solar and interplanetary sources of major geomagnetic storms ( Dst ≤ −100 nT) during 1996-2005 : SOURCES OF GEOMAGNETIC STORMS journal October 2007
Pairs of solar wind-magnetosphere coupling functions: Combining a merging term with a viscous term works best: PAIRS OF COUPLING FUNCTIONS journal April 2008
Ring current simulations of the 90 intense storms during solar cycle 23: RING CURRENT MODELING FOR A SOLAR CYCLE journal March 2008
Evaluation of solar wind-magnetosphere coupling functions during geomagnetic storms with the WINDMI model: EVALUATION OF COUPLING FUNCTIONS journal February 2009
Real-time prediction of magnetospheric activity using the Boyle Index: KP USING BOYLE INDEX journal April 2009
Real-time predictions of geomagnetic storms and substorms: Use of the Solar Wind Magnetosphere-Ionosphere System model: REAL-TIME WINDMI STORM PREDICTIONS journal July 2009
Hyperbolic decay of the Dst Index during the recovery phase of intense geomagnetic storms: HYPERBOLIC DECAY OF THE DST INDEX journal July 2010
Solar wind density influence on geomagnetic storm intensity: SOLAR WIND DENSITY AND STORM INTENSITY journal September 2010
Effects of the low-latitude ionospheric boundary condition on the global magnetosphere: IONOSPHERE/MAGNETOSPHERE GLOBAL EFFECTS journal October 2010
Comparative study of ring current development using empirical, dipolar, and self-consistent magnetic field simulations: RING CURRENT AND MAGNETIC FIELD DYNAMICS journal December 2010
Quantitative assessment of the probability forecast for the geomagnetic storm occurrence: GEOMAGNETIC STORM FORECAST journal December 2010
Evaluation of SuperMAG auroral electrojet indices as indicators of substorms and auroral power: SUPERMAG AURORAL ELECTROJET INDICES journal December 2011
Kinetic model of the inner magnetosphere with arbitrary magnetic field: TECHNIQUE journal April 2012
Is the storm time response of the inner magnetospheric hot ions universally similar or driver dependent?: STORM TIME INNER MAGNETOSPHERIC RESPONSE journal April 2012
Forecasting F 10.7 with solar magnetic flux transport modeling : FORECASTING F journal February 2012
The SuperMAG data processing technique: TECHNIQUE journal September 2012
Evidence for potential and inductive convection during intense geomagnetic events using normalized superposed epoch analysis: GEOMAGNETIC STORM CONVECTION ANALYSIS journal January 2013
Improvements in short-term forecasting of geomagnetic activity: SHORT-TERM FORECASTING OF journal June 2012
Real-Time SWMF at CCMC: Assessing the Dst Output From Continuous Operational Simulations journal October 2018
Perturbed Input Ensemble Modeling With the Space Weather Modeling Framework journal September 2018
An ISEE/whistler model of equatorial electron density in the magnetosphere journal January 1992
Prediction of geomagnetic activity journal May 1993
Comment on “Prediction of geomagnetic activity” by C. K. Goertz, Lin-Hua Shan, and R. A. Smith journal May 1993
What is a geomagnetic storm? journal January 1994
A bounce-averaged kinetic model of the ring current ion population journal December 1994
A correlative comparison of the ring current and auroral electrojets using geomagnetic indices journal January 1995
Three-Dimensional Ring Current Decay Model journal January 1995
Collisional losses of ring current ions journal January 1996
Two-component auroral electrojet: Importance for substorm studies journal June 1996
A low-dimensional energy-conserving state space model for substorm dynamics journal December 1996
Empirical polar cap potentials journal January 1997
Neural network prediction of AE data journal October 1997
Geomagnetic storm predictions from solar wind data with the use of dynamic neural networks journal January 1997
October 1995 magnetic cloud and accompanying storm activity: Ring current evolution journal January 1998
A low-dimensional dynamical model for the solar wind driven geotail-ionosphere system journal March 1998
Geomagnetic response to magnetic clouds of different polarity journal August 1998
Dst index prediction using data-derived analogues of the magnetospheric dynamics journal September 1998
A high time resolution study of interplanetary parameter correlations with AE journal February 1981
Solar wind control of auroral zone geomagnetic activity journal August 1981
Derivation, Meaning, and Use of Geomagnetic Indices book January 1980
A semiempirical model of large-scale magnetospheric electric fields journal January 1973
The motion of a proton in the equatorial magnetosphere journal February 1975
Isolated cold plasma regions: Observations and their relation to possible production mechanisms journal March 1975
Differential rotation of the magnetospheric plasma as cause of the Svalgaard-Mansurov effect journal June 1975
An empirical relationship between interplanetary conditions and Dst journal November 1975
Quantitative simulation of a magnetospheric substorm 1. Model logic and overview journal January 1981
An evaluation of three predictors of geomagnetic activity journal January 1982
Comment on ‘An evaluation of three predictors of geomagnetic activity’ by R. E. Holzer and J. A. Slavin journal January 1983
Auroral electrojet activity index AE and its universal time variations journal February 1966
Geomagnetic indices journal January 1972
Coupling function between solar wind parameters and geomagnetic indices journal January 1982
The three-hour-range index measuring geomagnetic activity journal January 1939
Verification of space weather forecasting at the Regional Warning Center in Belgium journal January 2014
Forecasting Kp from solar wind data: input parameter study using 3-hour averages and 3-hour range values journal January 2017
Approximate is Better than “Exact” for Interval Estimation of Binomial Proportions journal May 1998
Identification of MIMO non-linear systems using a forward-regression orthogonal estimator journal June 1989
Caveats and pitfalls of ROC analysis in clinical microarray research (and how to avoid them) journal March 2011
ROC Curves—What are They and How are They Used? journal January 2012
CIR versus CME drivers of the ring current during intense magnetic storms
  • Liemohn, Michael W.; Jazowski, Matt; Kozyra, Janet U.
  • Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 466, Issue 2123 https://doi.org/10.1098/rspa.2010.0075
journal May 2010
The Finley Affair: A Signal Event in the History of Forecast Verification journal March 1996
Use of the “Odds Ratio” for Diagnosing Forecast Skill journal April 2000
Cross-Validation in Statistical Climate Forecast Models journal November 1987
Verification of Short-Range Weather Forecasts (A Survey of the Literature) I. journal January 1944
Skill Scores Based on the Mean Square Error and Their Relationships to the Correlation Coefficient journal December 1988
An Introduction to the Bootstrap book May 1994
Approximate Is Better than "Exact" for Interval Estimation of Binomial Proportions journal May 1998
Berechnung des Erfolges und der Güte der Windstärkevorhersagen im Sturmwarnungsdienst journal January 1926
3-D force-balanced magnetospheric configurations journal January 2004
A comparison of the probability distribution of observed substorm magnitude with that predicted by a minimal substorm model journal January 2007
Forecasting intense geomagnetic activity using interplanetary magnetic field data journal January 2008
Revised Dst and the epicycles of magnetic disturbance: 1958–2007 journal January 2009
Data derived NARMAX Dst model journal January 2011
Storm-time ring current: model-dependent results journal January 2012
On the prediction of the auroral westward electrojet index journal January 2012
The use and misuse of statistics in space physics. journal January 1990
Storm-time magnetospheric currents inferred from mid-latitude geomagnetic field variations. journal January 1990
Dataset for Space Weather journal article 2018SW002067 dataset January 2018

Cited By (9)

Recent Advancements and Remaining Challenges Associated With Inner Magnetosphere Cross‐Energy/Population Interactions (IMCEPI) journal February 2019
The Challenge of Machine Learning in Space Weather: Nowcasting and Forecasting journal August 2019
Extension of an Empirical Electron Flux Model From 6 to 20 Earth Radii Using Cluster/RAPID Observations journal May 2019
Space Weather Modeling Capabilities Assessment: Auroral Precipitation and High‐Latitude Ionospheric Electrodynamics journal February 2019
A Statistical Classifier for Historical Geomagnetic Storm Drivers Derived Solely From Ground‐Based Magnetic Field Measurements journal October 2019
Quantifying the Performance of Geomagnetically Induced Current Models journal July 2019
Systematic Analysis of Machine Learning and Feature Selection Techniques for Prediction of the Kp Index journal October 2019
A Statistical Classifier for Historical Geomagnetic Storm Drivers Derived Solely From Ground-Based Magnetic Field Measurements text January 2019
A Statistical Classifier for Historical Geomagnetic Storm Drivers Derived Solely From Ground-Based Magnetic Field Measurements text January 2020

Similar Records

Related Subjects

79 ASTRONOMY AND ASTROPHYSICS
Heliospheric and Magnetospheric Physics