skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Model evaluation guidelines for geomagnetic index predictions

Abstract

Geomagnetic indices are convenient quantities that distill the complicated physics of some region or aspect of near–Earth space into a single parameter. Most of the best–known indices are calculated from ground–based magnetometer data sets, such as Dst, SYM–H, Kp, AE, AL, and PC. Many models have been created that predict the values of these indices, often using solar wind measurements upstream from Earth as the input variables to the calculation. This document reviews the current state of models that predict geomagnetic indices and the methods used to assess their ability to reproduce the target index time series. These existing methods are synthesized into a baseline collection of metrics for benchmarking a new or updated geomagnetic index prediction model. These methods fall into two categories: (1) fit performance metrics such as root mean square error (RMSE) and mean absolute error (MAE) that are applied to a time–series comparison of model output and observations; and (2) event detection performance metrics such as Heidke Skill Score and probability of detection (POD) that are derived from a contingency table that compares model and observation values exceeding (or not) a threshold value. Furthermore, a few examples of codes being used with this set of metricsmore » are presented, and other aspects of metrics assessment best practices, limitations, and uncertainties are discussed, including several caveats to consider when using geomagnetic indices.« less

Authors:
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3]; ORCiD logo [4]; ORCiD logo [3]; ORCiD logo [5]; ORCiD logo [6]; ORCiD logo [7]; ORCiD logo [8]; ORCiD logo [9];  [10];  [11]; ORCiD logo [12]; ORCiD logo [3]; ORCiD logo [13]; ORCiD logo [14]; ORCiD logo [15]; ORCiD logo [16]; ORCiD logo [17]; ORCiD logo [18] more »;  [19] « less
  1. Univ. of Michigan, Ann Arbor, MI (United States)
  2. Air Force Research Lab., Kirtland AFB, NM (United States)
  3. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  4. The Catholic Univ. of America, Washington, D.C. (United States); NASA Goddard Space Flight Center (GSFC), Greenbelt, MD (United States)
  5. Univ. of Alcala, Madrid (Spain)
  6. Space Environment Technologies, Pacific Palisades, CA (United States)
  7. Swedish Institute of Space Physics, Lund (Sweden)
  8. Univ. of Michigan, Ann Arbor, MI (United States); Finnish Meteorological Institute, Helsinki (Finland)
  9. Univ. of Michigan, Ann Arbor, MI (United States); Univ. of Texas, Arlington, TX (United States)
  10. UK Met Office, Devon (United Kingdom)
  11. Univ. of Sheffield, South Yorkshire (United Kingdom)
  12. Swedish Institute of Space Physics, Uppsala (Sweden)
  13. George Mason Univ., Fairfax, VA (United States)
  14. National Oceanic and Atmospheric Administration, Boulder, CO (United States)
  15. CAS, Prague (Czech Republic)
  16. Department of Space Geodesy CNES, Toulouse (France)
  17. GFZ German Research Centre for Geosciences, Telegrafenberg, Potsdam (Germany); Univ. of Potsdam, Potsdam (Germany)
  18. GFZ German Research Centre for Geosciences, Telegrafenberg, Potsdam (Germany); Univ. of Potsdam, Potsdam (Germany); Univ. of California, Los Angeles, CA (United States)
  19. GFZ German Research Centre for Geosciences, Telegrafenberg, Potsdam (Germany)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE Laboratory Directed Research and Development (LDRD) Program
OSTI Identifier:
1482013
Report Number(s):
LA-UR-18-28370
Journal ID: ISSN 1542-7390
Grant/Contract Number:  
AC52-06NA25396
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
Space Weather
Additional Journal Information:
Journal Name: Space Weather; Journal ID: ISSN 1542-7390
Publisher:
American Geophysical Union
Country of Publication:
United States
Language:
English
Subject:
79 ASTRONOMY AND ASTROPHYSICS; Heliospheric and Magnetospheric Physics

Citation Formats

Liemohn, Michael W., McCollough, James P., Jordanova, Vania K., Ngwira, Chigomezyo M., Morley, Steven Karl, Cid, Consuelo, Tobiska, W. Kent, Wintoft, Peter, Ganushkina, Natalia Yu, Welling, Daniel T., Bingham, Suzy, Balikhin, Michael A., Opgenoorth, Hermann J., Engel, Miles A., Weigel, Robert S., Singer, Howard J., Buresova, Dalia, Bruinsma, Sean, Zhelavskaya, Irina S., Shprits, Yuri Y., and Vasile, Ruggero. Model evaluation guidelines for geomagnetic index predictions. United States: N. p., 2018. Web. doi:10.1029/2018SW002067.
Liemohn, Michael W., McCollough, James P., Jordanova, Vania K., Ngwira, Chigomezyo M., Morley, Steven Karl, Cid, Consuelo, Tobiska, W. Kent, Wintoft, Peter, Ganushkina, Natalia Yu, Welling, Daniel T., Bingham, Suzy, Balikhin, Michael A., Opgenoorth, Hermann J., Engel, Miles A., Weigel, Robert S., Singer, Howard J., Buresova, Dalia, Bruinsma, Sean, Zhelavskaya, Irina S., Shprits, Yuri Y., & Vasile, Ruggero. Model evaluation guidelines for geomagnetic index predictions. United States. doi:10.1029/2018SW002067.
Liemohn, Michael W., McCollough, James P., Jordanova, Vania K., Ngwira, Chigomezyo M., Morley, Steven Karl, Cid, Consuelo, Tobiska, W. Kent, Wintoft, Peter, Ganushkina, Natalia Yu, Welling, Daniel T., Bingham, Suzy, Balikhin, Michael A., Opgenoorth, Hermann J., Engel, Miles A., Weigel, Robert S., Singer, Howard J., Buresova, Dalia, Bruinsma, Sean, Zhelavskaya, Irina S., Shprits, Yuri Y., and Vasile, Ruggero. Tue . "Model evaluation guidelines for geomagnetic index predictions". United States. doi:10.1029/2018SW002067.
@article{osti_1482013,
title = {Model evaluation guidelines for geomagnetic index predictions},
author = {Liemohn, Michael W. and McCollough, James P. and Jordanova, Vania K. and Ngwira, Chigomezyo M. and Morley, Steven Karl and Cid, Consuelo and Tobiska, W. Kent and Wintoft, Peter and Ganushkina, Natalia Yu and Welling, Daniel T. and Bingham, Suzy and Balikhin, Michael A. and Opgenoorth, Hermann J. and Engel, Miles A. and Weigel, Robert S. and Singer, Howard J. and Buresova, Dalia and Bruinsma, Sean and Zhelavskaya, Irina S. and Shprits, Yuri Y. and Vasile, Ruggero},
abstractNote = {Geomagnetic indices are convenient quantities that distill the complicated physics of some region or aspect of near–Earth space into a single parameter. Most of the best–known indices are calculated from ground–based magnetometer data sets, such as Dst, SYM–H, Kp, AE, AL, and PC. Many models have been created that predict the values of these indices, often using solar wind measurements upstream from Earth as the input variables to the calculation. This document reviews the current state of models that predict geomagnetic indices and the methods used to assess their ability to reproduce the target index time series. These existing methods are synthesized into a baseline collection of metrics for benchmarking a new or updated geomagnetic index prediction model. These methods fall into two categories: (1) fit performance metrics such as root mean square error (RMSE) and mean absolute error (MAE) that are applied to a time–series comparison of model output and observations; and (2) event detection performance metrics such as Heidke Skill Score and probability of detection (POD) that are derived from a contingency table that compares model and observation values exceeding (or not) a threshold value. Furthermore, a few examples of codes being used with this set of metrics are presented, and other aspects of metrics assessment best practices, limitations, and uncertainties are discussed, including several caveats to consider when using geomagnetic indices.},
doi = {10.1029/2018SW002067},
journal = {Space Weather},
issn = {1542-7390},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
This content will become publicly available on November 6, 2019
Publisher's Version of Record

Save / Share: