Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Evaluation of individual and ensemble probabilistic forecasts of COVID-19 mortality in the United States

Journal Article · · Proceedings of the National Academy of Sciences of the United States of America
 [1];  [1];  [2];  [3];  [4];  [5];  [5];  [6];  [5];  [5];  [5];  [5];  [5];  [5];  [7];  [8];  [5];  [5];  [5];  [5] more »;  [5];  [9];  [10];  [10];  [10];  [10];  [10];  [10];  [10];  [10];  [10];  [11];  [12];  [13];  [12];  [13];  [14];  [12];  [15];  [16];  [17];  [18];  [19];  [18];  [18];  [18];  [18];  [18];  [19];  [18];  [20];  [18];  [19];  [21];  [22];  [21];  [21];  [23];  [24];  [23];  [25];  [26];  [27];  [25];  [25];  [28];  [25];  [25];  [25];  [29];  [30];  [31];  [32];  [33];  [34];  [35];  [35];  [35];  [36];  [37];  [38];  [39];  [39];  [39];  [39];  [39];  [39];  [40];  [40];  [41];  [42];  [42];  [43];  [42];  [44];  [45];  [46];  [47];  [48];  [49];  [50];  [50];  [51];  [50];  [52];  [53];  [54];  [55];  [56];  [57];  [52];  [52];  [52];  [58];  [52];  [57];  [58];  [57];  [57];  [52];  [58];  [59];  [60];  [56];  [61];  [61];  [61];  [62];  [61];  [63];  [63];  [64];  [64];  [64];  [64];  [64];  [64];  [64];  [64];  [64];  [65];  [66];  [66];  [66];  [66];  [66];  [67];  [67];  [68];  [67];  [68];  [67];  [67];  [69];  [69];  [69];  [70];  [69];  [69];  [69];  [71];  [71];  [71];  [72];  [72];  [73];  [74];  [75];  [76];  [76];  [75];  [77];  [78];  [78];  [78];  [78];  [78];  [78];  [79];  [79];  [79];  [80];  [79];  [81];  [82];  [79];  [83];  [79];  [79];  [79];  [84];  [79];  [85];  [84];  [86];  [86];  [86];  [87];  [8];  [88];  [8];  [8];  [89];  [90];  [8];  [91];  [92];  [92];  [92];  [92];  [92];  [92];  [92];  [93];  [93];  [94];  [93];  [95];  [96];  [95];  [95];  [97];  [95];  [98];  [99];  [98];  [98];  [98];  [100];  [100];  [100];  [100];  [100];  [100];  [100];  [100];  [100];  [101];  [100];  [100];  [100];  [102];  [102];  [102];  [102];  [102];  [102];  [103];  [104];  [105];  [105];  [105];  [67];  [68];  [67];  [67];  [67];  [106];  [107];  [107];  [108];  [107];  [109];  [107];  [98];  [110];  [111];  [111];  [112];  [113];  [114];  [113];  [113];  [113];  [115];  [116];  [117];  [118];  [117];  [119];  [113];  [118];  [113];  [113];  [120];  [120];  [121];  [122];  [123];  [124];  [125];  [125];  [125];  [125];  [5] « less
  1. University of Massachusetts, Amherst, MA (United States)
  2. Centers for Disease Control and Prevention, Atlanta, GA (United States)
  3. Karlsruhe Institute of Technology, Karlsruhe (Germany); ,Computational Statistics Group, Heidelberg Institute for Theoretical Studies, 69118 Heidelberg, Germany
  4. IQT Labs, In-Q-Tel, Waltham, MA 02451
  5. Department of Biostatistics and Epidemiology, University of Massachusetts, Amherst, MA 01003
  6. Computational Statistics Group, Heidelberg Institute for Theoretical Studies, 69118 Heidelberg, Germany; Institute of Stochastics, Karlsruhe Institute of Technology, 69118 Karlsruhe, Germany
  7. Institute of Mathematical Statistics and Actuarial Science, University of Bern, CH-3012 Bern, Switzerland
  8. Department of Statistics, Iowa State University, Ames, IA 50011
  9. Unaffiliated, New York, NY 10016
  10. Wadhwani Institute of Artificial Intelligence, Andheri East, Mumbai, 400093, India
  11. University of Washington, Seattle, WA 98109
  12. Department of Integrative Biology, University of Texas at Austin, Austin, TX 78712
  13. Texas Advanced Computing Center, Austin, TX 78758
  14. Santa Fe Institute, Santa Fe, NM 87501
  15. Department of Information, Risk, and Operations Management, University of Texas at Austin, Austin, TX 78712
  16. Department of Statistics and Data Sciences, University of Texas at Austin, Austin, TX 78712
  17. Ming Hsieh Department of Computer and Electrical Engineering, University of Southern California, Los Angeles, CA 90089
  18. US Army Engineer Research and Development Center, Vicksburg, MS 39180
  19. US Army Engineer Research and Development Center, Concord, MA 01742
  20. US Army Engineer Research and Development Center, Hanover, NH 03755
  21. Department of Psychiatry and Behavioral Sciences, State University of New York Upstate Medical University, Syracuse, NY 13210
  22. School of Medicine, State University of New York Upstate Medical University, Syracuse, NY 13210
  23. Department of Public Health & Preventive Medicine, State University of New York Upstate Medical University, Syracuse, NY 13210
  24. Department of Electrical Engineering and Computer Science, Syracuse University, Syracuse, NY 13207
  25. Department of Physics, University of Michigan, Ann Arbor, MI, 48109
  26. Department of Physics, Trinity University, San Antonio, TX 78212
  27. Department of Complex Systems, University of Michigan, Ann Arbor, MI 48109; Department of Mathematics, University of Michigan, Ann Arbor, MI 48109; School of Public Health, Department of Epidemiology, University of Michigan, Ann Arbor, MI 48109
  28. School of Public Health, Department of Epidemiology, University of Michigan, Ann Arbor, MI 48109
  29. College of Information and Computer Sciences, University of Massachusetts, Amherst, MA 01003
  30. School of Public Health and Health Sciences, University of Massachusetts, Amherst, MA 01003
  31. Department of Computer Science and Engineering, University of California, San Diego, CA 92093; Khoury College of Computer Sciences, Northeastern University, Boston, MA 02115
  32. Department of Statistics, University of Washington, Seattle, WA 98185
  33. Halıcıoğlu Data Science Institute, University of California, San Diego, CA 92093
  34. Department of Computer Science and Engineering, University of California, San Diego, CA 92093
  35. Department of Computer Science, University of California, Santa Barbara, CA 93106
  36. Mechatronics, Embedded Systems and Automation Lab, Department of Mechanical Engineering, University of California, Merced, CA 95301
  37. Jilin University, Changchun City, Jilin Province, 130012, People's Republic of China
  38. University of Science and Technology of China, Heifei, Anhui, 230027, People's Republic of China
  39. Department of Computer Science, University of California, Los Angeles, CA 90095
  40. Department of Mathematics, University of Arizona, Tucson, AZ 85721
  41. Construx, Bellevue, WA 98004
  42. Quality Assurance and Data Science, Signature Science, LLC, Charlottesville, VA 22911
  43. Quality Assurance and Data Science, Signature Science, LLC, Austin, TX 78759
  44. Department of Materials Science and Engineering, Rensselaer Polytechnic Institute, Troy, NY 12309
  45. Department of Civil and Environmental Engineering, University of Washington, Seattle, WA 98195
  46. Unaffiliated, Davis, CA 95616
  47. School for Engineering of Matter, Transport and Energy, Arizona State University, Tempe, AZ 85287; School of Engineering, Brown University, Providence, RI 02912
  48. Manhasset Secondary School, Manhasset, NY 11030
  49. School of Engineering, Brown University, Providence, RI 02912
  50. Infectious Disease Group, Predictive Science, Inc, San Diego, CA 92121
  51. Medical Research Council Centre for Global Infectious Disease Analysis, Department of Infectious Disease Epidemiology, School of Public Health, Imperial College, W2 1PG London, United Kingdom
  52. Financial Services, Oliver Wyman, New York, NY 10036
  53. Oliver Wyman Digital, Oliver Wyman, Boston, MA 02110
  54. Oliver Wyman Digital, Oliver Wyman, Sao Paolo, Brazil 04711-904
  55. Health & Life Sciences, Oliver Wyman, Boston, MA 02110
  56. Oliver Wyman Digital, Oliver Wyman, New York, NY 10036
  57. Health & Life Sciences, Oliver Wyman, New York, NY 10036
  58. Core Consultant Group, Oliver Wyman, New York, NY 10036
  59. Financial Services, Oliver Wyman Digital, Toronto, ON, Canada M5J 0A1
  60. Financial Services, Oliver Wyman, London, UK W1U 8EW
  61. Department of Biological Sciences, University of Notre Dame, Notre Dame, IN 46556
  62. Department of Biological Sciences, University of Notre Dame, Notre Dame, IN 46556; Department of Ecology and Evolution, University of Chicago, Chicago, IL 60637
  63. Department of Mathematics and Statistics, Masaryk University, 61137 Brno, Czech Republic
  64. Microsoft, Redmond, WA 98029
  65. Institute for Scientific Interchange Foundation, Turin, 10133, Italy; Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115
  66. Laboratory for the Modeling of Biological and Socio-technical Systems, Northeastern University, Boston, MA 02115
  67. Operations Research Center, Massachusetts Institute of Technology; Cambridge, MA 02139
  68. Sloan School of Management, Massachusetts Institute of Technology, Cambridge, MA 02142
  69. Massachusetts Institute of Technology, Cambridge, MA 02142
  70. Technology, Operations and Statistics (TOPS) group, Stern School of Business, New York University, New York, NY 10012
  71. Institute for Data, Systems, and Society, Massachusetts Institute of Technology, Cambridge, MA 02139
  72. Laboratory for Computational Physiology, Massachusetts Institute of Technology, Cambridge, MA 02139
  73. River Hill High School, Clarksville, MD 21029
  74. JMP Life Sciences, SAS Institute Inc, Cary, NC 27513
  75. Statistical Sciences Group, Los Alamos National Laboratory, Los Alamos, NM 87545
  76. Information Systems and Modeling Group, Los Alamos National Laboratory, Los Alamos, NM 87545
  77. Department of Physics and Astronomy, University of Victoria, Victoria, BC, V8W 2Y2, Canada; Physical Sciences Division, TRIUMF, Vancouver, BC, V8W 2Y2, Canada
  78. Johns Hopkins University Applied Physics Laboratory, Laurel, MD 20723
  79. Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD 21215
  80. Institute for Computational Medicine, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD 21218
  81. Unaffiliated, Baltimore, MD 21205
  82. Division of Epidemiology, Department of Internal Medicine, University of Utah, Salt Lake City, UT 84108
  83. Laboratory of Ecohydrology, School of Architecture, Civil and Environmental Engineering, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland
  84. Unaffiliated, San Francisco, CA 94122
  85. Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD 21215; International Vaccine Access Center, Johns Hopkins University, Baltimore, MD 21231; Department of International Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD 21231
  86. Department of Civil and Systems Engineering, Johns Hopkins University, Baltimore, MD 21218
  87. Unaffiliated, Amsterdam, The Netherlands
  88. Department of Finance, Iowa State University, Ames, IA 50011
  89. School of Mathematical and Statistical Sciences, Clemson University, Clemson, SC 29634
  90. Department of Mathematics, College of William & Mary, Williamsburg, VA 23187
  91. Department of Statistics, University of Virginia, Charlottesville, VA 22904
  92. Institute for Health Metrics and Evaluation, University of Washington, Seattle, WA 98195
  93. Emerging Technologies, IEM, Inc., Bel Air, MD 21015
  94. Emerging Technologies, IEM, Inc., Baton Rouge, LA 70809
  95. College of Computing, Georgia Institute of Technology, Atlanta, GA 30308
  96. Department of Computer Science, University of Iowa, Iowa City, IA 52242
  97. Department of Computer Science, Virginia Tech, Falls Church, VA 22043
  98. H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, 30332
  99. Advanced Data Analytics, Metron, Inc., Reston, VA 20190
  100. Google Cloud, Sunnyvale, CA 94089
  101. Department of Health Policy and Management, Harvard University, Cambridge, MA 02138
  102. Centre for Mathematical Modelling of Infectious Diseases, London School of Hygiene & Tropical Medicine, WC1E 7HT London, United Kingdom
  103. McCombs School of Business, The University of Texas at Austin, Austin, TX 78712
  104. Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712
  105. Department of Environmental Health Sciences, Columbia University, New York, NY 10032
  106. H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, GA, 30332; Winship Cancer Institute, Emory University Medical School, Atlanta, GA 30322
  107. Radiology-Institute for Technology Assessment, Massachusetts General Hospital, Boston, MA 02114
  108. Health Economic Modeling, Value Analytics Labs, 34776 İstanbul, Turkey
  109. Department of Medicine, Section of Infectious Diseases, Boston University School of Medicine, Boston, MA 02118
  110. Department of Biostatistics, Columbia University, New York, NY 10032; Department of Psychiatry, Columbia University, New York, NY 10032
  111. Department of Biostatistics, Columbia University, New York, NY 10032
  112. Department of Biostatistics, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599
  113. Department of Statistics, Carnegie Mellon University, Pittsburgh, PA 15213
  114. Marshall School of Business, Department of Data Sciences and Operations (DSO), University of Southern California, Los Angeles, CA 90089
  115. Department of Statistics, University of British Columbia, Vancouver, BC, V6T 1Z4, Canada
  116. Department of Biomedical Data Sciences, Stanford University, Stanford, CA 94305; Department of Statistics, Stanford University, Stanford, CA 94305
  117. Machine Learning Department, Carnegie Mellon University, Pittsburgh, PA 15213
  118. Department of Statistics, Stanford University, Stanford, CA 94305
  119. Department of Biostatistics, University of Washington, Seattle, WA 98195
  120. Odum School of Ecology, University of Georgia, Athens, GA 30602
  121. Unaffiliated, Tucson, AZ 85704
  122. Catalog Data Science, Walmart Inc., Sunnyvale, CA 94085
  123. Department of Mathematics and Statistics, Dalhousie University, Halifax, NS, B3H 4R2, Canada
  124. Virtual Power System Inc, Milpitas, CA 95035
  125. COVID-19 Response, Centers for Disease Control and Prevention; Atlanta, GA 30333
+ Show Author Affiliations
This paper compares the probabilistic accuracy of short-term forecasts of reported deaths due to COVID-19 during the first year and a half of the pandemic in the United States. Results show high variation in accuracy between and within stand-alone models and more consistent accuracy from an ensemble model that combined forecasts from all eligible models. This demonstrates that an ensemble model provided a reliable and comparatively accurate means of forecasting deaths during the COVID-19 pandemic that exceeded the performance of all of the models that contributed to it. This work strengthens the evidence base for synthesizing multiple models to support public-health action.
Research Organization:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Organization:
USDOE Laboratory Directed Research and Development (LDRD) Program: National Science Foundation (NSF)
Contributing Organization:
COVID-19 Forecast Hub
Grant/Contract Number:
89233218CNA000001
OSTI ID:
1893661
Report Number(s):
LA-UR-21-20490
Journal Information:
Proceedings of the National Academy of Sciences of the United States of America, Journal Name: Proceedings of the National Academy of Sciences of the United States of America Journal Issue: 15 Vol. 119; ISSN 0027-8424
Publisher:
National Academy of SciencesCopyright Statement
Country of Publication:
United States
Language:
English

References (21)

An interactive web-based dashboard to track COVID-19 in real time journal May 2020
The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt journal March 2018
Ensemble forecasting journal March 2008
A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave journal August 2021
The Zoltar forecast archive, a tool to standardize and store interdisciplinary prediction research journal February 2021
Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016 journal January 2019
The Combination of Forecasts journal December 1969
A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States journal January 2019
An open challenge to advance probabilistic forecasting for dengue epidemics journal November 2019
Epidemic Forecasting is Messier Than Weather Forecasting: The Role of Human Behavior and Internet Data Streams in Epidemic Forecast journal November 2016
Outbreak analytics: a developing data science for informing the response to emerging pathogens
  • Polonsky, Jonathan A.; Baidjoe, Amrish; Kamvar, Zhian N.
  • Philosophical Transactions of the Royal Society B: Biological Sciences, Vol. 374, Issue 1776 https://doi.org/10.1098/rstb.2018.0276
journal May 2019
Ensemble based systems in decision making journal January 2006
ATMOSPHERIC SCIENCE: Weather Forecasting with Ensemble Methods journal October 2005
Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble journal September 1999
Web metasearch conference January 2003
Applying infectious disease forecasting to public health: a path forward using influenza forecasting examples journal December 2019
Strictly Proper Scoring Rules, Prediction, and Estimation journal March 2007
Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S. journal November 2019
Evaluating epidemic forecasts in an interval format journal February 2021
Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines journal October 2021
Impact of Rank-Based Normalizing Transformations on the Accuracy of Test Scores journal November 2009

Similar Records

The United States COVID-19 Forecast Hub dataset
Journal Article · Sun Jul 31 20:00:00 EDT 2022 · Scientific Data · OSTI ID:1975006
A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave
Journal Article · Thu Aug 26 20:00:00 EDT 2021 · Nature Communications · OSTI ID:1832375

Related Subjects

60 APPLIED LIFE SCIENCES
COVID-19
ensemble forecast
forecasting
model evaluation