Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.

Abstract

Seasonal influenza results in substantial annual morbidity and mortality in the United States and worldwide. Accurate forecasts of key features of influenza epidemics, such as the timing and severity of the peak incidence in a given season, can inform public health response to outbreaks. As part of ongoing efforts to incorporate data and advanced analytical methods into public health decision-making, the United States Centers for Disease Control and Prevention (CDC) has organized seasonal influenza forecasting challenges since the 2013/2014 season. In the 2017/2018 season, 22 teams participated. A subset of four teams created a research consortium called the FluSight Network in early 2017. During the 2017/2018 season they worked together to produce a collaborative multi-model ensemble that combined 21 separate component models into a single model using a machine learning technique called stacking. This approach creates a weighted average of predictive densities where the weight for each component is determined by maximizing overall ensemble accuracy over past seasons. In the 2017/2018 influenza season, one of the largest seasonal outbreaks in the last 15 years, this multi-model ensemble performed better on average than all individual component models and placed second overall in the CDC challenge. It also outperformed the baseline multi-modelmore » ensemble created by the CDC that took a simple average of all models submitted to the forecasting challenge. This project shows that collaborative efforts between research teams to develop ensemble forecasting approaches can bring measurable improvements in forecast accuracy and important reductions in the variability of performance from year to year. Efforts such as this, that emphasize real-time testing and evaluation of forecasting models and facilitate the close collaboration between public health officials and modeling researchers, are essential to improving our understanding of how best to use forecasts to improve public health response to seasonal and emerging epidemic threats.« less

Authors:
ORCiD logo [1];  [2]; ORCiD logo [3]; ORCiD logo [1]; ORCiD logo [4]; ORCiD logo [5]; ORCiD logo [3];  [6];  [7];  [1];  [1];  [8]; ORCiD logo [2];  [9]; ORCiD logo [6]; ORCiD logo [3]
+ Show Author Affiliations
  1. Univ. of Massachusetts, Amherst, MA (United States)
  2. Centers for Disease Control and Prevention, Atlanta, GA (United States)
  3. Columbia Univ., New York, NY (United States)
  4. Mount Holyoke College, South Hadley, MA (United States)
  5. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  6. Carnegie Mellon Univ., Pittsburgh, PA (United States)
  7. Smith College, Northampton, MA (United States)
  8. Amherst College, MA (United States)
  9. Centers for Disease Control and Prevention, San Juan, PR (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA); National Institutes of Health (NIH); Defense Advanced Research Projects Agency (DARPA); Defense Threat Reduction Agency (DTRA); Uptake Technologies
OSTI Identifier:
1604056
Report Number(s):
LA-UR-20-22025
Journal ID: ISSN 1553-7358
Grant/Contract Number:  
89233218CNA000001; R35GM119582; HDTRA1-18-C-0008; 5U54GM088491; 0946825; DGE-1252522; DGE-1745016; GM110748
Resource Type:
Accepted Manuscript
Journal Name:
PLoS Computational Biology (Online)
Additional Journal Information:
Journal Name: PLoS Computational Biology (Online); Journal Volume: 15; Journal Issue: 11; Journal ID: ISSN 1553-7358
Publisher:
Public Library of Science
Country of Publication:
United States
Language:
English
Subject:
59 BASIC BIOLOGICAL SCIENCES; 97 MATHEMATICS AND COMPUTING; Biological Science; Mathematics

Citation Formats

Reich, Nicholas G., McGowan, Craig J., Yamana, Teresa K., Tushar, Abhinav, Ray, Evan L., Osthus, David Allen, Kandula, Sasikiran, Brooks, Logan C., Crawford-Crudell, Willow, Gibson, Graham Casey, Moore, Evan, Silva, Rebecca, Biggerstaff, Matthew, Johansson, Michael A., Rosenfeld, Roni, and Shaman, Jeffrey. Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.. United States: N. p., 2019. Web. doi:10.1371/journal.pcbi.1007486.
Copy to clipboard
Reich, Nicholas G., McGowan, Craig J., Yamana, Teresa K., Tushar, Abhinav, Ray, Evan L., Osthus, David Allen, Kandula, Sasikiran, Brooks, Logan C., Crawford-Crudell, Willow, Gibson, Graham Casey, Moore, Evan, Silva, Rebecca, Biggerstaff, Matthew, Johansson, Michael A., Rosenfeld, Roni, & Shaman, Jeffrey. Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.. United States. https://doi.org/10.1371/journal.pcbi.1007486
Copy to clipboard
Reich, Nicholas G., McGowan, Craig J., Yamana, Teresa K., Tushar, Abhinav, Ray, Evan L., Osthus, David Allen, Kandula, Sasikiran, Brooks, Logan C., Crawford-Crudell, Willow, Gibson, Graham Casey, Moore, Evan, Silva, Rebecca, Biggerstaff, Matthew, Johansson, Michael A., Rosenfeld, Roni, and Shaman, Jeffrey. Fri . "Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.". United States. https://doi.org/10.1371/journal.pcbi.1007486. https://www.osti.gov/servlets/purl/1604056.
Copy to clipboard
@article{osti_1604056,
title = {Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.},
author = {Reich, Nicholas G. and McGowan, Craig J. and Yamana, Teresa K. and Tushar, Abhinav and Ray, Evan L. and Osthus, David Allen and Kandula, Sasikiran and Brooks, Logan C. and Crawford-Crudell, Willow and Gibson, Graham Casey and Moore, Evan and Silva, Rebecca and Biggerstaff, Matthew and Johansson, Michael A. and Rosenfeld, Roni and Shaman, Jeffrey},
abstractNote = {Seasonal influenza results in substantial annual morbidity and mortality in the United States and worldwide. Accurate forecasts of key features of influenza epidemics, such as the timing and severity of the peak incidence in a given season, can inform public health response to outbreaks. As part of ongoing efforts to incorporate data and advanced analytical methods into public health decision-making, the United States Centers for Disease Control and Prevention (CDC) has organized seasonal influenza forecasting challenges since the 2013/2014 season. In the 2017/2018 season, 22 teams participated. A subset of four teams created a research consortium called the FluSight Network in early 2017. During the 2017/2018 season they worked together to produce a collaborative multi-model ensemble that combined 21 separate component models into a single model using a machine learning technique called stacking. This approach creates a weighted average of predictive densities where the weight for each component is determined by maximizing overall ensemble accuracy over past seasons. In the 2017/2018 influenza season, one of the largest seasonal outbreaks in the last 15 years, this multi-model ensemble performed better on average than all individual component models and placed second overall in the CDC challenge. It also outperformed the baseline multi-model ensemble created by the CDC that took a simple average of all models submitted to the forecasting challenge. This project shows that collaborative efforts between research teams to develop ensemble forecasting approaches can bring measurable improvements in forecast accuracy and important reductions in the variability of performance from year to year. Efforts such as this, that emphasize real-time testing and evaluation of forecasting models and facilitate the close collaboration between public health officials and modeling researchers, are essential to improving our understanding of how best to use forecasts to improve public health response to seasonal and emerging epidemic threats.},
doi = {10.1371/journal.pcbi.1007486},
journal = {PLoS Computational Biology (Online)},
number = 11,
volume = 15,
place = {United States},
year = {2019},
month = {11}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1371/journal.pcbi.1007486
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 43 works
Citation information provided by
Web of Science
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

Comparison of Filtering Methods for the Modeling and Retrospective Forecasting of Influenza Epidemics
journal, April 2014

The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt
journal, March 2018

Individual versus superensemble forecasts of seasonal influenza outbreaks in the United States
journal, November 2017

A human judgment approach to epidemiological forecasting
journal, March 2017

Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble
journal, September 1999

Real-time influenza forecasts during the 2012–2013 season
journal, December 2013

  • Shaman, Jeffrey; Karspeck, Alicia; Yang, Wan
  • Nature Communications, Vol. 4, Issue 1
  • DOI: 10.1038/ncomms3837

Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions
journal, June 2018

Subregional Nowcasts of Seasonal Influenza Using Search Trends
journal, January 2017

  • Kandula, Sasikiran; Hsu, Daniel; Shaman, Jeffrey
  • Journal of Medical Internet Research, Vol. 19, Issue 11
  • DOI: 10.2196/jmir.7486

Type- and Subtype-Specific Influenza Forecast
journal, February 2017

  • Kandula, Sasikiran; Yang, Wan; Shaman, Jeffrey
  • American Journal of Epidemiology, Vol. 185, Issue 5
  • DOI: 10.1093/aje/kww211

Accurate estimation of influenza epidemics using Google search data via ARGO
journal, November 2015

  • Yang, Shihao; Santillana, Mauricio; Kou, S. C.
  • Proceedings of the National Academy of Sciences, Vol. 112, Issue 47
  • DOI: 10.1073/pnas.1515373112

The Combination of Forecasts
journal, December 1969

  • Bates, J. M.; Granger, C. W. J.
  • Journal of the Operational Research Society, Vol. 20, Issue 4
  • DOI: 10.1057/jors.1969.103

Superensemble forecasts of dengue outbreaks
journal, October 2016

  • Yamana, Teresa K.; Kandula, Sasikiran; Shaman, Jeffrey
  • Journal of The Royal Society Interface, Vol. 13, Issue 123
  • DOI: 10.1098/rsif.2016.0410

Strictly Proper Scoring Rules, Prediction, and Estimation
journal, March 2007

  • Gneiting, Tilmann; Raftery, Adrian E.
  • Journal of the American Statistical Association, Vol. 102, Issue 477
  • DOI: 10.1198/016214506000001437

Prediction of infectious disease epidemics via weighted density ensembles
journal, February 2018

Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture: L. HELD, S. MEYER AND J. BRACHER
journal, June 2017

  • Held, Leonhard; Meyer, Sebastian; Bracher, Johannes
  • Statistics in Medicine, Vol. 36, Issue 22
  • DOI: 10.1002/sim.7363

Ensemble based systems in decision making
journal, January 2006

Influenza Forecasting in Human Populations: A Scoping Review
journal, April 2014

Evaluating Density Forecasts with Applications to Financial Risk Management
journal, November 1998

  • Diebold, Francis X.; Gunther, Todd A.; Tay, Anthony S.
  • International Economic Review, Vol. 39, Issue 4
  • DOI: 10.2307/2527342

Forecasting the spatial transmission of influenza in the United States
journal, February 2018

  • Pei, Sen; Kandula, Sasikiran; Yang, Wan
  • Proceedings of the National Academy of Sciences, Vol. 115, Issue 11
  • DOI: 10.1073/pnas.1708856115

The Probability Integral Transform and Related Results
journal, December 1994

Using Bayesian Model Averaging to Calibrate Forecast Ensembles
journal, May 2005

  • Raftery, Adrian E.; Gneiting, Tilmann; Balabdaoui, Fadoua
  • Monthly Weather Review, Vol. 133, Issue 5, p. 1155-1174
  • DOI: 10.1175/MWR2906.1

Predicting lymphatic filariasis transmission and elimination dynamics using a multi-model ensemble framework
journal, March 2017

Climate change will affect global water availability through compounding changes in seasonal precipitation and evaporation
journal, June 2020

Optimization of probiotic therapeutics using machine learning in an artificial human gastrointestinal tract
journal, January 2021

The Combination of Forecasts
journal, January 1983

  • Winkler, Robert L.; Makridakis, Spyros
  • Journal of the Royal Statistical Society. Series A (General), Vol. 146, Issue 2
  • DOI: 10.2307/2982011

Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture
text, January 2017

  • Held, Leonhard; Meyer, Sebastian; Bracher, Johannes
  • Wiley-Blackwell Publishing, Inc.
  • DOI: 10.5167/uzh-146024

Comparison of Filtering Methods for the Modeling and Retrospective Forecasting of Influenza Epidemics
text, January 2014

  • Yang, Wan; Karspeck, Alicia; Shaman, Jeffrey L.
  • Columbia University
  • DOI: 10.7916/d80865m6

Individual versus superensemble forecasts of seasonal influenza outbreaks in the United States
text, January 2017

  • Yamana, Teresa K.; Kandula, Sasikiran; Shaman, Jeffrey L.
  • Columbia University
  • DOI: 10.7916/d8pz6n92

Influenza Forecasting in Human Populations: A Scoping Review
text, January 2014

  • Chretien, Jean-Paul; George, Dylan; Shaman, Jeffrey L.
  • Columbia University
  • DOI: 10.7916/d8sn0981

Subregional Nowcasts of Seasonal Influenza Using Search Trends
text, January 2017

  • Kandula, Sasikiran; Hsu, Daniel Joseph; Shaman, Jeffrey L.
  • Columbia University
  • DOI: 10.7916/d8-vab1-sz56

The Combination of Forecasts
journal, December 1969

  • Bates, J. M.; Granger, C. W. J.
  • OR, Vol. 20, Issue 4
  • DOI: 10.2307/3008764

The Elements of Statistical Learning
book, January 2001

Accurate estimation of influenza epidemics using Google search data via ARGO
text, January 2015

Prediction of infectious disease epidemics via weighted density ensembles
text, January 2017

    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    Improving probabilistic infectious disease forecasting through coherence
    journal, January 2021

      Sort by:
      [ × clear filter / sort ]

      Similar Records in DOE PAGES and OSTI.GOV collections: