skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.

Abstract

Seasonal influenza results in substantial annual morbidity and mortality in the United States and worldwide. Accurate forecasts of key features of influenza epidemics, such as the timing and severity of the peak incidence in a given season, can inform public health response to outbreaks. As part of ongoing efforts to incorporate data and advanced analytical methods into public health decision-making, the United States Centers for Disease Control and Prevention (CDC) has organized seasonal influenza forecasting challenges since the 2013/2014 season. In the 2017/2018 season, 22 teams participated. A subset of four teams created a research consortium called the FluSight Network in early 2017. During the 2017/2018 season they worked together to produce a collaborative multi-model ensemble that combined 21 separate component models into a single model using a machine learning technique called stacking. This approach creates a weighted average of predictive densities where the weight for each component is determined by maximizing overall ensemble accuracy over past seasons. In the 2017/2018 influenza season, one of the largest seasonal outbreaks in the last 15 years, this multi-model ensemble performed better on average than all individual component models and placed second overall in the CDC challenge. It also outperformed the baseline multi-modelmore » ensemble created by the CDC that took a simple average of all models submitted to the forecasting challenge. This project shows that collaborative efforts between research teams to develop ensemble forecasting approaches can bring measurable improvements in forecast accuracy and important reductions in the variability of performance from year to year. Efforts such as this, that emphasize real-time testing and evaluation of forecasting models and facilitate the close collaboration between public health officials and modeling researchers, are essential to improving our understanding of how best to use forecasts to improve public health response to seasonal and emerging epidemic threats.« less

Authors:
ORCiD logo [1];  [2]; ORCiD logo [3]; ORCiD logo [1]; ORCiD logo [4]; ORCiD logo [5]; ORCiD logo [3];  [6];  [7];  [1];  [1];  [8]; ORCiD logo [2];  [9]; ORCiD logo [6]; ORCiD logo [3]
  1. Univ. of Massachusetts, Amherst, MA (United States)
  2. Centers for Disease Control and Prevention, Atlanta, GA (United States)
  3. Columbia Univ., New York, NY (United States)
  4. Mount Holyoke College, South Hadley, MA (United States)
  5. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  6. Carnegie Mellon Univ., Pittsburgh, PA (United States)
  7. Smith College, Northampton, MA (United States)
  8. Amherst College, MA (United States)
  9. Centers for Disease Control and Prevention, San Juan, PR (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA); National Institutes of Health (NIH); Defense Advanced Research Projects Agency (DARPA); Defense Threat Reduction Agency (DTRA); Uptake Technologies
OSTI Identifier:
1604056
Report Number(s):
LA-UR-20-22025
Journal ID: ISSN 1553-7358
Grant/Contract Number:  
89233218CNA000001; R35GM119582; HDTRA1-18-C-0008; 5U54GM088491; 0946825; DGE-1252522; DGE-1745016; GM110748
Resource Type:
Accepted Manuscript
Journal Name:
PLoS Computational Biology (Online)
Additional Journal Information:
Journal Name: PLoS Computational Biology (Online); Journal Volume: 15; Journal Issue: 11; Journal ID: ISSN 1553-7358
Publisher:
Public Library of Science
Country of Publication:
United States
Language:
English
Subject:
59 BASIC BIOLOGICAL SCIENCES; 97 MATHEMATICS AND COMPUTING; Biological Science; Mathematics

Citation Formats

Reich, Nicholas G., McGowan, Craig J., Yamana, Teresa K., Tushar, Abhinav, Ray, Evan L., Osthus, David Allen, Kandula, Sasikiran, Brooks, Logan C., Crawford-Crudell, Willow, Gibson, Graham Casey, Moore, Evan, Silva, Rebecca, Biggerstaff, Matthew, Johansson, Michael A., Rosenfeld, Roni, and Shaman, Jeffrey. Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.. United States: N. p., 2019. Web. doi:10.1371/journal.pcbi.1007486.
Reich, Nicholas G., McGowan, Craig J., Yamana, Teresa K., Tushar, Abhinav, Ray, Evan L., Osthus, David Allen, Kandula, Sasikiran, Brooks, Logan C., Crawford-Crudell, Willow, Gibson, Graham Casey, Moore, Evan, Silva, Rebecca, Biggerstaff, Matthew, Johansson, Michael A., Rosenfeld, Roni, & Shaman, Jeffrey. Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.. United States. doi:https://doi.org/10.1371/journal.pcbi.1007486
Reich, Nicholas G., McGowan, Craig J., Yamana, Teresa K., Tushar, Abhinav, Ray, Evan L., Osthus, David Allen, Kandula, Sasikiran, Brooks, Logan C., Crawford-Crudell, Willow, Gibson, Graham Casey, Moore, Evan, Silva, Rebecca, Biggerstaff, Matthew, Johansson, Michael A., Rosenfeld, Roni, and Shaman, Jeffrey. Fri . "Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.". United States. doi:https://doi.org/10.1371/journal.pcbi.1007486. https://www.osti.gov/servlets/purl/1604056.
@article{osti_1604056,
title = {Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.},
author = {Reich, Nicholas G. and McGowan, Craig J. and Yamana, Teresa K. and Tushar, Abhinav and Ray, Evan L. and Osthus, David Allen and Kandula, Sasikiran and Brooks, Logan C. and Crawford-Crudell, Willow and Gibson, Graham Casey and Moore, Evan and Silva, Rebecca and Biggerstaff, Matthew and Johansson, Michael A. and Rosenfeld, Roni and Shaman, Jeffrey},
abstractNote = {Seasonal influenza results in substantial annual morbidity and mortality in the United States and worldwide. Accurate forecasts of key features of influenza epidemics, such as the timing and severity of the peak incidence in a given season, can inform public health response to outbreaks. As part of ongoing efforts to incorporate data and advanced analytical methods into public health decision-making, the United States Centers for Disease Control and Prevention (CDC) has organized seasonal influenza forecasting challenges since the 2013/2014 season. In the 2017/2018 season, 22 teams participated. A subset of four teams created a research consortium called the FluSight Network in early 2017. During the 2017/2018 season they worked together to produce a collaborative multi-model ensemble that combined 21 separate component models into a single model using a machine learning technique called stacking. This approach creates a weighted average of predictive densities where the weight for each component is determined by maximizing overall ensemble accuracy over past seasons. In the 2017/2018 influenza season, one of the largest seasonal outbreaks in the last 15 years, this multi-model ensemble performed better on average than all individual component models and placed second overall in the CDC challenge. It also outperformed the baseline multi-model ensemble created by the CDC that took a simple average of all models submitted to the forecasting challenge. This project shows that collaborative efforts between research teams to develop ensemble forecasting approaches can bring measurable improvements in forecast accuracy and important reductions in the variability of performance from year to year. Efforts such as this, that emphasize real-time testing and evaluation of forecasting models and facilitate the close collaboration between public health officials and modeling researchers, are essential to improving our understanding of how best to use forecasts to improve public health response to seasonal and emerging epidemic threats.},
doi = {10.1371/journal.pcbi.1007486},
journal = {PLoS Computational Biology (Online)},
number = 11,
volume = 15,
place = {United States},
year = {2019},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 4 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Comparison of Filtering Methods for the Modeling and Retrospective Forecasting of Influenza Epidemics
journal, April 2014


The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt
journal, March 2018


Individual versus superensemble forecasts of seasonal influenza outbreaks in the United States
journal, November 2017


A human judgment approach to epidemiological forecasting
journal, March 2017


Improved Weather and Seasonal Climate Forecasts from Multimodel Superensemble
journal, September 1999


Real-time influenza forecasts during the 2012–2013 season
journal, December 2013

  • Shaman, Jeffrey; Karspeck, Alicia; Yang, Wan
  • Nature Communications, Vol. 4, Issue 1
  • DOI: 10.1038/ncomms3837

Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions
journal, June 2018


Stacked generalization
journal, January 1992


Subregional Nowcasts of Seasonal Influenza Using Search Trends
journal, January 2017

  • Kandula, Sasikiran; Hsu, Daniel; Shaman, Jeffrey
  • Journal of Medical Internet Research, Vol. 19, Issue 11
  • DOI: 10.2196/jmir.7486

Type- and Subtype-Specific Influenza Forecast
journal, February 2017

  • Kandula, Sasikiran; Yang, Wan; Shaman, Jeffrey
  • American Journal of Epidemiology, Vol. 185, Issue 5
  • DOI: 10.1093/aje/kww211

Accurate estimation of influenza epidemics using Google search data via ARGO
journal, November 2015

  • Yang, Shihao; Santillana, Mauricio; Kou, S. C.
  • Proceedings of the National Academy of Sciences, Vol. 112, Issue 47
  • DOI: 10.1073/pnas.1515373112

The Combination of Forecasts
journal, December 1969

  • Bates, J. M.; Granger, C. W. J.
  • Journal of the Operational Research Society, Vol. 20, Issue 4
  • DOI: 10.1057/jors.1969.103

Superensemble forecasts of dengue outbreaks
journal, October 2016

  • Yamana, Teresa K.; Kandula, Sasikiran; Shaman, Jeffrey
  • Journal of The Royal Society Interface, Vol. 13, Issue 123
  • DOI: 10.1098/rsif.2016.0410

Strictly Proper Scoring Rules, Prediction, and Estimation
journal, March 2007

  • Gneiting, Tilmann; Raftery, Adrian E.
  • Journal of the American Statistical Association, Vol. 102, Issue 477
  • DOI: 10.1198/016214506000001437

Prediction of infectious disease epidemics via weighted density ensembles
journal, February 2018


Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture: L. HELD, S. MEYER AND J. BRACHER
journal, June 2017

  • Held, Leonhard; Meyer, Sebastian; Bracher, Johannes
  • Statistics in Medicine, Vol. 36, Issue 22
  • DOI: 10.1002/sim.7363

Ensemble based systems in decision making
journal, January 2006


Influenza Forecasting in Human Populations: A Scoping Review
journal, April 2014


Evaluating Density Forecasts with Applications to Financial Risk Management
journal, November 1998

  • Diebold, Francis X.; Gunther, Todd A.; Tay, Anthony S.
  • International Economic Review, Vol. 39, Issue 4
  • DOI: 10.2307/2527342

Forecasting the spatial transmission of influenza in the United States
journal, February 2018

  • Pei, Sen; Kandula, Sasikiran; Yang, Wan
  • Proceedings of the National Academy of Sciences, Vol. 115, Issue 11
  • DOI: 10.1073/pnas.1708856115

The Probability Integral Transform and Related Results
journal, December 1994


Using Bayesian Model Averaging to Calibrate Forecast Ensembles
journal, May 2005

  • Raftery, Adrian E.; Gneiting, Tilmann; Balabdaoui, Fadoua
  • Monthly Weather Review, Vol. 133, Issue 5, p. 1155-1174
  • DOI: 10.1175/MWR2906.1

Predicting lymphatic filariasis transmission and elimination dynamics using a multi-model ensemble framework
journal, March 2017