Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Oidtman, Rachel J.; Omodei, Elisa; Kraemer, Moritz U.G.; Castañeda-Orjuela, Carlos A.; Cruz-Rivera, Erica; Misnaza-Castrillón, Sandra; Cifuentes, Myriam Patricia; Rincon, Luz Emilse; Cañon, Viviana; Alarcon, Pedro de; España, Guido; Huber, John H.; Hill, Sarah C.; Barker, Christopher M.; Johansson, Michael A.; Manore, Carrie A.; Reiner, Jr., Robert C.; Rodriguez-Barraquer, Isabel; Siraj, Amir S.; Frias-Martinez, Enrique; García-Herranz, Manuel; Perkins, T. Alex

doi:10.1038/s41467-021-25695-0

Title: Trade-offs between individual and ensemble forecasts of an emerging infectious disease

Journal Article · 10 September 2021 · Nature Communications

DOI: https://doi.org/10.1038/s41467-021-25695-0 · OSTI ID:2470493

^[1];

^[2];

^[3];

^[4]; Cruz-Rivera, Erica ^[4]; Misnaza-Castrillón, Sandra ^[4];

^[5]; Rincon, Luz Emilse ^[5]; Cañon, Viviana ^[6];

^[7]; España, Guido ^[8]; Huber, John H. ^[8];

^[9];

^[10];

^[11]; Manore, Carrie A. ^[12];

^[13]; Rodriguez-Barraquer, Isabel ^[14]; Siraj, Amir S. ^[8];

^[15] more »

University of Notre Dame, IN (United States); UNICEF, New York, NY (United States); University of Chicago, IL (United States)
UNICEF, New York, NY (United States)
University of Oxford (United Kingdom); Boston Children’s Hospital, MA (United States); Harvard Medical School, Boston, MA (United States)
Instituto Nacional de Salud, Bogotá (Colombia)
Ministerio de Salud y Protección Social, Bogotá (Colombia)
UNICEF, Bogotá (Colombia)
LUCA Telefonica Data Unit, Madrid (Spain)
University of Notre Dame, IN (United States)
University of Oxford (United Kingdom); The Royal Veterinary College, London (United Kingdom)
University of California, Davis, CA (United States)
Centers for Disease Control and Prevention, San Juan (Puerto Rico)
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
University of Washington, Seattle, WA (United States)
University of California, San Francisco, CA (United States)
Telefonica Research, Madrid (Spain)

Probabilistic forecasts play an indispensable role in answering questions about the spread of newly emerged pathogens. However, uncertainties about the epidemiology of emerging pathogens can make it difficult to choose among alternative model structures and assumptions. To assess the potential for uncertainties about emerging pathogens to affect forecasts of their spread, we evaluated the performance 16 forecasting models in the context of the 2015-2016 Zika epidemic in Colombia. Each model featured a different combination of assumptions about human mobility, spatiotemporal variation in transmission potential, and the number of virus introductions. We found that which model assumptions had the most ensemble weight changed through time. We additionally identified a trade-off whereby some individual models outperformed ensemble models early in the epidemic, but on average the ensembles outperformed all individual models. Our results suggest that multiple models spanning uncertainty across alternative assumptions are necessary to obtain robust forecasts for emerging infectious diseases.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: USDOE

Grant/Contract Number:: 89233218CNA000001

OSTI ID:: 2470493

Journal Information:: Nature Communications, Journal Name: Nature Communications Journal Issue: 1 Vol. 12; ISSN 2041-1723

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United States

Language:: English

References (59)

Making predictive modelling ART: accurate, reliable, and transparent Bodner, Korryn; Fortin, Marie‐Josée; Molnár, Péter K. Ecosphere, Vol. 11, Issue 6 https://doi.org/10.1002/ecs2.3160	journal	June 2020
Aggregating predictions from experts: A review of statistical methods, experiments, and applications McAndrew, Thomas; Wattanachit, Nutcha; Gibson, Graham C. WIREs Computational Statistics, Vol. 13, Issue 2 https://doi.org/10.1002/wics.1514	journal	June 2020
Global burden of the HIV pandemic Quinn, Thomas C. The Lancet, Vol. 348, Issue 9020 https://doi.org/10.1016/S0140-6736(96)01029-X	journal	July 1996
Potential for Zika virus introduction and transmission in resource-limited countries in Africa and the Asia-Pacific region: a modelling study Bogoch, Isaac I.; Brady, Oliver J.; Kraemer, Moritz U. G. The Lancet Infectious Diseases, Vol. 16, Issue 11 https://doi.org/10.1016/S1473-3099(16)30270-5	journal	November 2016
Spread of yellow fever virus outbreak in Angola and the Democratic Republic of the Congo 2015–16: a modelling study Kraemer, Moritz U. G.; Faria, Nuno R.; Reiner, Robert C. The Lancet Infectious Diseases, Vol. 17, Issue 3 https://doi.org/10.1016/S1473-3099(16)30513-8	journal	March 2017
The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt Viboud, Cécile; Sun, Kaiyuan; Gaffey, Robert Epidemics, Vol. 22 https://doi.org/10.1016/j.epidem.2017.08.002	journal	March 2018
Real-time forecasting of epidemic trajectories using computational dynamic ensembles Chowell, G.; Luo, R.; Sun, K. Epidemics, Vol. 30 https://doi.org/10.1016/j.epidem.2019.100379	journal	March 2020
Superspreading and the effect of individual variation on disease emergence Lloyd-Smith, J. O.; Schreiber, S. J.; Kopp, P. E. Nature, Vol. 438, Issue 7066 https://doi.org/10.1038/nature04153	journal	November 2005
Global trends in emerging infectious diseases Jones, Kate E.; Patel, Nikkita G.; Levy, Marc A. Nature, Vol. 451, Issue 7181 https://doi.org/10.1038/nature06536	journal	February 2008
A universal model for mobility and migration patterns Simini, Filippo; González, Marta C.; Maritan, Amos Nature, Vol. 484, Issue 7392 https://doi.org/10.1038/nature10856	journal	February 2012
Ensemble forecast of human West Nile virus cases and mosquito infection rates DeFelice, Nicholas B.; Little, Eliza; Campbell, Scott R. Nature Communications, Vol. 8, Issue 1 https://doi.org/10.1038/ncomms14592	journal	February 2017
Model-based projections of Zika virus infections in childbearing women in the Americas Perkins, T. Alex; Siraj, Amir S.; Ruktanonchai, Corrine W. Nature Microbiology, Vol. 1, Issue 9 https://doi.org/10.1038/nmicrobiol.2016.126	journal	July 2016
Counteracting structural errors in ensemble forecast of influenza outbreaks Pei, Sen; Shaman, Jeffrey Nature Communications, Vol. 8, Issue 1 https://doi.org/10.1038/s41467-017-01033-1	journal	October 2017
Inter-annual variation in seasonal dengue epidemics driven by multiple interacting factors in Guangzhou, China Oidtman, Rachel J.; Lai, Shengjie; Huang, Zhoujie Nature Communications, Vol. 10, Issue 1 https://doi.org/10.1038/s41467-019-09035-x	journal	March 2019
Technology to advance infectious disease forecasting for outbreak management George, Dylan B.; Taylor, Wendy; Shaman, Jeffrey Nature Communications, Vol. 10, Issue 1 https://doi.org/10.1038/s41467-019-11901-7	journal	September 2019
A new twenty-first century science for effective epidemic response Bedford, Juliet; Farrar, Jeremy; Ihekweazu, Chikwe Nature, Vol. 575, Issue 7781 https://doi.org/10.1038/s41586-019-1717-y	journal	November 2019
Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016 McGowan, Craig J.; Biggerstaff , Matthew; Johansson, Michael Scientific Reports, Vol. 9, Issue 1 https://doi.org/10.1038/s41598-018-36361-9	journal	January 2019
Utilizing general human movement models to predict the spread of emerging infectious diseases in resource poor settings Kraemer, M. U. G.; Golding, N.; Bisanzio, D. Scientific Reports, Vol. 9, Issue 1 https://doi.org/10.1038/s41598-019-41192-3	journal	March 2019
Spatiotemporal incidence of Zika and associated environmental drivers for the 2015-2016 epidemic in Colombia Siraj, Amir S.; Rodriguez-Barraquer, Isabel; Barker, Christopher M. Scientific Data, Vol. 5, Issue 1 https://doi.org/10.1038/sdata.2018.73	journal	April 2018
Transmission of Nipah Virus — 14 Years of Investigations in Bangladesh Nikolay, Birgit; Salje, Henrik; Hossain, M. Jahangir New England Journal of Medicine, Vol. 380, Issue 19 https://doi.org/10.1056/NEJMoa1805376	journal	May 2019
Geography and macroeconomics: New data and new findings Nordhaus, W. D. Proceedings of the National Academy of Sciences, Vol. 103, Issue 10 https://doi.org/10.1073/pnas.0509842103	journal	February 2006
Impact of human mobility on the emergence of dengue epidemics in Pakistan Wesolowski, Amy; Qureshi, Taimur; Boni, Maciej F. Proceedings of the National Academy of Sciences, Vol. 112, Issue 38 https://doi.org/10.1073/pnas.1504964112	journal	September 2015
Essential information: Uncertainty and optimal control of Ebola outbreaks Li, Shou-Li; Bjørnstad, Ottar N.; Ferrari, Matthew J. Proceedings of the National Academy of Sciences, Vol. 114, Issue 22 https://doi.org/10.1073/pnas.1617482114	journal	May 2017
Forecasting the spatial transmission of influenza in the United States Pei, Sen; Kandula, Sasikiran; Yang, Wan Proceedings of the National Academy of Sciences, Vol. 115, Issue 11 https://doi.org/10.1073/pnas.1708856115	journal	February 2018
A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States Reich, Nicholas G.; Brooks, Logan C.; Fox, Spencer J. Proceedings of the National Academy of Sciences, Vol. 116, Issue 8 https://doi.org/10.1073/pnas.1812594116	journal	January 2019
An open challenge to advance probabilistic forecasting for dengue epidemics Johansson, Michael A.; Apfeldorf, Karyn M.; Dobson, Scott Proceedings of the National Academy of Sciences, Vol. 116, Issue 48 https://doi.org/10.1073/pnas.1909865116	journal	November 2019
Prevalent Eurasian avian-like H1N1 swine influenza virus with 2009 pandemic viral genes facilitating human infection Sun, Honglei; Xiao, Yihong; Liu, Jiyu Proceedings of the National Academy of Sciences, Vol. 117, Issue 29 https://doi.org/10.1073/pnas.1921186117	journal	June 2020
Estimating unobserved SARS-CoV-2 infections in the United States Perkins, T. Alex; Cavany, Sean M.; Moore, Sean M. Proceedings of the National Academy of Sciences, Vol. 117, Issue 36 https://doi.org/10.1073/pnas.2005476117	journal	August 2020
Measles Metapopulation Dynamics: A Gravity Model for Epidemiological Coupling and Dynamics Xia, Yingcun; Bjørnstad, Ottar N.; Grenfell, Bryan T. The American Naturalist, Vol. 164, Issue 2 https://doi.org/10.1086/422341	journal	August 2004
Global rise in human infectious disease outbreaks Smith, Katherine F.; Goldberg, Michael; Rosenthal, Samantha Journal of The Royal Society Interface, Vol. 11, Issue 101 https://doi.org/10.1098/rsif.2014.0950	journal	December 2014
Superensemble forecasts of dengue outbreaks Yamana, Teresa K.; Kandula, Sasikiran; Shaman, Jeffrey Journal of The Royal Society Interface, Vol. 13, Issue 123 https://doi.org/10.1098/rsif.2016.0410	journal	October 2016
A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking Arulampalam, M. S.; Maskell, S.; Gordon, N. IEEE Transactions on Signal Processing, Vol. 50, Issue 2 https://doi.org/10.1109/78.978374	journal	January 2002
Probabilistic forecasts, calibration and sharpness Gneiting, Tilmann; Balabdaoui, Fadoua; Raftery, Adrian E. Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 69, Issue 2 https://doi.org/10.1111/j.1467-9868.2007.00587.x	journal	April 2007
Opportunities and challenges in modeling emerging infectious diseases Metcalf, C. Jessica E.; Lessler, Justin Science, Vol. 357, Issue 6347 https://doi.org/10.1126/science.aam8335	journal	July 2017
The effect of human mobility and control measures on the COVID-19 epidemic in China Kraemer, Moritz U. G.; Yang, Chia-Hung; Gutierrez, Bernardo Science, Vol. 368, Issue 6490 https://doi.org/10.1126/science.abb4218	journal	March 2020
Aggregated mobility data could help fight COVID-19 Buckee, Caroline O.; Balsari, Satchit; Chan, Jennifer Science, Vol. 368, Issue 6487 https://doi.org/10.1126/science.abb8021	journal	April 2020
Harnessing multiple models for outbreak management Shea, Katriona; Runge, Michael C.; Pannell, David Science, Vol. 368, Issue 6491 https://doi.org/10.1126/science.abb9934	journal	May 2020
Susceptible supply limits the role of climate in the early SARS-CoV-2 pandemic Baker, Rachel E.; Yang, Wenchang; Vecchi, Gabriel A. Science, Vol. 369, Issue 6501 https://doi.org/10.1126/science.abc2535	journal	July 2020
Summary results of the 2014-2015 DARPA Chikungunya challenge Del Valle, Sara Y.; McMahon, Benjamin H.; Asher, Jason BMC Infectious Diseases, Vol. 18, Issue 1 https://doi.org/10.1186/s12879-018-3124-7	journal	May 2018
Genomic epidemiology supports multiple introductions and cryptic transmission of Zika virus in Colombia Black, Allison; Moncla, Louise H.; Laiton-Donato, Katherine BMC Infectious Diseases, Vol. 19, Issue 1 https://doi.org/10.1186/s12879-019-4566-2	journal	November 2019
Local and regional dynamics of chikungunya virus transmission in Colombia: the role of mismatched spatial heterogeneity Moore, Sean M.; ten Bosch, Quirine A.; Siraj, Amir S. BMC Medicine, Vol. 16, Issue 1 https://doi.org/10.1186/s12916-018-1127-2	journal	August 2018
Phenomenological forecasting of disease incidence using heteroskedastic Gaussian processes: A dengue case study Johnson, Leah R.; Gramacy, Robert B.; Cohen, Jeremy The Annals of Applied Statistics, Vol. 12, Issue 1 https://doi.org/10.1214/17-AOAS1090	journal	March 2018
Comparison of Filtering Methods for the Modeling and Retrospective Forecasting of Influenza Epidemics Yang, Wan; Karspeck, Alicia; Shaman, Jeffrey PLoS Computational Biology, Vol. 10, Issue 4 https://doi.org/10.1371/journal.pcbi.1003583	journal	April 2014
A Bayesian Ensemble Approach for Epidemiological Projections Lindström, Tom; Tildesley, Michael; Webb, Colleen PLOS Computational Biology, Vol. 11, Issue 4 https://doi.org/10.1371/journal.pcbi.1004187	journal	April 2015
Nonmechanistic forecasts of seasonal influenza with iterative one-week-ahead distributions Brooks, Logan C.; Farrow, David C.; Hyun, Sangwon PLOS Computational Biology, Vol. 14, Issue 6 https://doi.org/10.1371/journal.pcbi.1006134	journal	June 2018
Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S. Reich, Nicholas G.; McGowan, Craig J.; Yamana, Teresa K. PLOS Computational Biology, Vol. 15, Issue 11 https://doi.org/10.1371/journal.pcbi.1007486	journal	November 2019
Time varying methods to infer extremes in dengue transmission dynamics Lim, Jue Tao; Han, Yiting; Sue Lee Dickens, Borame PLOS Computational Biology, Vol. 16, Issue 10 https://doi.org/10.1371/journal.pcbi.1008279	journal	October 2020
Using Modelling to Disentangle the Relative Contributions of Zoonotic and Anthroponotic Transmission: The Case of Lassa Fever Lo Iacono, Giovanni; Cunningham, Andrew A.; Fichet-Calvet, Elisabeth PLoS Neglected Tropical Diseases, Vol. 9, Issue 1 https://doi.org/10.1371/journal.pntd.0003398	journal	January 2015
Detecting the impact of temperature on transmission of Zika, dengue, and chikungunya using mechanistic models Mordecai, Erin A.; Cohen, Jeremy M.; Evans, Michelle V. PLOS Neglected Tropical Diseases, Vol. 11, Issue 4 https://doi.org/10.1371/journal.pntd.0005568	journal	April 2017
Temperature modulates dengue virus epidemic growth rates through its effects on reproduction numbers and generation intervals Siraj, Amir S.; Oidtman, Rachel J.; Huber, John H. PLOS Neglected Tropical Diseases, Vol. 11, Issue 7 https://doi.org/10.1371/journal.pntd.0005797	journal	July 2017
Seasonal and interannual risks of dengue introduction from South-East Asia into China, 2005-2015 Lai, Shengjie; Johansson, Michael A.; Yin, Wenwu PLOS Neglected Tropical Diseases, Vol. 12, Issue 11 https://doi.org/10.1371/journal.pntd.0006743	journal	November 2018
Heterogeneous Mobile Phone Ownership and Usage Patterns in Kenya Wesolowski, Amy; Eagle, Nathan; Noor, Abdisalan M. PLoS ONE, Vol. 7, Issue 4 https://doi.org/10.1371/journal.pone.0035319	journal	April 2012
Nowcasting the Spread of Chikungunya Virus in the Americas Johansson, Michael A.; Powers, Ann M.; Pesik, Nicki PLoS ONE, Vol. 9, Issue 8 https://doi.org/10.1371/journal.pone.0104915	journal	August 2014
The risk of sustained sexual transmission of Zika is underestimated Allard, Antoine; Althouse, Benjamin M.; Hébert-Dufresne, Laurent PLOS Pathogens, Vol. 13, Issue 9 https://doi.org/10.1371/journal.ppat.1006633	journal	September 2017
Local and regional spread of chikungunya fever in the Americas Cauchemez, S.; Ledrans, M.; Poletto, C. Eurosurveillance, Vol. 19, Issue 28 https://doi.org/10.2807/1560-7917.ES2014.19.28.20854	journal	July 2014
Quantifying Reporting Timeliness to Improve Outbreak Control Marinović, Axel Bonačić; Swaan, Corien; van Steenbergen, Jim Emerging Infectious Diseases, Vol. 21, Issue 2 https://doi.org/10.3201/eid2102.130504	journal	February 2015
Data from: Spatiotemporal incidence of Zika and associated environmental drivers for the 2015-2016 epidemic in Colombia Siraj, Amir S.; Rodriguez-Barraquer, I.; Barker, Christopher M. Dryad https://doi.org/10.5061/dryad.83nj1	dataset	January 2019
The global distribution of the arbovirus vectors Aedes aegypti and Ae. albopictus Kraemer, Moritz UG; Sinka, Marianne E.; Duda, Kirsten A. eLife, Vol. 4 https://doi.org/10.7554/eLife.08347	journal	June 2015
MERS-CoV spillover at the camel-human interface Dudas, Gytis; Carvalho, Luiz Max; Rambaut, Andrew eLife, Vol. 7 https://doi.org/10.7554/eLife.31257	journal	January 2018