A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave
Abstract
Disease modelling has had considerable policy impact during the ongoing COVID-19 pandemic, and it is increasingly acknowledged that combining multiple models can improve the reliability of outputs. Here we report insights from ten weeks of collaborative short-term forecasting of COVID-19 in Germany and Poland (12 October–19 December 2020). The study period covers the onset of the second wave in both countries, with tightening non-pharmaceutical interventions (NPIs) and subsequently a decay (Poland) or plateau and renewed increase (Germany) in reported cases. Thirteen independent teams provided probabilistic real-time forecasts of COVID-19 cases and deaths. These were reported for lead times of one to four weeks, with evaluation focused on one- and two-week horizons, which are less affected by changing NPIs. Heterogeneity between forecasts was considerable both in terms of point predictions and forecast spread. Ensemble forecasts showed good relative performance, in particular in terms of coverage, but did not clearly dominate single-model predictions. The study was preregistered and will be followed up in future phases of the pandemic.
- Authors:
-
;
;
;
;
;
;
;
;
;
;
;
;
;
;
;
more »
- Publication Date:
- Research Org.:
- Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
- Sponsoring Org.:
- USDOE Laboratory Directed Research and Development (LDRD) Program; Wellcome Trust; National Science Foundation (NSF)
- Contributing Org.:
- CovidAnalytics-DELPHI; epiforecasts-EpiExpert and epiforecasts-EpiNow2; FIAS FZK-Epi1Ger; German and Polish Forecast Hubb Coordination Team; ICM-agentModel; Imperial-ensemble2; ITWW-county repro; LANL-GrowthRate; LeipzigIMISE-SECIR; MIMUW-StochSEIR; MOCOS-agent1; SDSC ISG-TrendModel; UCLA-SuEIR; USC-SlkJalpha
- OSTI Identifier:
- 1832375
- Report Number(s):
- LA-UR-21-20048
Journal ID: ISSN 2041-1723
- Grant/Contract Number:
- 89233218CNA000001; 219415; NIHR200908; 210758/Z/18/Z; 2027007
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Nature Communications
- Additional Journal Information:
- Journal Volume: 12; Journal Issue: 1; Journal ID: ISSN 2041-1723
- Publisher:
- Nature Publishing Group
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 60 APPLIED LIFE SCIENCES; computational models; epidemiology; SARS-CoV-2; statistics
Citation Formats
Bracher, J., Wolffram, D., Deuschel, J., Görgen, K., Ketterer, J. L., Ullrich, A., Abbott, S., Barbarossa, M. V., Bertsimas, D., Bhatia, S., Bodych, M., Bosse, N. I., Burgard, J. P., Castro, L., Fairchild, G., Fuhrmann, J., Funk, S., Gogolewski, K., Gu, Q., Heyder, S., Hotz, T., Kheifetz, Y., Kirsten, H., Krueger, T., Krymova, E., Li, M. L., Meinke, J. H., Michaud, I. J., Niedzielewski, K., Ożański, T., Rakowski, F., Scholz, M., Soni, S., Srivastava, A., Zieliński, J., Zou, D., Gneiting, T., Schienle, M., Li, Michael Lingzhi, Bertsimas, Dimitris, Bouardi, Hamza Tazi, Lami, Omar Skali, Soni, Saksham, Abbott, Sam, Bosse, Nikos I., Funk, Sebastian, Barbarossa, Maria Vittoria, Fuhrmann, Jan, Meinke, Jan H., Bracher, Johannes, Deuschel, Jannik, Gneiting, Tilmann, Görgen, Konstantin, Ketterer, Jakob, Schienle, Melanie, Ullrich, Alexander, Wolffram, Daniel, Górski, Łukasz, Gruziel-Słomka, Magdalena, Kaczorek, Artur, Moszyński, Antoni, Niedzielewski, Karol, Nowosielski, Jedrzej, Radwan, Maciej, Rakowski, Franciszek, Semeniuk, Marcin, Zieliński, Jakub, Bartczuk, Rafał, Kisielewski, Jan, Bhatia, Sangeeta, Biecek, Przemyslaw, Bezborodov, Viktor, Bodych, Marcin, Krueger, Tyll, Burgard, Jan Pablo, Heyder, Stefan, Hotz, Thomas, Osthus, Dave A., Michaud, Isaac J., Castro, Lauren, Fairchild, Geoffrey, Kheifetz, Yuri, Kirsten, Holger, Scholz, Markus, Gambin, Anna, Gogolewski, Krzysztof, Miasojedow, Błażej, Szczurek, Ewa, Rabczenko, Daniel, Rosińska, Magdalena, Bawiec, Marek, Bodych, Marcin, Ożański, Tomasz, Pabjan, Barbara, Rafajłlowicz, Ewaryst, Skubalska-Rafajłowicz, Ewa, Rafajłowicz, Wojciech, Migalska, Agata, Szczurek, Ewa, Flahault, Antoine, Manetti, Elisa, Choirat, Christine, Haro, Benjamin Bejar, Krymova, Ekaterina, Lee, Gavin, Obozinski, Guillaume, Sun, Tao, Thanou, Dorina, Gu, Quanquan, Xu, Pan, Chen, Jinghui, Wang, Lingxiao, Zou, Difan, Zhang, Weitong, Srivastava, Ajitesh, Prasanna, Viktor K., and Xu, Frost Tianjian. A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave. United States: N. p., 2021.
Web. doi:10.1038/s41467-021-25207-0.
Bracher, J., Wolffram, D., Deuschel, J., Görgen, K., Ketterer, J. L., Ullrich, A., Abbott, S., Barbarossa, M. V., Bertsimas, D., Bhatia, S., Bodych, M., Bosse, N. I., Burgard, J. P., Castro, L., Fairchild, G., Fuhrmann, J., Funk, S., Gogolewski, K., Gu, Q., Heyder, S., Hotz, T., Kheifetz, Y., Kirsten, H., Krueger, T., Krymova, E., Li, M. L., Meinke, J. H., Michaud, I. J., Niedzielewski, K., Ożański, T., Rakowski, F., Scholz, M., Soni, S., Srivastava, A., Zieliński, J., Zou, D., Gneiting, T., Schienle, M., Li, Michael Lingzhi, Bertsimas, Dimitris, Bouardi, Hamza Tazi, Lami, Omar Skali, Soni, Saksham, Abbott, Sam, Bosse, Nikos I., Funk, Sebastian, Barbarossa, Maria Vittoria, Fuhrmann, Jan, Meinke, Jan H., Bracher, Johannes, Deuschel, Jannik, Gneiting, Tilmann, Görgen, Konstantin, Ketterer, Jakob, Schienle, Melanie, Ullrich, Alexander, Wolffram, Daniel, Górski, Łukasz, Gruziel-Słomka, Magdalena, Kaczorek, Artur, Moszyński, Antoni, Niedzielewski, Karol, Nowosielski, Jedrzej, Radwan, Maciej, Rakowski, Franciszek, Semeniuk, Marcin, Zieliński, Jakub, Bartczuk, Rafał, Kisielewski, Jan, Bhatia, Sangeeta, Biecek, Przemyslaw, Bezborodov, Viktor, Bodych, Marcin, Krueger, Tyll, Burgard, Jan Pablo, Heyder, Stefan, Hotz, Thomas, Osthus, Dave A., Michaud, Isaac J., Castro, Lauren, Fairchild, Geoffrey, Kheifetz, Yuri, Kirsten, Holger, Scholz, Markus, Gambin, Anna, Gogolewski, Krzysztof, Miasojedow, Błażej, Szczurek, Ewa, Rabczenko, Daniel, Rosińska, Magdalena, Bawiec, Marek, Bodych, Marcin, Ożański, Tomasz, Pabjan, Barbara, Rafajłlowicz, Ewaryst, Skubalska-Rafajłowicz, Ewa, Rafajłowicz, Wojciech, Migalska, Agata, Szczurek, Ewa, Flahault, Antoine, Manetti, Elisa, Choirat, Christine, Haro, Benjamin Bejar, Krymova, Ekaterina, Lee, Gavin, Obozinski, Guillaume, Sun, Tao, Thanou, Dorina, Gu, Quanquan, Xu, Pan, Chen, Jinghui, Wang, Lingxiao, Zou, Difan, Zhang, Weitong, Srivastava, Ajitesh, Prasanna, Viktor K., & Xu, Frost Tianjian. A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave. United States. https://doi.org/10.1038/s41467-021-25207-0
Bracher, J., Wolffram, D., Deuschel, J., Görgen, K., Ketterer, J. L., Ullrich, A., Abbott, S., Barbarossa, M. V., Bertsimas, D., Bhatia, S., Bodych, M., Bosse, N. I., Burgard, J. P., Castro, L., Fairchild, G., Fuhrmann, J., Funk, S., Gogolewski, K., Gu, Q., Heyder, S., Hotz, T., Kheifetz, Y., Kirsten, H., Krueger, T., Krymova, E., Li, M. L., Meinke, J. H., Michaud, I. J., Niedzielewski, K., Ożański, T., Rakowski, F., Scholz, M., Soni, S., Srivastava, A., Zieliński, J., Zou, D., Gneiting, T., Schienle, M., Li, Michael Lingzhi, Bertsimas, Dimitris, Bouardi, Hamza Tazi, Lami, Omar Skali, Soni, Saksham, Abbott, Sam, Bosse, Nikos I., Funk, Sebastian, Barbarossa, Maria Vittoria, Fuhrmann, Jan, Meinke, Jan H., Bracher, Johannes, Deuschel, Jannik, Gneiting, Tilmann, Görgen, Konstantin, Ketterer, Jakob, Schienle, Melanie, Ullrich, Alexander, Wolffram, Daniel, Górski, Łukasz, Gruziel-Słomka, Magdalena, Kaczorek, Artur, Moszyński, Antoni, Niedzielewski, Karol, Nowosielski, Jedrzej, Radwan, Maciej, Rakowski, Franciszek, Semeniuk, Marcin, Zieliński, Jakub, Bartczuk, Rafał, Kisielewski, Jan, Bhatia, Sangeeta, Biecek, Przemyslaw, Bezborodov, Viktor, Bodych, Marcin, Krueger, Tyll, Burgard, Jan Pablo, Heyder, Stefan, Hotz, Thomas, Osthus, Dave A., Michaud, Isaac J., Castro, Lauren, Fairchild, Geoffrey, Kheifetz, Yuri, Kirsten, Holger, Scholz, Markus, Gambin, Anna, Gogolewski, Krzysztof, Miasojedow, Błażej, Szczurek, Ewa, Rabczenko, Daniel, Rosińska, Magdalena, Bawiec, Marek, Bodych, Marcin, Ożański, Tomasz, Pabjan, Barbara, Rafajłlowicz, Ewaryst, Skubalska-Rafajłowicz, Ewa, Rafajłowicz, Wojciech, Migalska, Agata, Szczurek, Ewa, Flahault, Antoine, Manetti, Elisa, Choirat, Christine, Haro, Benjamin Bejar, Krymova, Ekaterina, Lee, Gavin, Obozinski, Guillaume, Sun, Tao, Thanou, Dorina, Gu, Quanquan, Xu, Pan, Chen, Jinghui, Wang, Lingxiao, Zou, Difan, Zhang, Weitong, Srivastava, Ajitesh, Prasanna, Viktor K., and Xu, Frost Tianjian. Fri .
"A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave". United States. https://doi.org/10.1038/s41467-021-25207-0. https://www.osti.gov/servlets/purl/1832375.
@article{osti_1832375,
title = {A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave},
author = {Bracher, J. and Wolffram, D. and Deuschel, J. and Görgen, K. and Ketterer, J. L. and Ullrich, A. and Abbott, S. and Barbarossa, M. V. and Bertsimas, D. and Bhatia, S. and Bodych, M. and Bosse, N. I. and Burgard, J. P. and Castro, L. and Fairchild, G. and Fuhrmann, J. and Funk, S. and Gogolewski, K. and Gu, Q. and Heyder, S. and Hotz, T. and Kheifetz, Y. and Kirsten, H. and Krueger, T. and Krymova, E. and Li, M. L. and Meinke, J. H. and Michaud, I. J. and Niedzielewski, K. and Ożański, T. and Rakowski, F. and Scholz, M. and Soni, S. and Srivastava, A. and Zieliński, J. and Zou, D. and Gneiting, T. and Schienle, M. and Li, Michael Lingzhi and Bertsimas, Dimitris and Bouardi, Hamza Tazi and Lami, Omar Skali and Soni, Saksham and Abbott, Sam and Bosse, Nikos I. and Funk, Sebastian and Barbarossa, Maria Vittoria and Fuhrmann, Jan and Meinke, Jan H. and Bracher, Johannes and Deuschel, Jannik and Gneiting, Tilmann and Görgen, Konstantin and Ketterer, Jakob and Schienle, Melanie and Ullrich, Alexander and Wolffram, Daniel and Górski, Łukasz and Gruziel-Słomka, Magdalena and Kaczorek, Artur and Moszyński, Antoni and Niedzielewski, Karol and Nowosielski, Jedrzej and Radwan, Maciej and Rakowski, Franciszek and Semeniuk, Marcin and Zieliński, Jakub and Bartczuk, Rafał and Kisielewski, Jan and Bhatia, Sangeeta and Biecek, Przemyslaw and Bezborodov, Viktor and Bodych, Marcin and Krueger, Tyll and Burgard, Jan Pablo and Heyder, Stefan and Hotz, Thomas and Osthus, Dave A. and Michaud, Isaac J. and Castro, Lauren and Fairchild, Geoffrey and Kheifetz, Yuri and Kirsten, Holger and Scholz, Markus and Gambin, Anna and Gogolewski, Krzysztof and Miasojedow, Błażej and Szczurek, Ewa and Rabczenko, Daniel and Rosińska, Magdalena and Bawiec, Marek and Bodych, Marcin and Ożański, Tomasz and Pabjan, Barbara and Rafajłlowicz, Ewaryst and Skubalska-Rafajłowicz, Ewa and Rafajłowicz, Wojciech and Migalska, Agata and Szczurek, Ewa and Flahault, Antoine and Manetti, Elisa and Choirat, Christine and Haro, Benjamin Bejar and Krymova, Ekaterina and Lee, Gavin and Obozinski, Guillaume and Sun, Tao and Thanou, Dorina and Gu, Quanquan and Xu, Pan and Chen, Jinghui and Wang, Lingxiao and Zou, Difan and Zhang, Weitong and Srivastava, Ajitesh and Prasanna, Viktor K. and Xu, Frost Tianjian},
abstractNote = {Disease modelling has had considerable policy impact during the ongoing COVID-19 pandemic, and it is increasingly acknowledged that combining multiple models can improve the reliability of outputs. Here we report insights from ten weeks of collaborative short-term forecasting of COVID-19 in Germany and Poland (12 October–19 December 2020). The study period covers the onset of the second wave in both countries, with tightening non-pharmaceutical interventions (NPIs) and subsequently a decay (Poland) or plateau and renewed increase (Germany) in reported cases. Thirteen independent teams provided probabilistic real-time forecasts of COVID-19 cases and deaths. These were reported for lead times of one to four weeks, with evaluation focused on one- and two-week horizons, which are less affected by changing NPIs. Heterogeneity between forecasts was considerable both in terms of point predictions and forecast spread. Ensemble forecasts showed good relative performance, in particular in terms of coverage, but did not clearly dominate single-model predictions. The study was preregistered and will be followed up in future phases of the pandemic.},
doi = {10.1038/s41467-021-25207-0},
journal = {Nature Communications},
number = 1,
volume = 12,
place = {United States},
year = {2021},
month = {8}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Summary results of the 2014-2015 DARPA Chikungunya challenge
journal, May 2018
- Del Valle, Sara Y.; McMahon, Benjamin H.; Asher, Jason
- BMC Infectious Diseases, Vol. 18, Issue 1
The RAPIDD ebola forecasting challenge: Synthesis and lessons learnt
journal, March 2018
- Viboud, Cécile; Sun, Kaiyuan; Gaffey, Robert
- Epidemics, Vol. 22
Automatic Time Series Forecasting: The forecast Package for R
journal, January 2008
- Hyndman, Rob J.; Khandakar, Yeasmin
- Journal of Statistical Software, Vol. 27, Issue 3
Real-time Epidemic Forecasting: Challenges and Opportunities
journal, August 2019
- Desai, Angel N.; Kraemer, Moritz U. G.; Bhatia, Sangeeta
- Health Security, Vol. 17, Issue 4
Accuracy of real-time multi-model ensemble forecasts for seasonal influenza in the U.S.
journal, November 2019
- Reich, Nicholas G.; McGowan, Craig J.; Yamana, Teresa K.
- PLOS Computational Biology, Vol. 15, Issue 11
An open challenge to advance probabilistic forecasting for dengue epidemics
journal, November 2019
- Johansson, Michael A.; Apfeldorf, Karyn M.; Dobson, Scott
- Proceedings of the National Academy of Sciences, Vol. 116, Issue 48
A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States
journal, January 2019
- Reich, Nicholas G.; Brooks, Logan C.; Fox, Spencer J.
- Proceedings of the National Academy of Sciences, Vol. 116, Issue 8
Nowcasting the COVID‐19 pandemic in Bavaria
journal, December 2020
- Günther, Felix; Bender, Andreas; Katz, Katharina
- Biometrical Journal, Vol. 63, Issue 3
Modeling the spread of COVID-19 in Germany: Early assessment and possible scenarios
journal, September 2020
- Barbarossa, Maria Vittoria; Fuhrmann, Jan; Meinke, Jan H.
- PLOS ONE, Vol. 15, Issue 9
Evaluating epidemic forecasts in an interval format
journal, February 2021
- Bracher, Johannes; Ray, Evan L.; Gneiting, Tilmann
- PLOS Computational Biology, Vol. 17, Issue 2
An interactive web-based dashboard to track COVID-19 in real time
journal, May 2020
- Dong, Ensheng; Du, Hongru; Gardner, Lauren
- The Lancet Infectious Diseases, Vol. 20, Issue 5
Sequential aggregation of probabilistic forecasts—Application to wind speed ensemble forecasts
journal, November 2020
- Zamo, Michaël; Bel, Liliane; Mestre, Olivier
- Journal of the Royal Statistical Society: Series C (Applied Statistics), Vol. 70, Issue 1
Assessing the performance of real-time epidemic forecasts: A case study of Ebola in the Western Area region of Sierra Leone, 2014-15
journal, February 2019
- Funk, Sebastian; Camacho, Anton; Kucharski, Adam J.
- PLOS Computational Biology, Vol. 15, Issue 2
Strictly Proper Scoring Rules, Prediction, and Estimation
journal, March 2007
- Gneiting, Tilmann; Raftery, Adrian E.
- Journal of the American Statistical Association, Vol. 102, Issue 477
Forecasting the novel coronavirus COVID-19
journal, March 2020
- Petropoulos, Fotios; Makridakis, Spyros
- PLOS ONE, Vol. 15, Issue 3
The spread of awareness and its impact on epidemic outbreaks
journal, March 2009
- Funk, S.; Gilad, E.; Watkins, C.
- Proceedings of the National Academy of Sciences, Vol. 106, Issue 16
Influenza epidemic spread simulation for Poland — a large scale, individual based model study
journal, August 2010
- Rakowski, Franciszek; Gruziel, Magdalena; Bieniasz-Krzywiec, Łukasz
- Physica A: Statistical Mechanics and its Applications, Vol. 389, Issue 16
Probabilistic forecasting in infectious disease epidemiology: the 13th Armitage lecture: L. HELD, S. MEYER AND J. BRACHER
journal, June 2017
- Held, Leonhard; Meyer, Sebastian; Bracher, Johannes
- Statistics in Medicine, Vol. 36, Issue 22
Epidemic Forecasting is Messier Than Weather Forecasting: The Role of Human Behavior and Internet Data Streams in Epidemic Forecast
journal, November 2016
- Moran, Kelly R.; Fairchild, Geoffrey; Generous, Nicholas
- Journal of Infectious Diseases, Vol. 214, Issue suppl 4
Ensemble forecast modeling for the design of COVID-19 vaccine efficacy trials
journal, October 2020
- Dean, Natalie E.; Pastore y. Piontti, Ana; Madewell, Zachary J.
- Vaccine, Vol. 38, Issue 46
Collaborative efforts to forecast seasonal influenza in the United States, 2015–2016
journal, January 2019
- McGowan, Craig J.; Biggerstaff , Matthew; Johansson, Michael
- Scientific Reports, Vol. 9, Issue 1
Combining probabilistic forecasts of COVID-19 mortality in the United States
journal, June 2021
- Taylor, James W.; Taylor, Kathryn S.
- European Journal of Operational Research
Is It Better to Average Probabilities or Quantiles?
journal, July 2013
- Lichtendahl, Kenneth C.; Grushka-Cockayne, Yael; Winkler, Robert L.
- Management Science, Vol. 59, Issue 7
The future of influenza forecasts
journal, February 2019
- Viboud, Cécile; Vespignani, Alessandro
- Proceedings of the National Academy of Sciences, Vol. 116, Issue 8
ATMOSPHERIC SCIENCE: Weather Forecasting with Ensemble Methods
journal, October 2005
- Gneiting, T.
- Science, Vol. 310, Issue 5746
Mechanistic models versus machine learning, a fight worth fighting for the biological community?
journal, May 2018
- Baker, Ruth E.; Peña, Jose-Maria; Jayamohan, Jayaratnam
- Biology Letters, Vol. 14, Issue 5