Soon after the beginning of the pandemic, researchers from the Heidelberg Institute for Theoretical Studies (HITS) and the Karlsruhe Institute of Technology (KIT) developed a web platform for bundling short-term forecasts of COVID-19 cases and deaths in Germany and Poland with contributions by numerous independent modelling teams. Now, these researchers have published their insights from a systematic evaluation study in the open access journal Nature Communications. The results: Combining different models leads to better predictions, but predictions regarding COVID-19 remain highly challenging.
Sometimes, you can obtain a clearer picture by combining different approaches in a collaborative way. Scientists from the German-Polish COVID-19 Forecast Hub took this approach in evaluating the strengths and weaknesses of forecasting models and assessed whether combined ensemble forecasts can improve upon single-model predictions.
The hub: A transparent and fair comparison
“We created a comparison platform for different models and fostered the exchange between modelling teams with the goal of improving individual models and the overall combined prediction,” states Melanie Schienle, professor of Statistics and Econometrics at KIT and currently a guest scientist at HITS, who leads the project team together with Tilmann Gneiting from the Computational Statistics group at HITS. The Forecast Hub follows an open science approach. It evaluates forecasts and preserves the entire history of the model outputs.“Models often look a bit better than they are when applied in retrospect, and we wanted to avoid that pitfall”, adds Johannes Bracher (HITS and KIT), the first author of the study. “We publicly specified the rules of the comparison in advance, similar to in pharmaceutical or medical trials.”
More than the sum of its parts: Ensemble forecasts perform better
The study period on which the team reported in Nature Communications covers the onset of the second wave in Germany and Poland from 12 October to 19 December 2020. This period is characterized by tightening non-pharmaceutical interventions, which subsequently led to a decrease in cases in Poland and a plateau and renewed increase in cases in Germany. Thirteen independent teams provided probabilistic real-time forecasts of COVID-19 cases and deaths for lead times of one to four weeks. Overall, ensemble forecasts showed good relative performance – especially in terms of uncertainty quantification – and were typically on par with the best single-model predictions. Moreover, their performance was less variable across different weeks.
“Thus, we can clearly see the benefit of this collaborative approach, which provides more stable performance,” says Johannes Bracher. “It´s like the famous quote by Aristotle: ‘The whole is more than the sum of its parts.’”
Forecasts: A matter of uncertainty
The evaluations also highlight the fact that COVID-19 remains highly challenging to predict, and ensemble prediction is also sometimes off target. “Overall, the predictions were very heterogeneous across models, so different models did not at all agree about what was going to happen next,” Melanie Schienle adds. “By considering the different forecasts together, we can gain a more realistic assessment of the degree of uncertainty.”
The German and Polish COVID-19 Forecast Hub was inspired by the US COVID-19 Forecast Hub, a US project in which Johannes Bracher and Tilmann Gneiting have been actively involved from its launch. Since March 2021, the German and Polish COVID-19 Forecast Hub has gradually been integrated into a new European platform – the European Covid-19 Forecast Hub – run by the European Centre for Disease Prevention and Control and the London School of Hygiene and Tropical Medicine.
The study has been published in Nature Communications and featured in Nature Communications’ Editors´ Highlights.
Bracher, J., Wolffram, D., Deuschel, J. et al. A pre-registered short-term forecasting study of COVID-19 in Germany and Poland during the second wave. Nat Commun 12, 5173 (2021). https://doi.org/10.1038/s41467-021-25207-0
Dr. Johannes Bracher
Chair of Statistics and Econometrics at KIT
Computational Statistics Group at HITS
johannes bracher@kit edu
Prof. Dr. Melanie Schienle
Chair of Statistics and Econometrics, department of Economics and Management, KIT
melanie schienle@kit edu
Dr. Peter Saueressig
Head of Communications
Heidelberg Institute for Theoretical Studies (HITS)
Strategic Corporate Development and Communications
Karlsruhe Institute of Technology (KIT)
HITS, the Heidelberg Institute for Theoretical Studies, was established in 2010 by physicist and SAP co-founder Klaus Tschira (1940-2015) and the Klaus Tschira Foundation as a private, non-profit research institute. HITS conducts basic research in the natural, mathematical, and computer sciences. Major research directions include complex simulations across scales, making sense of data, and enabling science via computational research. Application areas range from molecular biology to astrophysics. An essential characteristic of the Institute is interdisciplinarity, implemented in numerous cross-group and cross-disciplinary projects. The base funding of HITS is provided by the Klaus Tschira Foundation.