Evaluating probabilistic forecasts with scoringRules

28. June 2021

Comparing predictive performance

Forecasts are generally surrounded by uncertainty and being able to quantify this uncertainty is key to good decision making. Accordingly, probabilistic forecasts in the form of predictive probability distributions over future quantities or events have become popular over the last decades in various fields including meteorology, climate science, hydrology, seismology, economics, finance, demography, and political science.

In typical applications, many alternative statistical models and data sources can be used to produce probabilistic forecasts. Hence, evaluating and selecting among competing methods is an important task. The scoringRules package for R provides functionality for comparative evaluation of probabilistic models based on proper scoring rules, covering a wide range of situations in applied work.

The scoringRules package aims to be a comprehensive library for computing scoring rules. We offer implementations of several known (but not routinely applied) formulas and implement some closed-form expressions that were previously unavailable. Whenever more than one implementation variant exists, we offer statistically principled default choices. The package contains the continuous ranked probability score (CRPS) and the logarithmic score, as well as the multivariate energy score and variogram score. All of these scoring rules are proper, which means that forecasters have an incentive to state their true belief.

scoringRules was developed in the Computational Statistics group at HITS is available on CRAN, and developed on GitHub.

References:

Alexander Jordan, Fabian Krüger, Sebastian Lerch
Evaluating Probabilistic Forecasts with scoringRules
J. Stat. Softw. (2019) 90(12):1–37, DOI: 10.18637/jss.v090.i12

About HITS

HITS, the Heidelberg Institute for Theoretical Studies, was established in 2010 by physicist and SAP co-founder Klaus Tschira (1940-2015) and the Klaus Tschira Foundation as a private, non-profit research institute. HITS conducts basic research in the natural, mathematical, and computer sciences. Major research directions include complex simulations across scales, making sense of data, and enabling science via computational research. Application areas range from molecular biology to astrophysics. An essential characteristic of the Institute is interdisciplinarity, implemented in numerous cross-group and cross-disciplinary projects. The base funding of HITS is provided by the Klaus Tschira Foundation.

Switch to the German homepage or stay on this page