Papers / Updated at: 2026-10-31
Adachi et al. (2025): Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature
Authors:
- Masaki Adachi
- Masahiro Fujisawa
- Michael A Osborne
Source: https://arxiv.org/abs/2503.06079
DOI: https://doi.org/10.48550/arXiv.2503.06079
Mentioned in:
- CRPS as Evaluation Metric in GIFT-eval
- Adachi et al. (2025): Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature
I came across this paper while looking into how GluonTS calculates CRPS as evaluation metric based on marginal quantile forecasts (despite CRPS requiring samples from the forecast distribution or closed-form equations for the Normal distribution). This in turn was motivated by GIFT-eval ranking models by CRPS.
GluonTS does not provide a CRPS metric, but only the MeanWeightedSumQuantileLoss. GIFT-eval renames it in its Hugging Face app to CRPS.
The paper discusses shortcomings of using approximations such as the MeanWeightedSumQuantileLoss to evaluate the CRPS.