2026-10-31

CRPS as Evaluation Metric in GIFT-eval

While going over the GIFT-eval leaderboard the other day, I perceived the usage of CRPS as evaluation metric as a laudable choice—it being a proper scoring function. But as I continued to browse the GIFT-eval repository, reading about their preferred usage of GluonTS for evaluating forecasts, keeping in mind that most of the foundation models return only quantile forecasts, it dawned on me: You can’t calculate the CRPS from quantiles!

I never before questioned how papers and benchmarks report the CRPS for quantile predictions. But the CRPS can only be calculated from samples or via closed-form equations for the Normal distribution. So how does GluonTS–and by extension GIFT-eval–do it?

Approximate is how. GluonTS doesn’t even call the metric CRPS, but MeanWeightedSumQuantileLoss, because that’s what it is. The metric is only renamed to CRPS in the GIFT-eval’s Hugging Face app. The notebooks in the GIFT-eval Github repository call the MeanWeightedSumQuantileLoss.

The paper Learning Quantile Functions Without Quantile Crossing for Distribution-free Time Series Forecasting (Park et al., 2021) defines the metric in appendix F.2 and calls out that they use it as an approximation of the CRPS.

The approximation of the CRPS is discussed in Fixing the Pitfalls of Probabilistic Time Series Forecasting Evaluation by Kernel Quadrature (Adachi et al., 2025). They specifically reference GluonTS in their abstract:

We found that popular CRPS estimators–specifically, the quantile-based estimator implemented in the widely used GluonTS library and the probability-weighted moment approximation–both exhibit inherent estimation biases.