let's do it like they did in 1924 and draw forecasts by hand
Evergreen / Updated on: 2026-11-15

Variogram Score

The variogram score keeps coming up in the context of evaluating the joint distribution forecast over the forecast horizon. Yao et al. (2026) use it to evaluate sample paths generated by PathFM. Hirsch (2025) uses it to evaluate distributional electricity price forecasts.

While the continuous ranked probability score (CRPS) can be used as strictly proper scoring rule for marginal distribution forecasts, the variogram score can be used as (not strictly) proper scoring rule for jointly distributed quantities such as a time series’ realizations at multiple future points in time.

Scheuerer and Hamill (2015) introduce it in “Variogram-Based Proper Scoring Rules for Probabilistic Forecasts of Multivariate Quantities” as alternative to the energy score (the multivariate extension of the CRPS):

The basic idea of the class of multivariate scoring rules proposed in the following is to consider pairwise differences of the components of the multivariate quantity of interest.

Following equation 3 in section 2 of Scheuerer and Hamill (2015), and equation 7 in supplement 1 of Hirsch (2025), the variogram score of order $p$ for time series $k$ is defined as

$$ VS_k^p = \sum_{i=1}^H \sum_{j=1}^H w_{i,j} \left( | y_{k,i} - y_{k,j} |^p - \frac{1}{S} \sum_{s=1}^S | F_{k,s,i} - F_{k,s,j}|^p \right)^2 $$

where $y_{k,i}$ and $y_{k,j}$ are the realized observations at step $i$ and $j$ of the $H$ steps of the forecast horizon, and $F$ is the forecast of $y$, represented by $S$ samples from the predicted forecast distribution. The left component of the variogram score is the absolute difference between each pair of observations, while the right component is the mean of the forecasted pairwise differences. There will be a total of $H^2$ pairwise differences for both observations and forecasts. The variogram score takes the difference of realized and forecasted pairwise difference, and sums them up, weighing each with a weight $w_{i,j}$ to optionally emphasize differences by, for example, how close their observations are in time. Typical values for $p$ are 0.5, 1, maybe 2.

The idea of comparing pairwise differences is at the core of the variogram score. As Scheuerer and Hamill (2015) show in equation 2 of section 2, by evaluating pairwise differences, the variogram evaluates distributions not only in terms of their marginal mean and variance, but also in terms of their joint correlation.

The pairwise differences and the resulting sum over $H^2$ elements are, however, also at the core of either memory hungry or slow implementations of the variogram score.

A memory-hungry implementation that expands arrays naively to an additional dimension of size $H$ to calculate pairwise differences can look as follows:

import numpy as np

def variogram(
	y_true: np.ndarray, # (N, H)
	y_pred: np.ndarray, # (N, S, H)
	w: np.ndarray | None = None, # (H, H)
	p: float = 1.0
):
	H = y_true.shape[1]
	if w is None:
		w = np.ones((H, H))
	w = w[None, :, :] # 1, H, H for broadcasting across items

	item_true_abs_pairwise_diff = np.abs( # N, H, H
		y_true[:, None, :] - y_true[:, :, None]
	) ** p

	item_sample_predicted_pairwise_diff = ( # (N, S, H, H)
		y_pred[:, :, None, :] - y_pred[:, :, :, None]
	)

	item_predicted_mean_abs_pairwise_diff = np.mean( # (N, H, H)
		np.abs(item_sample_predicted_pairwise_diff) ** p,
		axis=1
	)

	item_variogram_score = np.sum( # (N,)
		w * ((
			item_true_abs_pairwise_diff -
				item_predicted_mean_abs_pairwise_diff
		) ** 2),
		axis = (1, 2) # Sum over H**2 elements
	)

	# Aggregate as suggested in Hirsch (2025), supplement 1.
	mean_variogram_score = np.mean(item_variogram_score) / H / H # (1,)

	return mean_variogram_score

Draw some data from a multivariate normal with a simple autocorrelation structure to test this implementation. Simulate a 24-observation forecast horizon for five time series. The forecast distribution is approximated by 10 samples. The number of time series and samples is chosen to be small on purpose due to the memory consumption of the above implementation.

horizon = 24
n_items = 5
n_samples = 10

rng = np.random.default_rng(seed=256)

def draw_multivariate_normal(size, corr):
	return rng.multivariate_normal(
		mean=np.zeros((horizon)),
		cov=np.fromfunction(
			lambda i, j: corr ** abs(i-j),
			(horizon, horizon)
		),
		size=size,
	)

y_true = draw_multivariate_normal(size=(n_items), corr=0.7)
y_pred = draw_multivariate_normal(size=(n_items, n_samples), corr=0.7)

print(f"{y_true.shape=}")
print(f"{y_pred.shape=}")
y_true.shape=(5, 24)
y_pred.shape=(5, 10, 24)

The variogram score across the five time series is:

variogram(y_true=y_true, y_pred=y_pred)
np.float64(0.6931243499882981)

Validate this result using the Scoringrules library as reference implementation:

import scoringrules as sr

sr_vs_a = sr.vs_ensemble( # Returns one score per item (N,)
	obs=y_true,
	fct=y_pred,
	w=np.ones((horizon, horizon)),
	p=1,
)

print(np.mean(sr_vs_a) / horizon / horizon)
0.6931243499882981

A memory-safe alternative implementation of the variogram score avoids expansion of multi-dimensional arrays and uses loops instead, exploiting the symmetry of the pairwise difference (H, H) array to iterate over the upper-triangle only. Using only Numpy, this isn’t fast, but it will finish even for larger input arrays—in contrast to the variogram() function implemented above that will consume all available memory and crash. The implementation here could be made faster using Numba, Jax, or similar.

def variogram_memory_safe(
	y_true: np.ndarray, # (N, H)
	y_pred: np.ndarray, # (N, S, H)
	w: np.ndarray | None = None, # (H, H)
	p: float = 1.0
):
	N, S, H = y_pred.shape

	if w is None:
		w = np.ones((H, H))

	score = 0.0

	for n in range(N):
		item_score = 0.0
		y_true_n = y_true[n, :]
		y_pred_n = y_pred[n, :, :]
		
		for i in range(H-1): # Only upper triangle, excl. diagonal
			row_score = 0.0
			y_true_ni = y_true_n[i]
			
			for j in range(i+1, H):
				w_ij = 1.0 if w is None else w[i,j]
				if w_ij == 0.0:
					continue

				true_diff_nij = np.abs(y_true_n[j] - y_true_ni) ** p

				# Optionally turn this into a loop over S
				pred_diff_nij = np.mean(np.abs(
					y_pred_n[:, i] - y_pred_n[:, j]
				) ** p)

				diff_of_diffs = true_diff_nij - pred_diff_nij
				row_score += w_ij * diff_of_diffs * diff_of_diffs

			item_score += row_score
		
		score += item_score

	return 2 * score / N / H / H

This implementation reproduces the variogram score calculated above for the small arrays:

variogram_memory_safe(y_true=y_true, y_pred=y_pred)
np.float64(0.6931243499882981)

And it will also return a result for larger input arrays such as:

y_true_large = draw_multivariate_normal(size=(5000), corr=0.7)
y_pred_large = draw_multivariate_normal(size=(5000, 2501), corr=0.7)

variogram_memory_safe(
	y_true=y_true_large,
	y_pred=y_pred_large
)
np.float64(0.5687425481981133)

The Scoringrules implementation using Numpy will crash on the second example, too. Scoringrules offers Numba, Jax, Torch alternatives.


Mentioned in: