CRPS
How it works
CRPS compares the predicted CDF F with the "perfect" CDF of the observation — the step function H(x − y) that jumps from 0 to 1 at the observation y. It integrates the squared difference of the two CDFs over the real line: CRPS(F, y) = ∫_ℝ (F(x) − H(x − y))² dx. The closer F is to the ideal step CDF (good calibration and high sharpness), the smaller the integral. In practice, when the forecast is an ensemble of samples {x_1,…,x_m} (e.g. weather ensemble members), the energy form is estimated from samples: CRPS ≈ (1/m) Σ_i |x_i − y| − (1/(2m²)) Σ_i Σ_j |x_i − x_j|. The first term penalizes the distance of samples from the observation (accuracy); the second rewards ensemble spread (sharpness). The score carries the units of the forecast variable and, for a deterministic forecast, reduces to |x − y| = MAE.
Problem solved
Point metrics (MAE, RMSE) score only a single estimate and ignore forecast uncertainty and the shape of the predicted distribution. CRPS provides a fair, single-number evaluation of probabilistic forecasts over a continuous domain — rewarding both calibration and sharpness — and as a strictly proper rule it incentivizes reporting the true predictive distribution.
Components
The full cumulative distribution function of the forecast — the uncertainty representation that CRPS evaluates in its entirety rather than only its mean.
The definition ∫(F(x) − H(x − y))² dx: the integral of the squared difference between the forecast CDF and the observation step function.
The equivalent form E|X − y| − ½ E|X − X′| estimated from ensemble samples; enables Monte Carlo computation of CRPS without an explicit CDF.
Implementation
The standard ensemble CRPS estimator is biased for a small number of members m and systematically underestimates spread.
CRPS carries the units of the forecast variable, so values are not comparable across variables of different scale without normalization.
Evolution
Matheson and Winkler introduce scoring rules for continuous probability distributions, defining CRPS.
Gneiting and Raftery place CRPS within the general theory of strictly proper scoring rules and its energy form.
GenCast (Google DeepMind) reports beating the operational ENS ensemble on CRPS for over 96% of verification targets.
Hyperparameters (configurable axes)
Choice of ensemble estimator: biased vs. unbiased/"fair" (finite-m correction), PWM form, or numerical CDF quadrature.
Number of samples/ensemble members used to estimate CRPS; small m inflates estimator bias unless a "fair" correction is applied.
Computational complexity
Time complexity: O(m log m). Space complexity: O(m).
Parallelism
CRPS computation is independent per verification target (time, location, variable), so it is embarrassingly parallel across verification targets.
Hardware requirements
CRPS is a lightweight scalar metric computed post-hoc on model outputs; it needs no specialized hardware.