ECE
How it works
1) For each prediction take the confidence (usually the maximum softmax probability) and whether the prediction was correct. 2) Split the confidence range [0,1] into M bins (e.g. 10–15), usually of equal width. 3) Assign each prediction to a bin by its confidence. 4) In each bin B_m compute the mean confidence conf(B_m) and the empirical accuracy acc(B_m). 5) Compute the bin's contribution as (|B_m|/n)·|acc(B_m) − conf(B_m)|. 6) Sum the contributions of all bins — the result is the ECE. 7) Optionally visualize via a reliability diagram and use adaptive binning when the confidence distribution is uneven.
Problem solved
It solves the problem of assessing the trustworthiness of a model's confidence: accuracy alone does not tell whether the stated probabilities are truthful. Modern networks tend to be overconfident; ECE quantifies this gap in a single number, enabling comparison of models and calibration methods.
Components
The prediction's stated probability, usually the maximum softmax value.
Division of the confidence range into M bins (equal-width or adaptive/equal-mass).
The absolute difference between empirical accuracy and mean confidence within each bin.
Averaging the gaps with weights equal to the share of samples per bin, yielding the single ECE number.
Evolution
Naeini, Cooper, and Hauskrecht formalize binned calibration assessment (Bayesian Binning), the foundation of ECE.
Guo et al. show the overconfidence of modern networks and popularize ECE and temperature scaling.
Work highlights ECE's sensitivity to binning and proposes adaptive/equal-mass variants and complementary measures.
Hyperparameters (configurable axes)
The number of confidence bins; too few underestimates ECE, too many increases variance with small samples.
Equal-width vs adaptive (equal-mass) — affects the estimate's stability under a skewed confidence distribution.
Computational complexity
Time complexity: O(n + M). Space complexity: O(M).
Parallelism
The metric is ideally suited to map-reduce-style computation over large datasets.
Hardware requirements
ECE is a simple statistical metric computed over predictions; it needs no specialized accelerator.