Robots Atlas>ROBOTS ATLAS
Evaluation

ECE

2015ActivePublished: 17 November 2026Updated: 17 November 2026Published
Key innovation
Reduces a model's calibration quality to a single number: the weighted-average gap between stated confidence and actual accuracy, computed over confidence bins.
Category
Evaluation
Abstraction level
Pojęcie
Operation level
Evaluation (runtime)Inference
Use cases
Assessing the calibration of classifiers and neural networksComparing calibration methods (temperature scaling, Platt scaling)Monitoring model trust in risk-sensitive applicationsReporting reliability alongside accuracy in benchmarks

How it works

1) For each prediction take the confidence (usually the maximum softmax probability) and whether the prediction was correct. 2) Split the confidence range [0,1] into M bins (e.g. 10–15), usually of equal width. 3) Assign each prediction to a bin by its confidence. 4) In each bin B_m compute the mean confidence conf(B_m) and the empirical accuracy acc(B_m). 5) Compute the bin's contribution as (|B_m|/n)·|acc(B_m) − conf(B_m)|. 6) Sum the contributions of all bins — the result is the ECE. 7) Optionally visualize via a reliability diagram and use adaptive binning when the confidence distribution is uneven.

Problem solved

It solves the problem of assessing the trustworthiness of a model's confidence: accuracy alone does not tell whether the stated probabilities are truthful. Modern networks tend to be overconfident; ECE quantifies this gap in a single number, enabling comparison of models and calibration methods.

Components

Confidence signalThe input that is split into bins.

The prediction's stated probability, usually the maximum softmax value.

Binning schemeGroups predictions to compare accuracy with confidence.

Division of the confidence range into M bins (equal-width or adaptive/equal-mass).

Per-bin accuracy-confidence gapA local measure of miscalibration.

The absolute difference between empirical accuracy and mean confidence within each bin.

Weighted aggregationCollapses many bins into a single metric.

Averaging the gaps with weights equal to the share of samples per bin, yielding the single ECE number.

Evolution

Original paper · 2017 · ICML 2017 · Chuan Guo
On Calibration of Modern Neural Networks
Chuan Guo, Geoff Pleiss, Yu Sun, Kilian Q. Weinberger
2015
Binned calibration estimation
Inflection point

Naeini, Cooper, and Hauskrecht formalize binned calibration assessment (Bayesian Binning), the foundation of ECE.

2017
ECE as a standard for neural networks
Inflection point

Guo et al. show the overconfidence of modern networks and popularize ECE and temperature scaling.

2019
Critiques and adaptive variants

Work highlights ECE's sensitivity to binning and proposes adaptive/equal-mass variants and complementary measures.

Hyperparameters (configurable axes)

Number of binsHigh

The number of confidence bins; too few underestimates ECE, too many increases variance with small samples.

10The most common default.
15Seen in neural-network calibration literature.
Binning schemeMedium

Equal-width vs adaptive (equal-mass) — affects the estimate's stability under a skewed confidence distribution.

Równa szerokośćClassic ECE.
Equal-mass (adaptacyjne)Adaptive ECE, equal bin counts.

Computational complexity

Time complexity: O(n + M). Space complexity: O(M).

Parallelism

Parallelism level
Fully parallel

The metric is ideally suited to map-reduce-style computation over large datasets.

Scope
InferenceAcross devices
Constraints
!Assigning predictions to bins is fully parallel; only a final reduction of per-bin sums is required.

Hardware requirements

Primary

ECE is a simple statistical metric computed over predictions; it needs no specialized accelerator.