Model cascade
How it works
1) Order the models from cheapest to most expensive (e.g. small, medium, large). 2) Define a confidence signal for each stage: probability, margin, entropy, calibrated confidence, or a separate verifier model. 3) For a given input run stage 1; if confidence exceeds the deferral threshold, accept the answer (early exit). 4) Otherwise escalate to the next stage and repeat. 5) The final stage always returns an answer (no further escalation). 6) Tune the thresholds to hit a target cost/quality trade-off; thresholds can be calibrated on a validation set, minimizing cost subject to a bounded accuracy drop.
Problem solved
It resolves the cost-versus-quality tension in inference: the largest models are accurate but expensive and slow, and applying them to every query — including easy ones — wastes resources. A cascade pays the high cost only for the hard cases that actually need it.
Components
A sequence of models of increasing cost and accuracy, from cheapest to most expensive.
Official
A signal assessing the reliability of a stage's answer: probability, margin, entropy, calibrated confidence, or a verifier.
A decision threshold comparing confidence to a cutoff and choosing early exit or escalation.
Evolution
A cascade of simple classifiers enables fast real-time face detection — the classic example of a cascade.
Early-exit architectures (e.g. BranchyNet) bring the cascade idea inside a single network.
FrugalGPT-style approaches show that cascading cheap and expensive LLMs cuts inference cost while preserving quality.
Hyperparameters (configurable axes)
The confidence cutoff deciding acceptance vs escalation; the main knob for the cost/quality trade-off.
How many models form the cascade; more stages give finer cost gradation but complicate threshold calibration.
Computational complexity
Time complexity: E[cost] = c1 + (1-p1)·c2 + (1-p1)(1-p2)·c3 + ....
Execution paradigm
For easy inputs only the first stage is active; expensive models stay inactive.
Conditional routing: further, more expensive models are activated only when a cheaper stage's confidence does not exceed the threshold.
Parallelism
A single input is processed sequentially across stages, but many inputs can be processed in parallel (batched) within each stage.
Hardware requirements
A cascade is an orchestration technique over models; it runs on whatever hardware the constituent models run on.