EAGLE-3
How it works
1) The draft model generates a tree of candidate tokens. 2) Instead of predicting target-layer features (as in EAGLE-1/2), EAGLE-3 directly predicts tokens, using fused features from the target model's low, mid, and high layers. 3) The "training-time test" technique simulates the draft model's multi-step generation during training, reducing the train-inference mismatch. 4) The target model verifies the entire candidate tree in a single pass; the longest correct paths are accepted, guaranteeing output identical to standard decoding (lossless).
Problem solved
Autoregressive decoding in LLMs is memory-bound: each token requires a separate model pass, so generation underuses GPU compute. Earlier EAGLE versions sped this up, but their gains scaled poorly with training data due to feature-prediction constraints. EAGLE-3 addresses this scalability problem, letting the draft model improve as training data grows.
Components
A lightweight autoregressive network producing cheap token proposals that the target model verifies in a single pass.
Combines representations from the target model's low, mid, and high layers instead of relying solely on top-layer features.
Simulates the draft model's multi-step generation during training, reducing the train-inference mismatch and enabling the removal of feature prediction.
Candidate tokens are arranged into a tree (dynamic draft trees from EAGLE-2) and verified in parallel; the longest correct paths are accepted.
Implementation
A draft model trained without simulating multi-step generation achieves a lower acceptance rate at inference.
The draft model is trained per target model; acceptance rate and speedup depend on distribution alignment.
Evolution
Introduced feature-level autoregression for speculative sampling (ICML 2024).
Dynamic draft trees improving the acceptance rate (EMNLP 2024).
Dropped feature prediction in favour of direct token prediction and multi-layer feature fusion; training-time test technique (arXiv March 2025, accepted to NeurIPS 2025).
Hyperparameters (configurable axes)
Number of candidate tokens in the tree verified per target-model pass; affects acceptance rate and speedup.
Choice of target-model layers (low/mid/high) fused as the draft model's input.
Parallelism
Many candidate tokens (the draft tree) are verified in parallel in a single target-model pass, while the draft model itself remains autoregressive.
Hardware requirements
Mitigates the memory-bound nature of decoding on GPUs by making better use of compute; benchmarked on hardware such as RTX 3090 GPUs.
The method operates at the decoding-algorithm level and requires no specific hardware; it has been integrated into many serving frameworks.