Robots Atlas>ROBOTS ATLAS
Inference

Decoding

2018ActivePublished: 19 November 2026Updated: 19 November 2026Published
Key innovation
Framing text generation as a separate, swappable stage: given the model's probability distribution over the vocabulary, a decoding strategy (greedy, beam search, sampling) decides which tokens to pick — letting one steer quality, diversity and speed without changing the trained weights.
Category
Inference
Abstraction level
Pojęcie
Operation level
Inference
Use cases
Open-ended text generation (chat, creative writing) with top-p samplingMachine translation and summarization with beam searchDeterministic answers (code, data extraction) with greedy / temperature=0Controlling answer diversity via temperature and top-k/top-pSchema-guided decoding (JSON, grammar) for structured outputAccelerating generation with speculative decoding

How it works

1. The model runs a forward pass and, for the current position, returns a logits vector of vocabulary size. 2. The logits are divided by temperature T and passed through a softmax, giving the next-token probability distribution. 3. The decoding strategy narrows and/or transforms the distribution: greedy takes the argmax; top-k keeps the k highest probabilities; top-p keeps the smallest set summing to ≥ p; beam search expands k hypotheses in parallel. 4. One token is selected from the prepared distribution — randomly (sampling) or deterministically (greedy/beam). 5. The token is appended to the sequence, the KV cache is updated, and it is fed back as model input. 6. Steps 1–5 repeat until an end-of-sequence (EOS) token or a length limit. Parameters such as temperature, top-k, top-p, repetition/presence penalty and the number of beams control the trade-off between coherence and diversity.

Problem solved

A trained language model only returns a probability distribution over the vocabulary, not finished text — something must decide which token to pick at each step. Naively taking the most probable token (greedy) yields text that is correct but repetitive and dull, while pure sampling from the full distribution is incoherent. Decoding solves this by providing a family of strategies that balance quality, diversity and determinism of generation without retraining the model, while also letting one control the cost and speed of generation.

Components

Softmax distribution over logitsProducing the next-token distribution

Turning the model's raw logits (after temperature scaling) into a probability distribution over the whole vocabulary — the input for every token-selection strategy.

Greedy decodingDeterministic baseline strategy

Picking the highest-probability token (argmax) at each step. Deterministic and fast, but locally greedy, prone to repetition and globally suboptimal.

Official

Beam searchDeterministic search-based strategy

Keeping the k best partial sequences (beams) by cumulative log-probability. Better for closed-ended tasks (translation, summarization); for open-ended text it tends to be bland and repetitive.

Official

Temperature samplingControlling the randomness of the distribution

Scaling logits by temperature T before softmax: T<1 sharpens the distribution (closer to greedy), T>1 flattens it (more randomness), T=0 is equivalent to greedy.

Official

Top-k samplingSampling from a truncated vocabulary (fixed k)

Restricting the choice to the k most probable tokens, renormalizing their probabilities and sampling from that set. Cuts off the long tail of unlikely tokens.

Official

Top-p (nucleus) samplingSampling from an adaptive nucleus

Choosing the smallest set of tokens whose cumulative probability is ≥ p and sampling from it. The set size adapts to the shape of the distribution — more options when the model is uncertain.

Official

Implementation

Implementation pitfalls
Beam search for open-ended textMedium

Using beam search for open-ended generation gives bland, repetitive output (degeneration); top-p sampling is better for chat/creative writing.

Fix:For open-ended text use sampling (top-p/top-k with temperature); reserve beam search for closed-ended tasks.
Poorly chosen temperatureHigh

Too high a temperature leads to incoherence and hallucination, too low to repetition and rigidity.

Fix:Tune temperature to the task: ~0 for code/extraction, 0.7–1.0 for chat, higher for creativity.
No repetition penaltyMedium

Without a repetition penalty the model falls into repetition loops, especially at low temperature.

Fix:Enable a moderate repetition/presence penalty (e.g. 1.1–1.3) or no-repeat n-gram.
Unaware ordering of sampling filtersLow

The order in which temperature, top-k and top-p are applied, and how they combine, affects the result; libraries differ.

Fix:Check the filter order in your library and test parameter combinations together, not in isolation.

Evolution

2016
Beam search becomes standard in neural seq2seq

Beam search becomes the default decoding strategy in neural machine translation and other seq2seq tasks.

2018
Top-k sampling for open-ended generation

Fan et al. (Hierarchical Neural Story Generation) popularize top-k sampling as a way to get livelier, less repetitive generation.

2020
Top-p / nucleus sampling and the text-degeneration problem
Inflection point

Holtzman et al. (The Curious Case of Neural Text Degeneration) show that beam search and greedy lead to text degeneration, and introduce nucleus (top-p) sampling as a better strategy for open-ended text.

2023
Speculative decoding accelerates generation
Inflection point

Leviathan et al. and Chen et al. introduce speculative decoding: a small draft model proposes tokens and the target model verifies them — a speedup without changing the output distribution.

2024
EAGLE/EAGLE-3 and structure-guided decoding

Advances in speculative methods (EAGLE-3) and grammar/JSON-guided decoding make faster and structurally valid generation mainstream.

Hyperparameters (configurable axes)

TemperatureCritical

Scale of the logits before softmax; controls randomness. Lower = more deterministic, higher = more creative.

0Deterministic (greedy).
0.7Balanced, typical for chat.
>1.0More random, creative (risk of incoherence).
Top-p (nucleus)High

Cumulative-probability threshold for the sampling nucleus; 1.0 disables truncation.

1.0No truncation (full distribution).
0.9Typical value for open-ended text.
Top-kMedium

Number of most-probable tokens considered; 0 disables truncation.

0Disabled.
40-50Common range for sampling.
Beam widthMedium

Number of hypotheses expanded in parallel in beam search; 1 = greedy.

1Greedy.
4-8Typical for translation/summarization.
Repetition penaltyMedium

Lowers the probability of already-generated tokens, reducing repetition loops.

1.0No penalty.
1.1-1.3Mild anti-repetition penalty.

Computational complexity

Time complexity: O(V) na krok (dobór tokenu) + O(k·V) dla beam search. Space complexity: O(V) na rozkład; O(k·L) dla hipotez beam search.

Compute bottleneck

Model forward pass (memory-bound decode)

The decoding step time is dominated by the model forward pass (reading weights and the KV cache from memory), not by token selection — so decoding inherits the memory-bound nature of the decode phase.

Depends on
Przebieg forward modeluSzerokość wiązki / liczba próbek

Execution paradigm

Primary mode
Dense

Decoding is a token-selection layer on top of a dense autoregressive model pass.

Activation pattern
All paths active
Routing mechanism

Decoding introduces no conditional routing — it operates on the output of a dense forward pass (unless the base model is MoE).

Parallelism

Parallelism level
Sequential

Tokens are produced sequentially (each depends on the previous). Parallelism comes across requests (batching) and across beam-search hypotheses, not within a single sequence. Speculative decoding partly breaks this sequentiality.

Scope
Inference

Hardware requirements

Primary

Generation is dominated by the memory-bound model forward pass; GPUs with high memory bandwidth (HBM) set tokens per second. Token selection itself is computationally trivial.

Good fit

The decoding-strategy logic (argmax, top-k, top-p, beam) is lightweight and portable; its cost is negligible relative to the model pass on any hardware.