Inputs are processed (e.g. text tokenization) and passed through the network (a forward pass) with fixed weights. In autoregressive models, prefill happens first - computing the prompt's representation in parallel and populating the KV cache - then decode: in a loop the model predicts the next-token distribution, samples a token (per a decoding strategy, e.g. temperature/top-p), appends it to the context and repeats until a stop condition. To improve efficiency, techniques such as KV caching, request batching, weight/activation quantization and specialized hardware are used. The result is returned as a prediction or generated sequence.
A trained model is useless until it is run on new data. Inference is the stage that delivers actual value to users - and its latency, throughput and cost determine the viability and usefulness of a deployment.
Computing the input representation in parallel and populating the KV cache; determines time-to-first-token.
Sequential, autoregressive generation of the next tokens, one per step.
How a token is chosen from the distribution (e.g. greedy, temperature, top-p, top-k, beam search).
Official
Caching attention keys and values to avoid recomputing the whole context for every token.
Official
Larger batches raise throughput but also latency; it is hard to optimize all three at once.
The KV cache grows linearly with context length, quickly exhausting accelerator memory.
As LLMs spread, techniques to lower inference cost matured: KV cache, continuous batching, quantization.
Speeding up decoding via a draft model proposing tokens verified by a larger model.
Reasoning models began spending more compute at inference to improve quality - inference became a lever for capability too.
