The prompt is tokenized and processed in the prefill phase: the model computes representations for all tokens at once and stores key-value pairs for the attention layers (the KV cache). Then, in the decode phase, the model uses the KV cache to generate subsequent tokens one at a time; each new token appends its entries to the KV cache. The inference server batches requests, manages KV memory (e.g. PagedAttention), applies sampling parameters (temperature, top-p) and may use acceleration techniques (speculative decoding, quantization, FlashAttention). Generation ends when a stop token or length limit is reached.
Running an LLM to generate text is costly and slow because autoregressive decoding is sequential and memory-bandwidth bound. LLM Inference as a field addresses how to generate tokens fast, cheaply and at high throughput without changing the model weights.
Parallel processing of the whole prompt and populating the KV cache; usually compute-bound.
Autoregressive token-by-token generation using the KV cache; usually memory-bound.
A cache of attention key-value pairs, avoiding recomputation for earlier tokens.
Official
The server layer that batches requests and manages memory (e.g. continuous batching, PagedAttention).
Official
Long context and large batches quickly exhaust GPU memory via a growing KV cache.
Large batches raise throughput but hurt single-request latency, and vice versa.
Too-low precision (e.g. INT4) can degrade generation quality.
The architecture underlying token-by-token generation and the KV cache.
An attention speedup critical for prefill and long context.
Efficient KV-cache management and continuous batching drastically increased serving throughput.
A draft model speeds up generation without quality loss.
Separate resources for prefill and decode phases optimize cost and latency in large deployments.
Time complexity: O(n·d² + n²·d). Space complexity: O(P + L·n·d).
How requests are batched; a latency vs throughput trade-off.
The numeric format of weights/activations affecting speed, memory and quality.
Temperature, top-p, top-k controlling generation randomness.
For dense models all weights are active per token; MoE models activate a subset of experts (mixture).
Prefill is highly parallel across tokens; decode is sequential across tokens but parallel across requests (batching) and devices (tensor/pipeline parallelism).
LLM inference is heavy matrix multiplication; memory bandwidth (HBM) is also critical.
Models are also served on TPUs and other accelerators.
Possible (e.g. llama.cpp) for small models, but slower.
