Robots Atlas>ROBOTS ATLAS
Inference

Inference

ActivePublished: 6 August 2026Updated: 6 August 2026Published
Key innovation
Treating the phase of using a trained model (generating predictions for new data) as a distinct stage with its own latency, throughput and cost requirements - separate from training.
Category
Inference
Abstraction level
Primitive
Operation level
ModelServingApplication
Use cases
Running models in production (APIs, applications)Generating text/image/code with generative modelsReal-time classification, detection and predictionEdge and on-device inferenceTest-time compute scaling for better quality

How it works

Inputs are processed (e.g. text tokenization) and passed through the network (a forward pass) with fixed weights. In autoregressive models, prefill happens first - computing the prompt's representation in parallel and populating the KV cache - then decode: in a loop the model predicts the next-token distribution, samples a token (per a decoding strategy, e.g. temperature/top-p), appends it to the context and repeats until a stop condition. To improve efficiency, techniques such as KV caching, request batching, weight/activation quantization and specialized hardware are used. The result is returned as a prediction or generated sequence.

Problem solved

A trained model is useless until it is run on new data. Inference is the stage that delivers actual value to users - and its latency, throughput and cost determine the viability and usefulness of a deployment.

Components

Prefill (prompt processing)The initial phase of LLM inference.

Computing the input representation in parallel and populating the KV cache; determines time-to-first-token.

Decode (token generation)The generation phase.

Sequential, autoregressive generation of the next tokens, one per step.

Decoding strategy (sampling)Controls output quality/diversity.

How a token is chosen from the distribution (e.g. greedy, temperature, top-p, top-k, beam search).

Official

KV cacheThroughput/latency optimization.

Caching attention keys and values to avoid recomputing the whole context for every token.

Official

Implementation

Implementation pitfalls
Latency-throughput-cost trade-offMedium

Larger batches raise throughput but also latency; it is hard to optimize all three at once.

Fix:Continuous batching, request prioritization, and matching hardware/quantization to the traffic profile.
KV cache memory at long contextMedium

The KV cache grows linearly with context length, quickly exhausting accelerator memory.

Fix:KV compression/quantization, paged attention and context-window management.

Evolution

2022
LLM inference optimization (KV cache, batching)

As LLMs spread, techniques to lower inference cost matured: KV cache, continuous batching, quantization.

2023
Speculative decoding

Speeding up decoding via a draft model proposing tokens verified by a larger model.

2024
Test-time scaling (thinking at inference)
Inflection point

Reasoning models began spending more compute at inference to improve quality - inference became a lever for capability too.