Robots Atlas>ROBOTS ATLAS
Inference

Token Throughput

2023ActiveUpdated: 14 August 2026Published
Key innovation
Frames model-serving efficiency as the rate of token processing (tokens/s) at system scale, complementing latency and enabling optimization of deployment cost and scalability.
Category
Inference
Abstraction level
Primitive
Operation level
InferenceServing
Use cases
Capacity and cost planning for LLM deploymentsComparing serving-engine and hardware efficiencyOptimizing cost per token (tokens/s per accelerator)Choosing batch size and parallelism strategyBalancing the throughput vs latency trade-off

How it works

Throughput is measured as the number of generated (or processed) tokens divided by time, usually under a given load and batch size. It rises when more requests are processed in parallel (larger batch, continuous batching) and when hardware is better utilized (lower KV memory overhead via PagedAttention/GQA, quantization, FlashAttention). Because the decode phase is memory-bandwidth bound, batching amortizes weight reads across many requests, raising throughput โ€” but at the cost of higher single-request latency. A practical measure is 'goodput': throughput counting only requests that meet latency targets (SLA).

Problem solved

LLM deployments must serve many concurrent requests at an acceptable cost. Token Throughput as a metric lets teams measure and optimize scale and cost per token, independently of single-request latency.

Key mechanisms

Tokens per second (tok/s) at system and per-request level
Goodput โ€” throughput meeting SLA targets
Continuous batching amortizing weight reads
Parallelism (tensor / pipeline parallelism)
Efficient KV cache (PagedAttention), GQA, quantization
Bounded by memory bandwidth and compute

Strengths & limitations

Strengths
โœ“Measures serving scale and cost at server scale
โœ“Directly tied to cost per token
โœ“Enables capacity planning and system comparisons
โœ“Complements latency for a full performance picture
Limitations
โœ—In tension with latency (large batch = higher latency)
โœ—Raw throughput ignores SLAs โ€” 'goodput' is better
โœ—Strongly dependent on load, sequence length and hardware
โœ—Bounded by memory bandwidth during decoding
โœ—Hard to sustain under variable, bursty traffic

Components

System throughputServing scale

Total tokens/s across all concurrent requests on the server.

Per-request throughputUser experience

Tokens/s delivered to a single user (the inverse of TPOT).

GoodputUseful throughput

Throughput counted only for requests meeting SLA targets (e.g. latency limits).

Official

Implementation

Reference implementations
MLPerf Inference (benchmark)
โ€” ยท MLCommons
Official
vLLM โ€” benchmark serwowania
Python ยท vLLM
Official
Implementation pitfalls
Maximizing throughput at the cost of latencyHigh

A very large batch raises tokens/s but ruins single-request responsiveness.

Fix:Optimize 'goodput' against SLAs; continuous batching, priorities, autoscaling.
Reporting throughput without SLAsMedium

High raw throughput may include requests that violate latency limits.

Fix:Report goodput and latency distribution (p95/p99) alongside tok/s.
Benchmarking without realistic loadMedium

Measurements at fixed, ideal sequence lengths overstate throughput versus production.

Fix:Test on representative length distributions and variable traffic.

Evolution

2022
Throughput as a key LLM serving metric

Production LLM serving puts tokens/s alongside latency as a key performance and cost metric.

2023
Continuous batching / PagedAttention (vLLM)
Inflection point

A dramatic serving-throughput increase via efficient KV-cache management and continuous batching.

2024
Goodput and prefill/decode disaggregation

Optimizing throughput against SLA targets (goodput) and separate resources for prefill and decode phases.

Hyperparameters (configurable axes)

Batch sizeCritical

The main throughput driver; a larger batch = higher throughput, higher latency.

large / continuousMaximizing throughput (continuous batching).
smallLower throughput, better latency.
ParallelismHigh

Tensor/pipeline parallelism and the number of accelerators increasing throughput.

tensor / pipelineSplitting the model across many GPUs.
Precision / quantizationHigh

Lower precision increases throughput and reduces memory use.

FP8/INT8/INT4More tokens/s at some quality cost.

Computational complexity

Time complexity: TP โ‰ˆ (batch ร— tokeny) / czas.

Hardware requirements

Primary

Throughput is bounded by memory bandwidth (HBM) and Tensor Core compute; batching amortizes weight reads.

Good fit

High-memory-bandwidth accelerators achieve high token throughput.