Throughput is measured as the number of generated (or processed) tokens divided by time, usually under a given load and batch size. It rises when more requests are processed in parallel (larger batch, continuous batching) and when hardware is better utilized (lower KV memory overhead via PagedAttention/GQA, quantization, FlashAttention). Because the decode phase is memory-bandwidth bound, batching amortizes weight reads across many requests, raising throughput โ but at the cost of higher single-request latency. A practical measure is 'goodput': throughput counting only requests that meet latency targets (SLA).
LLM deployments must serve many concurrent requests at an acceptable cost. Token Throughput as a metric lets teams measure and optimize scale and cost per token, independently of single-request latency.
Total tokens/s across all concurrent requests on the server.
Tokens/s delivered to a single user (the inverse of TPOT).
Throughput counted only for requests meeting SLA targets (e.g. latency limits).
Official
A very large batch raises tokens/s but ruins single-request responsiveness.
High raw throughput may include requests that violate latency limits.
Measurements at fixed, ideal sequence lengths overstate throughput versus production.
Production LLM serving puts tokens/s alongside latency as a key performance and cost metric.
A dramatic serving-throughput increase via efficient KV-cache management and continuous batching.
Optimizing throughput against SLA targets (goodput) and separate resources for prefill and decode phases.
Time complexity: TP โ (batch ร tokeny) / czas.
The main throughput driver; a larger batch = higher throughput, higher latency.
Tensor/pipeline parallelism and the number of accelerators increasing throughput.
Lower precision increases throughput and reduces memory use.
Throughput is bounded by memory bandwidth (HBM) and Tensor Core compute; batching amortizes weight reads.
High-memory-bandwidth accelerators achieve high token throughput.
