Latency is measured along a single request's timeline: from sending the prompt to the first token (TTFT), then the gaps between subsequent tokens (ITL) and the moment of the last token (end-to-end latency). TTFT depends mainly on the prefill phase (prompt length, compute), while TPOT depends on the decode phase (memory-bandwidth bound: reading weights and the KV cache at each step). Optimization means shrinking these components: a smaller/quantized model, GQA and a smaller KV cache, FlashAttention, speculative decoding, better hardware (HBM), prefill/decode disaggregation, and choosing a batch size that meets the latency target.
Slow responses hurt user experience and limit interactive and agentic use. Inference Latency as a metric makes it possible to precisely measure and optimize responsiveness (TTFT, TPOT) independently of throughput.
Time from sending the prompt to the first generated token; dominated by the prefill phase.
Average time to generate each subsequent token during decoding.
Total time to complete the response (โ TTFT + number_of_tokens ร TPOT).
Mean latency hides bad experiences; the tail (p95/p99) is what matters.
Optimizing throughput (large batch) can worsen user latency.
Measurements without variable traffic, context length and network don't reflect production.
With production LLM use, latency becomes a first-class metric alongside throughput.
TTFT and time-per-token become standard metrics for LLM comparisons and SLAs.
Long 'thinking' chains materially increase end-to-end latency, forcing new trade-offs.
Time complexity: L โ TTFT + nยทTPOT.
A larger batch raises throughput but hurts single-request latency.
A smaller or quantized model reduces latency at some quality cost.
A longer prompt lengthens prefill (TTFT), and a growing KV cache raises TPOT.
Decode latency is memory-bound โ high GPU memory bandwidth (HBM) is key.
High-memory-bandwidth accelerators also lower TPOT.
