TTFT
How it works
TTFT is measured as the elapsed time between sending a request and receiving the first non-empty token (this requires streaming mode). The total time comprises: (1) queueing and admission of the request into a batch, (2) the prefill phase — processing all prompt tokens in parallel across all model layers while building the KV cache, (3) computing the logits at the last position and sampling the first token, and (4) detokenisation and network transfer. The prefill phase is compute-bound (a matrix-matrix operation that saturates the GPU), so TTFT grows with prompt length and model size. Common techniques to reduce TTFT include chunked-prefill, prefix/KV cache reuse, prefill-decode disaggregation, tensor parallelism and more powerful compute hardware.
Problem solved
Throughput (tokens/s) and total latency alone do not capture how quickly a user sees the start of a response. TTFT isolates the delay before generation begins — the factor that, in streaming and conversational interfaces, determines perceived responsiveness — and lets teams optimise and set SLOs for the prefill phase independently of the decode phase.
Components
Time a request waits to be admitted into the current batch by the inference server scheduler before computation begins.
Parallel processing of all prompt tokens across all model layers; the main, compute-bound component of TTFT.
Storing keys and values (K, V) for all prompt tokens and layers, later reused during the decode phase.
Decoding the first output token from the last-position logits (greedy, top-k, top-p, temperature).
Implementation
Without streaming the server returns the whole response at once, so real time-to-first-token cannot be measured.
Pure TTFT (from prefill start) and end-to-end TTFT (including queueing) yield different numbers, hampering comparisons.
Weight loading, graph compilation and memory allocation inflate the first TTFT measurement.
Some APIs return an empty or metadata first chunk (e.g. a role), which understates TTFT if counted as a token.
Network and detokenisation delays affect client-perceived TTFT even though server-side measurements often omit them.
Evolution
TTFT, together with TPOT and throughput, becomes a standard set of metrics in LLM inference performance engineering (e.g. Databricks and NVIDIA guides).
Splitting prompt computation and token generation across separate machines, enabling independent management of TTFT and generation throughput.
Disaggregating prefill and decode onto different GPUs allows independent optimisation of TTFT (prefill) and per-token time (decode) instead of trading them off.
Splitting prefill into chunks with stall-free scheduling sustains high throughput while controlling the impact of batching on TTFT and latency.
Hyperparameters (configurable axes)
Number of input tokens; the main driver of TTFT.
Parameter and layer count affects prefill cost per token.
Accelerator compute throughput determines prefill speed.
Larger batches and load increase queueing and can raise TTFT.
Caching a shared prefix shortens prefill and TTFT.
Splitting prefill into chunks balances TTFT against throughput.
Sharding the model across GPUs can shorten prefill for large models.
Computational complexity
Time complexity: O(n^2 * d). Space complexity: O(n * d * L).
Compute bottleneck
TTFT is dominated by the compute-bound prefill — a dense matrix-matrix operation that saturates the accelerator compute units.
Execution paradigm
TTFT measures the prefill cost — a dense matrix-matrix operation activating all model parameters over all prompt tokens.
Parallelism
Prefill processes all prompt tokens in parallel (matrix-matrix), unlike the sequential decode phase, which TTFT does not include.
Hardware requirements
Prefill is a compute-bound matrix-matrix operation that benefits strongly from high FLOPS and tensor cores, shortening TTFT.
Matrix accelerators (TPUs) handle compute-bound prefill for large prompts well.
On CPUs, prefill of long prompts is slow, leading to high TTFT.