Inference

Ultrafast Inference

ActivePublished: 1 October 2026Updated: 1 October 2026Published

Key innovation

Treating inference serving as a dedicated, heavily optimized layer that combines runtime techniques (continuous batching, KV-cache, speculative decoding, quantization) with specialized hardware to drastically cut token-generation latency and cost without changing model weights.

How it works

The layer intercepts incoming requests and merges them on the fly (continuous batching), adding new sequences to the running batch instead of waiting for prior ones to finish. Attention state is kept in a KV-cache, often managed in pages (PagedAttention) to limit memory fragmentation. Speculative decoding uses a small draft model to propose several tokens at once, which the large model verifies in parallel. Quantization (e.g. to 8 or 4 bits) reduces memory and bandwidth usage. Optimized attention kernels minimize data movement between HBM and registers. Everything runs on tensor-core GPUs or dedicated accelerators, frequently with tensor and pipeline parallelism across devices.

Problem solved

Naive LLM inference is slow and expensive: autoregressive generation produces one token at a time, the KV memory grows linearly with context length, and static request batching wastes GPU capacity under variable sequence lengths. An ultrafast inference layer addresses high latency and low hardware utilization when serving models in production.

Components

Continuous batchingRaises GPU utilization and throughput under variable sequence lengths.

Dynamically adding and removing sequences from the running batch during generation, without waiting for all requests to finish.

Official

KV-cache managementEliminates recomputation of attention for already-generated tokens.

Storing and reusing key-value tensors from prior decoding steps; often managed in pages (PagedAttention) to limit memory fragmentation.

Official

Speculative decodingReduces the number of expensive large-model passes per generated token.

A small draft model proposes several tokens at once, which the large model verifies in a single pass, accepting correct prefixes.

Official

QuantizationEnables larger batches and faster weight loading at the cost of potential accuracy loss.

Representing weights and/or activations in lower precision (e.g. INT8, FP8, INT4) to reduce memory and bandwidth usage.

Official

Optimized attention kernelsSpeeds up attention computation and lowers memory usage.

IO-aware attention kernels (e.g. FlashAttention) that minimize data movement between HBM and registers.

Official

Hardware accelerationProvides the raw compute and memory bandwidth required for low latency.

Use of tensor-core GPUs or dedicated inference accelerators, often with tensor and pipeline parallelism across devices.

Official

Implementation

Reference implementations

vLLM

Python · vLLM project

NVIDIA TensorRT-LLM

C++/Python · NVIDIA

Hugging Face Text Generation Inference (TGI)

Rust/Python · Hugging Face

SGLang

Python · SGLang team

llama.cpp

C/C++ · ggml / Georgi Gerganov

Implementation pitfalls

Latency-throughput trade-offHigh

Increasing batch size raises throughput but can increase single-request latency.

Fix:Tune batch size and scheduler policies to target latency SLOs; prioritize interactive requests.

KV-cache memory pressure and OOMHigh

Long contexts and many concurrent sequences can exhaust GPU memory.

Fix:Paged KV management (PagedAttention), context-length limits, preemption, and offload.

Quantization accuracy degradationMedium

Aggressive quantization (e.g. INT4) can reduce model output quality.

Fix:Calibration, mixed-precision quantization, and quality evaluation before deployment.

Low speculative-decoding acceptance rateMedium

Poor alignment between draft and target model limits or negates the speedup.

Fix:Choose a well-aligned draft model and tune the draft length.

Evolution

2022

FlashAttention — IO-aware attention kernels

Inflection point

Optimized exact-attention kernels reducing memory traffic, a foundation for fast GPU inference.

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (paper)

2022

Speculative decoding

Inflection point

Speeding up autoregressive generation via parallel verification of tokens proposed by a small model.

Fast Inference from Transformers via Speculative Decoding (paper)

2023

PagedAttention / vLLM

Inflection point

Paged KV-cache management and continuous batching in an open serving engine, popularizing high-throughput LLM serving.

Efficient Memory Management for Large Language Model Serving with PagedAttention (paper)

Hyperparameters (configurable axes)

Max batch sizeCritical

Maximum number of sequences served concurrently; a throughput-versus-latency trade-off.

Quantization precisionHigh

Numeric precision of weights/activations (e.g. FP16, FP8, INT8, INT4).

Context length / KV-cache sizeHigh

Maximum context length; directly determines KV-cache memory usage.

Speculative decoding draft lengthMedium

Number of tokens proposed at once by the draft model; affects the gain from parallel verification.

Parallelism

Parallelism level

Partially parallel

Parallelism across requests (batching) and across devices (tensor/pipeline parallelism); autoregressive decoding remains sequential per sequence, partially mitigated by speculative decoding.

Scope

InferenceAcross tokensAcross devices