Inference
Model Quantization
ActiveUpdated: 28 August 2026Published
Key
innovation
Reducing the numerical precision of weights/activations (e.g. FP16 to INT8/INT4/FP8/FP4) to cut memory and speed up inference.
Category
Inference
Abstraction level
Pattern
Operation level
Architecture blockModel
Use cases
Deploying LLMs on memory-limited GPUsEdge and on-device inferenceReducing model-serving costsFP8/FP4/INT4/AWQ/GPTQ formats in vLLM/SGLang
How it works
Values are mapped from higher to lower precision using a scale (and zero point), often per-channel or per-group. Quantization can be post-training (PTQ, e.g. GPTQ, AWQ) or quantization-aware (QAT); modern accelerators support FP8/FP4 formats in hardware.
Problem solved
Large models require huge memory and bandwidth; full precision is costly and hinders deployment on resource-constrained devices.