Robots Atlas>ROBOTS ATLAS
Training

QAT

2018ActivePublished: 16 October 2026Updated: 16 October 2026Published
Key innovation
Introduced simulated ("fake") quantization into the training loop, with gradients propagated via the straight-through estimator (STE), so the model learns to compensate for quantization error and retains accuracy under low-bit inference.
Category
Training
Abstraction level
Pattern
Operation level
TrainingPost-training
Use cases
On-device inference on mobile and edge hardwareCompressing large language models to low precisionDeployment on integer-arithmetic-capable hardwareReducing memory and energy use while preserving accuracyExtremely low-bit models (e.g. ~2-bit or ternary)

How it works

1) Simulated quantization (fake-quant) nodes are inserted on weights and usually activations, together with observers that estimate value ranges (scale, zero-point). 2) In the forward pass, values are quantized to a low-bit grid and dequantized, so the model computes on quantization-noised values. 3) In the backward pass the non-differentiable rounding is bypassed with the straight-through estimator (STE), treating quantization as an identity for the gradient. 4) Weights are updated to minimise the loss despite quantization. 5) After training, the model is exported to a real integer/low-bit representation for efficient inference. QAT is often applied as a short fine-tuning stage on a pretrained model.

Problem solved

Low-bit quantization drastically reduces model size and inference cost, but naively rounding the weights of a trained model (PTQ) degrades accuracy, especially below 8-bit precision. QAT addresses this by letting the model learn to compensate for quantization error while still training.

Components

Simulated (fake) quantizationReproduces the quantization effect in the forward pass (quantize + dequantize) so the model trains on noised values.
Range observers (scale / zero-point)Estimate value ranges of weights and activations to derive quantization parameters (scale and zero-point).
Straight-through estimator (STE)Enables gradient propagation through the non-differentiable rounding operation by treating it as an identity.

Implementation

Reference implementations
PyTorch Quantization (torch.ao.quantization)
Python
Official
TensorFlow Model Optimization Toolkit — QAT
Python
Official
Implementation pitfalls
Non-differentiable quantizationHigh

The rounding operation blocks the gradient; without a workaround training does not learn.

Fix:Use the straight-through estimator (STE) to propagate gradients through quantization.
Activation quantization and outliersHigh

Activations are harder to quantize than weights; outliers heavily degrade accuracy at low precision.

Training costMedium

Full QAT requires substantial training resources, which can be impractical for large models.

Fix:Efficient QAT methods (e.g. block-wise training / EfficientQAT) reduce the cost.

Evolution

Original paper · 2018 · CVPR 2018 · Benoit Jacob
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, Dmitry Kalenichenko
2013
Straight-through estimator (STE)

Bengio, Léonard and Courville formalised gradient propagation through non-differentiable/stochastic neurons — the foundation for later QAT.

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (paper)
2018
QAT for integer-only inference (Google)
Inflection point

Jacob et al. co-designed a training procedure with simulated quantization, defining modern QAT for efficient CPU inference (MobileNets).

Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (paper)
2023
LLM-QAT (Meta)

Brought QAT to large language models with a data-free approach, quantizing weights, activations and the KV cache.

LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (paper)
2024
BitNet b1.58 — the era of 1-bit LLMs
Inflection point

Language models trained from scratch with ternary (1.58-bit) weights, demonstrating extreme low-bit quantization-aware training.

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (paper)
2024
EfficientQAT

An efficient QAT algorithm (block-wise training) enabling low-bit LLM quantization, including ~2-bit, with reduced resources.

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models (paper)

Hyperparameters (configurable axes)

Bit widthCritical

Target precision of weights/activations (e.g. 8, 4, 2 bits). Lower precision = smaller model, harder training.

Quantization granularityHigh

Per-tensor, per-channel, or per-group — trades off accuracy against overhead.

Quantization schemeHigh

Symmetric vs asymmetric; separate handling of weights and activations.

Fine-tuning budgetMedium

Number of steps/data in the QAT stage — balances training cost against accuracy recovery.

Parallelism

Parallelism level
Partially parallel

QAT parallelizes like ordinary training (data/model parallelism); the extra overhead comes from fake-quant operations.

Scope
TrainingAcross devices

Hardware requirements

Good fit

The original scheme targeted integer-only inference on commonly available CPUs (e.g. on-device MobileNets).

Good fit

Low-bit (INT8/INT4) tensor-core paths accelerate inference of QAT-trained models.