Low-bit training
How it works
The core is representing tensors in a narrow format (e.g. 8, 4 or fewer bits) while controlling quantization error. In mixed precision training matrix multiplications run in low precision, but accumulation and weight updates use a higher-precision master copy, and loss scaling prevents underflow of small gradients. For FP8 and FP4, scaling factors (per-tensor or per-block, as in microscaling MX formats) align the data's dynamic range to the narrow format. In quantization-aware training, "fake quantization" ops are inserted in the forward pass and gradients flow through them via a straight-through estimator. 1-bit/ternary architectures (BitNet) replace linear layers with a BitLinear variant that trains extreme low-bit weights from scratch.
Problem solved
Growing models demand ever more memory, bandwidth and energy to train. Standard FP32/FP16 precision wastes resources because the numeric range and precision exceed what is needed. Low-bit training shrinks weights and activations, speeds up matrix multiplications on dedicated hardware, and reduces communication volume in distributed training.
Components
A copy of the weights kept in FP32/BF16 and updated by the optimizer; prevents accumulation of rounding error when updates are small.
Multiplying the loss by a factor before backprop so that small gradients do not underflow in low precision; can be static or dynamic.
Official
Per-tensor or per-block factors aligning the data's dynamic range to a narrow format (FP8, FP4, MXFP4); shared within a block in MX formats.
Official
In QAT a quantization op in the forward pass simulates low precision, while the gradient passes through it via a straight-through estimator (STE).
Official
Matrix multiply executed in low precision with higher-precision accumulation, typically on GPU/TPU tensor cores.
Implementation
LLM activations contain large-magnitude outliers that, in a narrow format, cause large quantization error and instability.
Small gradients in FP16/FP8 flush to zero, stalling learning.
The quantization op is non-differentiable; the straight-through estimator (STE) introduces gradient bias that can hurt convergence.
1-bit/ternary and FP4 training is sensitive to hyperparameters, initialization and the scaling scheme; divergence is easy to hit.
Evolution
NVIDIA introduces FP16 training with master weights and loss scaling, cutting cost without accuracy loss.
Jacob et al. combine simulated quantization with training to enable integer-only inference.
A joint NVIDIA/Arm/Intel proposal standardizes an 8-bit floating point format for training and inference.
First scalable Transformer trained from scratch with 1-bit weights (BitLinear layer).
MX formats with a per-block shared scale enable training generative language models below 8 bits.
Weights {-1,0,1} match FP16 models at equal parameters and tokens, defining a new scaling law.
DeepSeek-V3 (671B-parameter MoE) uses an FP8 mixed precision training framework at very large scale.
NVIDIA demonstrates pretraining of large language models in the 4-bit NVFP4 format.
Hyperparameters (configurable axes)
Choice of format: BF16/FP16, FP8 (E4M3/E5M2), FP4/MXFP4/NVFP4, INT8, 1-bit/ternary. Governs the memory/accuracy tradeoff.
Level at which the scaling factor is chosen: per-tensor, per-channel or per-block (microscaling). Finer granularity improves stability at lower bit-widths.
PTQ (post-training) vs QAT (quantization-aware) vs native low-bit training from scratch. Affects quality at low bit-widths.
Static or dynamic factor preventing gradient underflow; important for FP16/FP8.
Computational complexity
Space complexity: O(b).
Parallelism
Hardware requirements
GPU tensor cores natively support low-bit formats: FP8 since the Hopper architecture, FP4 (NVFP4/MXFP4) since the Blackwell architecture.
TPUs support BF16 and integer formats as well as increasingly low precisions in training.
Without dedicated low-bit units the performance benefit is limited; the gain is mainly in memory.