Numerical precision
How it works
A floating-point number encodes a sign, an exponent and a mantissa: more exponent bits mean a wider dynamic range, more mantissa bits mean higher precision. BF16 keeps FP32's range (8 exponent bits) with only 7 mantissa bits, so it usually needs no loss scaling, whereas FP16 (5 exponent bits) needs loss scaling to avoid gradient underflow. FP8 comes as E4M3 (more precision) and E5M2 (more range); 4-bit formats use block microscaling, where a block of values shares a common scale factor. In mixed-precision training the matmuls run in FP16/BF16/FP8 on Tensor Cores while master weights and accumulation stay in FP32. In inference, quantization maps weights and activations to INT8/INT4/FP8/FP4 using scale factors (per-tensor, per-channel or block). Halving the precision typically doubles Tensor Core throughput and halves data movement.
Problem solved
FP32 is expensive in memory, bandwidth and energy, which caps model size and the speed of training and inference. Lower precision cuts those costs dramatically but risks accuracy loss and numerical instability. Numerical precision is the central knob for trading compute cost against model quality.
Components
Splitting a number's bits into sign, exponent and mantissa decides the range and accuracy of the format (e.g. FP32 1-8-23, FP16 1-5-10, BF16 1-8-7).
At a fixed bit count, more exponent bits widen the range at the cost of precision — this is what separates FP16 from BF16.
Scale factors (global, per-channel or block-wise as in MXFP8/NVFP4) keep values inside the format's representable range.
Even when multiplications run in low precision, accumulators usually operate in FP32 to limit accumulated rounding error.
Implementation
FP16's narrow range flushes small gradients to zero; without loss scaling training diverges.
A few large activation values stretch the scale and degrade the accuracy of the remaining values.
Accumulating sums in low precision compounds rounding error and corrupts the matmul result.
Different precisions and reduction orders yield different bit-for-bit results across runs and devices.
Evolution
Establishes the ubiquitous floating-point standard that became the default precision for scientific computing and early machine learning.
Formalizes the 16-bit floating-point format later central to deep-learning acceleration.
Micikevicius et al. and NVIDIA Volta Tensor Cores make low precision mainstream for large-model training.
BF16 combines FP32's range with a 16-bit footprint, simplifying training without loss scaling.
TF32 keeps FP32's range with FP16's mantissa precision, speeding up matmuls with no code changes.
The FP8 format spec and Hopper's hardware support enable 8-bit training and inference.
The Open Compute Project standardizes block formats MXFP8/MXFP6/MXFP4, where a block of values shares a common scale.
Hardware support for 4-bit formats pushes the model-scale frontier while preserving quality through microscaling.
Hyperparameters (configurable axes)
Total number of bits per value (32, 16, 8, 4).
The number of exponent bits sets the format's dynamic range.
The number of mantissa bits sets representation precision.
Whether the scale is per-tensor, per-channel or block (microscaling).
The precision in which products are summed (usually FP32).
Computational complexity
Space complexity: O(n · b).
Compute bottleneck
In many AI workloads the limiter is data movement and matrix-unit throughput; lower precision relieves both bottlenecks, shifting the pressure onto representation quality.
Hardware requirements
Tensor Cores are designed for low precision — throughput rises with each bit reduction (FP16/BF16, FP8 on Hopper, FP4 on Blackwell).
TPUs natively support BF16, the format created specifically for machine learning.
CPU extensions (AVX-512 VNNI) accelerate INT8 inference, though low-precision floating point is limited.
Neuromorphic chips use their own heavily quantized representations, far from standard IEEE/OCP formats.