A BF16 number encodes a value as sign × 1.mantissa × 2^(exponent − 127), identically to FP32 but with the mantissa truncated to 7 bits. In mixed-precision training, weights, activations, and gradients are stored and multiplied in BF16, while the accumulation of products (e.g. in matrix multiplication on Tensor Cores / MXUs) is done in FP32, and a FP32 "master copy" of the weights is often kept for updates. Reducing mantissa precision increases rounding error, but the wide exponent range prevents overflow, so networks reach accuracy close to FP32 in the same number of iterations.
FP16 (half precision) has only 5 exponent bits, so its narrow dynamic range causes overflows and gradient underflow during training, forcing techniques such as loss scaling. BF16 solves this by preserving FP32's dynamic range and allowing reduced-precision training with no hyperparameter changes, while halving memory footprint and bandwidth versus FP32.
1 bit determining the sign of the value: 0 = positive, 1 = negative.
8 bits with a bias of 127 (range Emin = −126, Emax = 127) — identical to FP32, giving BF16 FP32's dynamic range.
7 explicitly stored bits plus 1 implicit leading bit (8 bits of precision total), versus 23+1 in FP32 — the source of BF16's reduced precision.
Only 7 mantissa bits (2–3 decimal digits) increase rounding errors; summing many small values can lose accuracy.
BF16 and FP16 share the same size (16 bits) but differ in bit layout; code assuming FP16's range/precision (e.g. loss scaling) can be unnecessary or wrong for BF16.
Older GPUs (pre-Ampere) and CPUs without AVX-512 BF16 emulate BF16 in software, losing the performance benefit.
The "Brain Floating Point" format is created as a 16-bit representation with FP32 range for TPU accelerators.
A comprehensive study (Intel) confirming that BF16 training reaches FP32-level results with no hyperparameter changes, across multiple domains.
The third-generation Tensor Cores in the A100 run BF16/FP32 mixed-precision at the same rate as FP16, spreading BF16 beyond TPUs.
Space complexity: 2 bajty / wartość.
Format designed by Google Brain for the MXU units in TPUs; natively supported since TPU v2.
Supported by NVIDIA Tensor Cores from the Ampere architecture (A100); BF16/FP32 mixed-precision operations run at the same throughput as FP16 (312 TFLOPS on A100).
Supported by AVX-512 BF16 extensions in Intel Xeon processors.
