Training

BF16

2017ActivePublished: 2 September 2026Updated: 2 September 2026Published

Key innovation

Truncating the FP32 mantissa to 7 bits while keeping the full 8-bit exponent yields a 16-bit format with FP32's dynamic range — eliminating the loss scaling required by FP16.

How it works

A BF16 number encodes a value as sign × 1.mantissa × 2^(exponent − 127), identically to FP32 but with the mantissa truncated to 7 bits. In mixed-precision training, weights, activations, and gradients are stored and multiplied in BF16, while the accumulation of products (e.g. in matrix multiplication on Tensor Cores / MXUs) is done in FP32, and a FP32 "master copy" of the weights is often kept for updates. Reducing mantissa precision increases rounding error, but the wide exponent range prevents overflow, so networks reach accuracy close to FP32 in the same number of iterations.

Problem solved

FP16 (half precision) has only 5 exponent bits, so its narrow dynamic range causes overflows and gradient underflow during training, forcing techniques such as loss scaling. BF16 solves this by preserving FP32's dynamic range and allowing reduced-precision training with no hyperparameter changes, while halving memory footprint and bandwidth versus FP32.

Components

Sign bitSign of the value

1 bit determining the sign of the value: 0 = positive, 1 = negative.

Exponent fieldEncodes the order of magnitude (scale) of the value

8 bits with a bias of 127 (range Emin = −126, Emax = 127) — identical to FP32, giving BF16 FP32's dynamic range.

Mantissa (significand)Encodes the significant digits (precision)

7 explicitly stored bits plus 1 implicit leading bit (8 bits of precision total), versus 23+1 in FP32 — the source of BF16's reduced precision.

Implementation

Reference implementations

ml_dtypes (bfloat16 for NumPy)

Python / C++ · Google / JAX

Official

PyTorch torch.bfloat16

Python / C++ · PyTorch Foundation

Official

Implementation pitfalls

Reduced mantissa precisionMedium

Only 7 mantissa bits (2–3 decimal digits) increase rounding errors; summing many small values can lose accuracy.

Fix:Accumulate products and reductions in FP32; keep an FP32 master copy of the weights.

Confusing BF16 with FP16Low

BF16 and FP16 share the same size (16 bits) but differ in bit layout; code assuming FP16's range/precision (e.g. loss scaling) can be unnecessary or wrong for BF16.

Fix:Treat the formats separately; BF16 usually needs no loss scaling.

Missing native hardware supportMedium

Older GPUs (pre-Ampere) and CPUs without AVX-512 BF16 emulate BF16 in software, losing the performance benefit.

Fix:Verify native BF16 support on the target hardware before choosing the format.

Evolution

Original paper · 2019 · arXiv (preprint), 2019 · Dhiraj Kalamkar

A Study of BFLOAT16 for Deep Learning Training

Dhiraj Kalamkar, Dheevatsa Mudigere, Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, Jiyan Yang, Jongsoo Park, Alexander Heinecke, Evangelos Georganas, Sudarshan Srinivasan, Abhisek Kundu, Misha Smelyanskiy, Bharat Kaul, Pradeep Dubey

2017

Google Brain introduces bfloat16 in TPU v2

Inflection point

The "Brain Floating Point" format is created as a 16-bit representation with FP32 range for TPU accelerators.

2019

Publication of "A Study of BFLOAT16 for Deep Learning Training"

A comprehensive study (Intel) confirming that BF16 training reaches FP32-level results with no hyperparameter changes, across multiple domains.

A Study of BFLOAT16 for Deep Learning Training (paper)

2020

NVIDIA adds hardware BF16 support in Tensor Cores (Ampere, A100)

Inflection point

The third-generation Tensor Cores in the A100 run BF16/FP32 mixed-precision at the same rate as FP16, spreading BF16 beyond TPUs.

Computational complexity

Space complexity: 2 bajty / wartość.

Hardware requirements

Primary

Format designed by Google Brain for the MXU units in TPUs; natively supported since TPU v2.

Primary

Supported by NVIDIA Tensor Cores from the Ampere architecture (A100); BF16/FP32 mixed-precision operations run at the same throughput as FP16 (312 TFLOPS on A100).

Good fit

Supported by AVX-512 BF16 extensions in Intel Xeon processors.