Robots Atlas>ROBOTS ATLAS
Hardware

Numerical precision

1985ActivePublished: 19 November 2026Updated: 19 November 2026Published
Key innovation
Turned the numeric data format into a deliberate, configurable system-level parameter — instead of defaulting to FP32 — unlocking training and inference at 16, 8 and even 4 bits per value.
Category
Hardware
Abstraction level
Pojęcie
Operation level
TrainingInferenceModel
Use cases
Mixed-precision training (FP16/BF16 + FP32 master weights)Post-training quantization (PTQ) to INT8/FP8 for cheaper inferenceFP8 training and inference on NVIDIA Hopper (Transformer Engine)FP4/NVFP4 on NVIDIA Blackwell for trillion-parameter-scale modelsKV-cache quantization for long-context inferenceEdge AI deployment in INT8/INT4 on memory-constrained devices

How it works

A floating-point number encodes a sign, an exponent and a mantissa: more exponent bits mean a wider dynamic range, more mantissa bits mean higher precision. BF16 keeps FP32's range (8 exponent bits) with only 7 mantissa bits, so it usually needs no loss scaling, whereas FP16 (5 exponent bits) needs loss scaling to avoid gradient underflow. FP8 comes as E4M3 (more precision) and E5M2 (more range); 4-bit formats use block microscaling, where a block of values shares a common scale factor. In mixed-precision training the matmuls run in FP16/BF16/FP8 on Tensor Cores while master weights and accumulation stay in FP32. In inference, quantization maps weights and activations to INT8/INT4/FP8/FP4 using scale factors (per-tensor, per-channel or block). Halving the precision typically doubles Tensor Core throughput and halves data movement.

Problem solved

FP32 is expensive in memory, bandwidth and energy, which caps model size and the speed of training and inference. Lower precision cuts those costs dramatically but risks accuracy loss and numerical instability. Numerical precision is the central knob for trading compute cost against model quality.

Components

Format anatomy (sign / exponent / mantissa)Determines how bits split between dynamic range and precision

Splitting a number's bits into sign, exponent and mantissa decides the range and accuracy of the format (e.g. FP32 1-8-23, FP16 1-5-10, BF16 1-8-7).

Range–precision trade-offBalances dynamic range against accuracy

At a fixed bit count, more exponent bits widen the range at the cost of precision — this is what separates FP16 from BF16.

Scaling (loss scaling / per-tensor / block microscaling)Fits values into the available range, preventing overflow and underflow

Scale factors (global, per-channel or block-wise as in MXFP8/NVFP4) keep values inside the format's representable range.

Accumulation precisionAccumulates products in higher precision for stability

Even when multiplications run in low precision, accumulators usually operate in FP32 to limit accumulated rounding error.

Implementation

Implementation pitfalls
FP16 overflow/underflow without loss scalingHigh

FP16's narrow range flushes small gradients to zero; without loss scaling training diverges.

Fix:Use loss scaling (static or dynamic) or switch to BF16, which has FP32's range.
Quantization error on activation outliersHigh

A few large activation values stretch the scale and degrade the accuracy of the remaining values.

Fix:Per-channel/block quantization, smoothing (e.g. SmoothQuant) or keeping outlier channels in higher precision.
Accumulation precision too lowMedium

Accumulating sums in low precision compounds rounding error and corrupts the matmul result.

Fix:Keep accumulators in FP32 even with FP16/FP8 inputs.
Non-determinism and reproducibility issuesMedium

Different precisions and reduction orders yield different bit-for-bit results across runs and devices.

Fix:Pin deterministic kernels, fix accumulation precision and report accuracy with a tolerance margin.

Evolution

Original paper · 2017 · ICLR 2018 · Paulius Micikevicius et al.
Mixed Precision Training
Paulius Micikevicius et al.
1985
IEEE 754 standardizes FP32 and FP64 floating point
Inflection point

Establishes the ubiquitous floating-point standard that became the default precision for scientific computing and early machine learning.

2008
IEEE 754-2008 adds half precision (FP16 / binary16)

Formalizes the 16-bit floating-point format later central to deep-learning acceleration.

2017
Mixed-precision training (FP16 + FP32 master weights)
Inflection point

Micikevicius et al. and NVIDIA Volta Tensor Cores make low precision mainstream for large-model training.

2019
bfloat16 popularized by Google TPUs

BF16 combines FP32's range with a 16-bit footprint, simplifying training without loss scaling.

2020
NVIDIA Ampere introduces TF32

TF32 keeps FP32's range with FP16's mantissa precision, speeding up matmuls with no code changes.

2022
FP8 (E4M3/E5M2) and the Transformer Engine on NVIDIA Hopper
Inflection point

The FP8 format spec and Hopper's hardware support enable 8-bit training and inference.

2023
OCP Microscaling (MX) Specification v1.0

The Open Compute Project standardizes block formats MXFP8/MXFP6/MXFP4, where a block of values shares a common scale.

2025
NVIDIA Blackwell adds FP4/NVFP4
Inflection point

Hardware support for 4-bit formats pushes the model-scale frontier while preserving quality through microscaling.

Hyperparameters (configurable axes)

Bit widthCritical

Total number of bits per value (32, 16, 8, 4).

Exponent bits (range)High

The number of exponent bits sets the format's dynamic range.

Mantissa bits (precision)High

The number of mantissa bits sets representation precision.

Scaling granularityHigh

Whether the scale is per-tensor, per-channel or block (microscaling).

Accumulation precisionMedium

The precision in which products are summed (usually FP32).

Computational complexity

Space complexity: O(n · b).

Compute bottleneck

Memory bandwidth and Tensor Cores

In many AI workloads the limiter is data movement and matrix-unit throughput; lower precision relieves both bottlenecks, shifting the pressure onto representation quality.

Hardware requirements

Primary

Tensor Cores are designed for low precision — throughput rises with each bit reduction (FP16/BF16, FP8 on Hopper, FP4 on Blackwell).

Good fit

TPUs natively support BF16, the format created specifically for machine learning.

Possible

CPU extensions (AVX-512 VNNI) accelerate INT8 inference, though low-precision floating point is limited.

Limited

Neuromorphic chips use their own heavily quantized representations, far from standard IEEE/OCP formats.