Tensor values (weights, activations, gradients) are converted from FP32/FP16 into one of the FP8 formats: E4M3 for the forward pass, E5M2 for gradients in the backward pass. Because 8 bits give a very narrow range of representable values, scaling factors are applied before conversion to shift the tensor's value distribution into FP8's representable range. NVIDIA's Transformer Engine uses "delayed scaling" — deriving the factor from historical maximum absolute values (amax) of previous iterations. Scaling can be per-tensor or fine-grained (per-block/per-tile). The matrix multiplications run on Tensor Cores in FP8, while accumulation happens in higher precision (FP32), after which the result is re-scaled. Master weights and accumulators typically stay in higher precision for stability.
Training and inference of large language models are constrained by GPU memory capacity and compute throughput. 16-bit formats (FP16/BF16) were the standard but take twice the memory and bandwidth of 8-bit formats. INT8 integer quantization reduces size but its narrow, uniform range makes stable training difficult. FP8 resolves this trade-off by delivering 8-bit size while retaining the dynamic range of a floating-point format.
8-bit floating-point format: 1 sign bit, 4 exponent bits, 3 mantissa bits. Maximum magnitude ±448, no infinity representation. Higher precision, narrower range — used in the forward pass.
8-bit floating-point format: 1 sign bit, 5 exponent bits, 2 mantissa bits. Maximum magnitude ±57344, follows IEEE-754 for special values (NaN, infinity). Wider dynamic range, lower precision — used for gradients.
A multiplier shifting a tensor's value distribution into FP8's representable range. Can be per-tensor or fine-grained (per-block/per-tile); the Transformer Engine uses delayed scaling based on amax history.
The narrow 8-bit range means that without proper scaling, values exceed the range (overflow to infinity/NaN) or collapse to zero (underflow).
Some layers (e.g., normalization, softmax, accumulators) are sensitive to low precision and must remain in FP16/FP32.
NVIDIA, Arm, and Intel propose a joint standard for two FP8 formats for deep learning.
H100 introduces FP8-capable Tensor Cores (2000 TFLOPS, 4000 with sparsity) and the Transformer Engine automating FP8/16-bit switching.
The FP8 formats are adopted as the open OCP industry standard.
DeepSeek-V3 (671B parameters) uses a fine-grained FP8 mixed-precision training framework, demonstrating FP8 for frontier-scale model training.
Format selection: E4M3 (more precision) or E5M2 (wider range).
The level at which the scaling factor is computed.
How the scaling factor is determined over time.
FP8 is a numeric format applied in dense matrix multiplications on Tensor Cores; accumulation happens in higher precision.
FP8 quantization and matmuls are fully parallel on GPU Tensor Cores, in both training and inference.
FP8 is natively supported by Tensor Cores in NVIDIA Hopper, Ada Lovelace, and Blackwell architectures, giving 2x throughput over FP16/BF16.
8-bit formats are being explored on non-NVIDIA accelerators, but FP8 is not their primary native tensor mode.
