Robots Atlas>ROBOTS ATLAS
Inference

FP8

2022ActivePublished: 2 September 2026Updated: 2 September 2026Published
Key innovation
Introduction of two 8-bit floating-point formats (E4M3 and E5M2) that let neural networks train and run at quality matching 16-bit while halving memory footprint and doubling throughput.
Category
Inference
Abstraction level
Pattern
Operation level
Architecture blockModelTraining
Use cases
Large language model trainingLow-latency LLM inferenceGPU memory footprint reductionIncreasing compute throughput on Tensor CoresMixed-precision training

How it works

Tensor values (weights, activations, gradients) are converted from FP32/FP16 into one of the FP8 formats: E4M3 for the forward pass, E5M2 for gradients in the backward pass. Because 8 bits give a very narrow range of representable values, scaling factors are applied before conversion to shift the tensor's value distribution into FP8's representable range. NVIDIA's Transformer Engine uses "delayed scaling" — deriving the factor from historical maximum absolute values (amax) of previous iterations. Scaling can be per-tensor or fine-grained (per-block/per-tile). The matrix multiplications run on Tensor Cores in FP8, while accumulation happens in higher precision (FP32), after which the result is re-scaled. Master weights and accumulators typically stay in higher precision for stability.

Problem solved

Training and inference of large language models are constrained by GPU memory capacity and compute throughput. 16-bit formats (FP16/BF16) were the standard but take twice the memory and bandwidth of 8-bit formats. INT8 integer quantization reduces size but its narrow, uniform range makes stable training difficult. FP8 resolves this trade-off by delivering 8-bit size while retaining the dynamic range of a floating-point format.

Components

E4M3 formatWeight and activation representation (forward pass)

8-bit floating-point format: 1 sign bit, 4 exponent bits, 3 mantissa bits. Maximum magnitude ±448, no infinity representation. Higher precision, narrower range — used in the forward pass.

E5M2 formatGradient representation (backward pass)

8-bit floating-point format: 1 sign bit, 5 exponent bits, 2 mantissa bits. Maximum magnitude ±57344, follows IEEE-754 for special values (NaN, infinity). Wider dynamic range, lower precision — used for gradients.

Scaling factorKeeping values within FP8 range and numerical stability

A multiplier shifting a tensor's value distribution into FP8's representable range. Can be per-tensor or fine-grained (per-block/per-tile); the Transformer Engine uses delayed scaling based on amax history.

Implementation

Reference implementations
NVIDIA Transformer Engine
Python / C++ / CUDA · NVIDIA
Official
Implementation pitfalls
Range overflow and underflowHigh

The narrow 8-bit range means that without proper scaling, values exceed the range (overflow to infinity/NaN) or collapse to zero (underflow).

Fix:Use per-tensor or fine-grained scaling and delayed scaling based on amax history.
Accuracy loss on sensitive layersMedium

Some layers (e.g., normalization, softmax, accumulators) are sensitive to low precision and must remain in FP16/FP32.

Fix:Use a mixed-precision approach: critical operations and master weights in higher precision.

Evolution

Original paper · 2022 · arXiv 2022 · Paulius Micikevicius
FP8 Formats for Deep Learning
Paulius Micikevicius, Dusan Stosic, Neil Burgess, Marius Cornea, Pradeep Dubey, Richard Grisenthwaite, Sangwon Ha, Alexander Heinecke, Patrick Judd, John Kamalu, Naveen Mellempudi, Stuart Oberman, Mohammad Shoeybi, Michael Siu, Hao Wu
2022
Publication of the FP8 format standard (E4M3, E5M2)
Inflection point

NVIDIA, Arm, and Intel propose a joint standard for two FP8 formats for deep learning.

2022
Hardware FP8 support in NVIDIA Hopper (H100) and Transformer Engine
Inflection point

H100 introduces FP8-capable Tensor Cores (2000 TFLOPS, 4000 with sparsity) and the Transformer Engine automating FP8/16-bit switching.

2023
OCP FP8 standardization by the Open Compute Project

The FP8 formats are adopted as the open OCP industry standard.

2024
DeepSeek-V3 large-scale FP8 training
Inflection point

DeepSeek-V3 (671B parameters) uses a fine-grained FP8 mixed-precision training framework, demonstrating FP8 for frontier-scale model training.

Hyperparameters (configurable axes)

FP8 format (E4M3 / E5M2)Critical

Format selection: E4M3 (more precision) or E5M2 (wider range).

E4M3Forward pass / weights and activations
E5M2Backward pass / gradients
Scaling granularityHigh

The level at which the scaling factor is computed.

per-tensorOne factor per whole tensor
per-block / per-tileFine-grained scaling (e.g., in DeepSeek-V3)
Scaling strategyMedium

How the scaling factor is determined over time.

delayed scalingBased on amax history
current scalingBased on the current tensor

Execution paradigm

Primary mode
Dense

FP8 is a numeric format applied in dense matrix multiplications on Tensor Cores; accumulation happens in higher precision.

Parallelism

Parallelism level
Fully parallel

FP8 quantization and matmuls are fully parallel on GPU Tensor Cores, in both training and inference.

Scope
TrainingInferenceAcross devices

Hardware requirements

Primary

FP8 is natively supported by Tensor Cores in NVIDIA Hopper, Ada Lovelace, and Blackwell architectures, giving 2x throughput over FP16/BF16.

Possible

8-bit formats are being explored on non-NVIDIA accelerators, but FP8 is not their primary native tensor mode.