Inference

MXFP8

2023ActivePublished: 8 September 2026Updated: 8 September 2026Published

Key innovation

Replaces the single per-tensor FP8 scale with a shared power-of-two exponent (E8M0) over blocks of 32 elements, capturing local magnitude variation and enabling 8-bit training of large models with no recipe changes.

How it works

A tensor is split into blocks of 32 consecutive elements. Each block gets one shared scaling factor in E8M0 format (an 8-bit power-of-two exponent), typically derived from the block's maximum absolute value. Elements are then scaled and stored in 8-bit FP8: E4M3 (4 exponent bits, 3 mantissa bits, range ~±448) is typically used for the forward pass, while E5M2 (5 exponent bits, 2 mantissa bits, wider range) is used for gradients in the backward pass. Matrix multiplications run on Tensor Cores that natively multiply FP8 values while accounting for the block scales, with accumulation in higher precision. The scale-conversion (rounding) algorithm materially affects how closely results match BF16.

Problem solved

FP8 quantization with a single per-tensor factor loses accuracy when a tensor spans a wide range of magnitudes (e.g. outliers in activations or gradients). MXFP8 addresses this by giving each 32-element block its own scaling factor, allowing models to be trained and served in 8 bits without quality loss and without modifying the training recipe.

Components

Microscaling block (32 elements)Grouping unit for shared scaling.

A group of 32 consecutive tensor values that share a single scaling factor.

Shared scale exponent (E8M0)Scales the block value range at low hardware cost.

8-bit exponent-only value representing the per-block scaling factor as a power of two.

FP8 elements (E4M3 / E5M2)Representation of individual elements after scaling.

The 8-bit values: E4M3 for higher precision (forward), E5M2 for wider range (gradients).

E4M34 exponent bits, 3 mantissa bits; range ~±448.

E5M25 exponent bits, 2 mantissa bits; wider range (~±57344).

Implementation

Reference implementations

NVIDIA TransformerEngine

Python / C++ / CUDA · NVIDIA

Official

Microsoft microxcaling (MX)

Python · Microsoft

Official

Implementation pitfalls

Scale conversion and rounding algorithmHigh

How the E8M0 factor is computed and rounded materially affects accuracy; a poor algorithm diverges from BF16.

Fix:Use validated conversion recipes (e.g. MXFP8-E4M3 with appropriate rounding) proven in training.

Blackwell hardware requirementMedium

Full native acceleration is available only on Blackwell Tensor Cores; older GPUs (Hopper, Ada) require emulation.

Fix:Check platform support (e.g. is_mxfp8_available) before enabling MXFP8.

Scaling under transposesMedium

32-element blocks are directional, so transposing a tensor requires recomputing scales (often two representations).

Fix:Use libraries (Transformer Engine) that handle MXFP8 transposes.

Evolution

Original paper · 2023 · arXiv 2023 (OCP Microscaling Formats) · Bita Darvish Rouhani

Microscaling Data Formats for Deep Learning

Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall

2023

OCP MX v1.0 specification and first sub-8-bit LLM training

Inflection point

The OCP consortium (AMD, Arm, Intel, Meta, Microsoft, NVIDIA, Qualcomm) publishes the MX standard defining MXFP8; the originating paper shows the first generative LLM training with sub-8-bit weights, activations and gradients.

Microscaling Data Formats for Deep Learning (paper)

2024

Native MXFP8 support in NVIDIA Blackwell Tensor Cores

Inflection point

NVIDIA Blackwell introduces hardware support for microscaling (MXFP8, MXFP6, MXFP4) directly in the Tensor Cores, unlike emulation on earlier GPUs.

2025

MXFP8 LLM pre-training recipes matching BF16

NVIDIA publishes recipes showing that MXFP8-E4M3 with a suitable conversion algorithm matches BF16 training for models up to 8B parameters on datasets up to 15T tokens.

Recipes for Pre-training LLMs with MXFP8 (paper)

Hyperparameters (configurable axes)

Block sizeCritical

Number of elements sharing one scaling factor. Fixed at 32 in the MX standard.

32Value defined in OCP MX v1.0.

Element formatCritical

FP8 variant used for elements: E4M3 or E5M2.

E4M3Typically forward pass.

E5M2Typically gradients (backward).

Scale formatHigh

Format of the shared scaling factor; in MX this is E8M0 (powers of two).

E8M08-bit exponent, no mantissa.

Computational complexity

Space complexity: ~8 bitów/element + 8 bitów na 32 elementy (E8M0).

Execution paradigm

Primary mode

Dense

MXFP8 is a numeric format used in dense matrix multiplications on Tensor Cores; scaling operates at block level and accumulation happens in higher precision.