Robots Atlas>ROBOTS ATLAS
Inference

FP4

2023ActivePublished: 2 September 2026Updated: 2 September 2026Published
Key innovation
Compressing neural network weights and activations to 4-bit floating-point formats (E2M1) with a block-wise scaling factor, enabling inference and training at roughly half the memory and higher throughput of FP8 while retaining comparable quality.
Category
Inference
Abstraction level
Pattern
Operation level
Architecture blockModelInferenceTraining
Use cases
Large language model inferenceServing MoE models (e.g., expert weights in MXFP4)Pretraining large language models in 4-bit (NVFP4)Reducing GPU memory footprint and compute costHigh-token-budget reasoning models

How it works

Tensor values are cast to the 4-bit E2M1 format, which represents only a handful of levels. To preserve dynamic range, the tensor is split into small blocks (16 values in NVFP4, 32 in MXFP4); each block gets a scaling factor (E4M3 in NVFP4, E8M0 in MXFP4) that brings the block into FP4-representable range. NVFP4 adds a second, global FP32 scale over the whole tensor (two-level scaling). Matrix multiplications run on Tensor Cores in FP4, while accumulation happens in higher precision. FP4 training adds stabilizing techniques: Random Hadamard transforms to tame outliers, two-dimensional quantization, stochastic rounding of gradients, and keeping selected layers in higher precision.

Problem solved

Large language models are bottlenecked by memory bandwidth and capacity. 8- and 16-bit formats still demand significant memory and bandwidth. FP4 halves weight size versus FP8 and doubles matrix-math throughput, lowering the cost and energy of inference and of frontier-model training.

Components

E2M1 formatRepresenting weights and activations in 4 bits

4-bit floating-point format: 1 sign bit, 2 exponent bits, 1 mantissa bit. Represents a handful of discrete values in the range of roughly -6 to +6 (0, 0.5, 1, 1.5, 2, 3, 4, 6 and their negatives). The shared bit layout for both NVFP4 and MXFP4.

NVFP4FP4 variant optimized for Blackwell Tensor Cores

NVIDIA’s FP4 format with two-level scaling: a 16-value micro-block with an E4M3 FP8 scale (allowing non-power-of-two scales with fractional precision) plus an additional per-tensor FP32 scale. Its smaller block than MXFP4 gives finer scaling and lower quantization error; about 1% or less accuracy degradation versus FP8 on DeepSeek-R1.

MXFP4Open, industry-standard FP4 variant

An FP4 format compliant with the Open Compute Project Microscaling (MX) standard: 32-value blocks with a shared E8M0 scale (power-of-two only, 8-bit exponent). Simpler in hardware than NVFP4 but with coarser scaling. Used, among others, in OpenAI’s open-weight gpt-oss models.

Block scaling (microscaling)Preserving dynamic range at 4 bits

The mechanism giving each small block of values its own scaling factor so the block’s distribution fits FP4’s narrow range. The block size and scale format (E4M3 vs E8M0) set the trade-off between accuracy and hardware complexity.

Implementation

Reference implementations
NVIDIA TensorRT Model Optimizer
Python · NVIDIA
Official
NVIDIA Transformer Engine
Python / C++ / CUDA · NVIDIA
Official
Implementation pitfalls
Extremely narrow 4-bit rangeCritical

4 bits represent only a handful of levels, so without block scaling there is large quantization error, overflow, or values collapsing to zero.

Fix:Use block scaling (microscaling): NVFP4 with 16-value blocks and E4M3 scale, or MXFP4 with 32-value blocks and E8M0 scale.
Outliers and sensitive layersHigh

Activation outliers and precision-sensitive layers (normalization, softmax, gradients) can lose significant accuracy in FP4, especially during training.

Fix:Random Hadamard transforms to tame outliers, stochastic rounding of gradients, and keeping selected layers in higher precision.
Dependence on Blackwell hardwareMedium

Full FP4 acceleration requires fifth-generation Tensor Cores (NVIDIA Blackwell); on older hardware the benefits are limited or software-emulated.

Fix:Target Blackwell GPUs for FP4 deployments; consider FP8 on older hardware.

Evolution

Original paper · 2023 · EMNLP 2023 · Shih-yang Liu
LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, Kwang-Ting Cheng
2023
LLM-FP4: post-training 4-bit float quantization of LLMs
Inflection point

The first method to quantize LLM weights and activations to 4-bit floating-point post-training, using per-channel activation quantization.

2023
OCP Microscaling (MX) standard — MXFP4 defined
Inflection point

The Open Compute Project standardizes the MX formats (including MXFP4) combining the narrow E2M1 float with a block-wise E8M0 scale (32-value blocks).

2025
Hardware FP4 in NVIDIA Blackwell and the NVFP4 format
Inflection point

Fifth-generation Blackwell Tensor Cores natively support FP4; NVIDIA introduces NVFP4 (16-value blocks, E4M3 + FP32 scale) for accurate low-precision inference.

2025
OpenAI releases gpt-oss with MoE weights in MXFP4

The open-weight gpt-oss models ship with MoE expert weights quantized to MXFP4, popularizing FP4 beyond the NVIDIA ecosystem.

2025
LLM pretraining in NVFP4 (12B on 10T tokens)
Inflection point

NVIDIA demonstrates pretraining a 12B model on 10T tokens in NVFP4 — the longest publicly documented 4-bit training run — with quality comparable to FP8.

Hyperparameters (configurable axes)

Format variant (NVFP4 / MXFP4)Critical

Choice of FP4 variant, determining block size and scale format.

NVFP4Block 16, E4M3 scale + per-tensor FP32
MXFP4Block 32, E8M0 scale (power of two)
Scaling block sizeHigh

Number of values sharing one scaling factor. Smaller block = finer scaling and lower error.

16NVFP4
32MXFP4
Scale factor formatHigh

Numeric format of the block’s shared scale.

E4M3 (FP8)NVFP4 — non-power-of-two scales
E8M0MXFP4 — power-of-two only
Application stageMedium

Whether FP4 is applied post-training (inference) or during pretraining.

post-training (inferencja)Most common use
pretraining (NVFP4)12B model trained on 10T tokens

Execution paradigm

Primary mode
Dense

FP4 is a numeric format applied in dense matrix multiplications on Tensor Cores; accumulation happens in higher precision.

Parallelism

Parallelism level
Fully parallel

FP4 quantization and matmuls are fully parallel on GPU Tensor Cores, in both training and inference.

Scope
TrainingInferenceAcross devices

Hardware requirements

Primary

FP4 (NVFP4, MXFP4) is natively supported by fifth-generation Tensor Cores in NVIDIA Blackwell, giving about 2x throughput over FP8 and about 1.8x lower memory usage.

Limited

4-bit formats are not the primary native tensor mode on non-NVIDIA accelerators; support is limited or software-based.