NVFP4
How it works
The tensor is first normalized by a single global FP32 factor across the whole tensor. It is then split into blocks of 16 consecutive elements; each block gets its own scaling factor in the FP8 E4M3 format (4-bit exponent, 3-bit mantissa — a fractional scale, not only powers of two). Elements are scaled and stored in the 4-bit FP4 E2M1 format, which encodes 16 values (0, ±0.5, ±1, ±1.5, ±2, ±3, ±4, ±6). Matrix multiplications run on fifth-generation NVIDIA Blackwell Tensor Cores, which natively multiply FP4 values, apply the block and tensor scales, and accumulate in higher precision. For training (NVIDIA's paper) the format is complemented by techniques such as random Hadamard transforms for outliers, two-dimensional quantization, stochastic rounding of gradients, and keeping sensitive layers in higher precision.
Problem solved
4-bit quantization with a single per-tensor factor, or with a coarse power-of-two scale (as in MXFP4), loses too much accuracy at such a low bit-width. NVFP4 reduces this loss via smaller blocks (16 elements) and a more precise, fractional FP8 E4M3 scale backed by a global FP32 normalization, delivering low memory footprint with accuracy close to FP8.
Components
A group of 16 consecutive tensor values that share a single FP8 E4M3 scaling factor.
An 8-bit floating-point factor (4 exponent bits, 3 mantissa bits) that scales a 16-element block; a fractional scale, not only powers of two.
A single FP32 factor for the whole tensor, normalizing its overall range before block scaling.
A 4-bit floating-point value: 1 sign bit, 2 exponent bits, 1 mantissa bit; encodes 16 levels, maximum ±6.
Implementation
NVFP4 uses a smaller block (16) with a per-block FP8 E4M3 scale plus a global FP32 scale. This gives higher accuracy than MXFP4 but a larger scale-bit overhead (~4.5 bits/element) and a more complex quantization/dequantization path.
FP4 E2M1 encodes only 16 values, so outliers strongly degrade quality, especially in training. NVIDIA's paper applies random Hadamard transforms, 2D quantization, stochastic rounding of gradients, and keeping sensitive layers in higher precision.
Evolution
NVIDIA presents NVFP4 as a 4-bit format with two-level scaling (FP8 E4M3 scale per 16-element block + a global FP32 scale) for efficient and accurate low-precision inference on the Blackwell architecture.
NVIDIA's paper "Pretraining Large Language Models with NVFP4" demonstrates training a 12B model on 10T tokens in 4-bit precision, with quality comparable to an FP8 baseline, using random Hadamard transforms, 2D quantization, and stochastic rounding.
Hyperparameters (configurable axes)
Number of elements sharing one scaling factor. In NVFP4 it is 16 (half of MXFP4).
Single-element format: FP4 E2M1 (2 exponent bits, 1 mantissa bit).
Format of the per-block scale; in NVFP4 it is FP8 E4M3 (fractional) instead of MXFP4's E8M0.
Format of the global per-tensor factor; in NVFP4 it is FP32.
Computational complexity
Space complexity: ~4 bity/element + skala FP8 (E4M3) na 16 elementów + jeden FP32 na tensor.
Compute bottleneck
NVFP4 aims to relieve memory bandwidth and capacity via 4-bit weight storage. The bottleneck shifts toward Tensor Core arithmetic throughput and — on hardware without native FP4 support — toward the cost of dequantizing blocks and applying the two scaling levels.
Execution paradigm
NVFP4 is a numeric format used in matrix multiplications on Tensor Cores; scaling operates at the 16-element block and whole-tensor level, and accumulation happens in higher precision.
Parallelism
Block scaling and NVFP4 matmuls are fully parallel on GPU Tensor Cores; the format is used for both inference and training.
Hardware requirements
Fifth-generation Tensor Cores in the NVIDIA Blackwell architecture natively support NVFP4 matmuls with two-level scaling (16-element block + tensor).
The format can be emulated in software, but without hardware acceleration or the performance benefits.
CPUs lack native FP4 matmuls; use is limited to emulation or dequantization to higher precision.