Tensor values are cast to the 4-bit E2M1 format, which represents only a handful of levels. To preserve dynamic range, the tensor is split into small blocks (16 values in NVFP4, 32 in MXFP4); each block gets a scaling factor (E4M3 in NVFP4, E8M0 in MXFP4) that brings the block into FP4-representable range. NVFP4 adds a second, global FP32 scale over the whole tensor (two-level scaling). Matrix multiplications run on Tensor Cores in FP4, while accumulation happens in higher precision. FP4 training adds stabilizing techniques: Random Hadamard transforms to tame outliers, two-dimensional quantization, stochastic rounding of gradients, and keeping selected layers in higher precision.
Large language models are bottlenecked by memory bandwidth and capacity. 8- and 16-bit formats still demand significant memory and bandwidth. FP4 halves weight size versus FP8 and doubles matrix-math throughput, lowering the cost and energy of inference and of frontier-model training.
4-bit floating-point format: 1 sign bit, 2 exponent bits, 1 mantissa bit. Represents a handful of discrete values in the range of roughly -6 to +6 (0, 0.5, 1, 1.5, 2, 3, 4, 6 and their negatives). The shared bit layout for both NVFP4 and MXFP4.
NVIDIA’s FP4 format with two-level scaling: a 16-value micro-block with an E4M3 FP8 scale (allowing non-power-of-two scales with fractional precision) plus an additional per-tensor FP32 scale. Its smaller block than MXFP4 gives finer scaling and lower quantization error; about 1% or less accuracy degradation versus FP8 on DeepSeek-R1.
An FP4 format compliant with the Open Compute Project Microscaling (MX) standard: 32-value blocks with a shared E8M0 scale (power-of-two only, 8-bit exponent). Simpler in hardware than NVFP4 but with coarser scaling. Used, among others, in OpenAI’s open-weight gpt-oss models.
The mechanism giving each small block of values its own scaling factor so the block’s distribution fits FP4’s narrow range. The block size and scale format (E4M3 vs E8M0) set the trade-off between accuracy and hardware complexity.
4 bits represent only a handful of levels, so without block scaling there is large quantization error, overflow, or values collapsing to zero.
Activation outliers and precision-sensitive layers (normalization, softmax, gradients) can lose significant accuracy in FP4, especially during training.
Full FP4 acceleration requires fifth-generation Tensor Cores (NVIDIA Blackwell); on older hardware the benefits are limited or software-emulated.
The first method to quantize LLM weights and activations to 4-bit floating-point post-training, using per-channel activation quantization.
The Open Compute Project standardizes the MX formats (including MXFP4) combining the narrow E2M1 float with a block-wise E8M0 scale (32-value blocks).
Fifth-generation Blackwell Tensor Cores natively support FP4; NVIDIA introduces NVFP4 (16-value blocks, E4M3 + FP32 scale) for accurate low-precision inference.
The open-weight gpt-oss models ship with MoE expert weights quantized to MXFP4, popularizing FP4 beyond the NVIDIA ecosystem.
NVIDIA demonstrates pretraining a 12B model on 10T tokens in NVFP4 — the longest publicly documented 4-bit training run — with quality comparable to FP8.
Choice of FP4 variant, determining block size and scale format.
Number of values sharing one scaling factor. Smaller block = finer scaling and lower error.
Numeric format of the block’s shared scale.
Whether FP4 is applied post-training (inference) or during pretraining.
FP4 is a numeric format applied in dense matrix multiplications on Tensor Cores; accumulation happens in higher precision.
FP4 quantization and matmuls are fully parallel on GPU Tensor Cores, in both training and inference.
FP4 (NVFP4, MXFP4) is natively supported by fifth-generation Tensor Cores in NVIDIA Blackwell, giving about 2x throughput over FP8 and about 1.8x lower memory usage.
4-bit formats are not the primary native tensor mode on non-NVIDIA accelerators; support is limited or software-based.
