Fractal-64
How it works
A block (block_q2_0, 30 bytes) encodes 64 weights in an 8ร8 hierarchy: one FP16 master scale (d) and eight 4-bit sub-scales, one per 8-weight sub-block โ twice the spatial resolution of Q3_K/MXFP4 (one sub-scale per 16โ32 weights). The raw 4-bit sub-scale codes {0..15} are shifted into a positive window mult = 4 + sc โ {4..19}, which removes "dead" codes and boosts sub-scale resolution by ~31% at zero bit overhead. The weights themselves are non-linear 3-bit indices decoded through a LUT {-14,-8,-4,-1,+1,+4,+8,+14} via VPSHUFB. The key performance trick: sub-scale multiplication is fused into a single AVX2 vpmaddwd instruction (_mm256_madd_epi16(p01, vmult01)) instead of multiplying by ones, while bit unpacking uses single-cycle variable shifts (_mm256_sllv/srlv_epi32) and _MM_HINT_NTA streaming that bypasses the L3 cache. The dot_q2_0_q8_0 kernel drops from 86.5 ms to 55.8 ms per 10M blocks (+55%). Files are built by build_fractal_moe_gguf.sh (fractal64 profile from BF16 weights); the attention/SSM/output trunk stays in Q8_0.
Problem solved
The inner dimensions of MoE experts (e.g. 1856 = 29ร64 and 2688 = 42ร64 in Nemotron-3.5-30B-A3B) are not divisible by QK_K = 256, so standard sub-4-bit K-quants and I-quants in llama.cpp (Q3_K, IQ3_S, IQ3_XXS) lose alignment and fall back to Q4_0 (4.50 bpw). Fractal-64 splits those tensors into blocks of 64 (with zero fallback tensors), reaching ~3.75 bpw while preserving quality (100/100 on the Nemotron-3.5 Quality Gate) and cutting GGUF size and RAM use, enabling large-MoE inference on memory-constrained CPUs.
Components
Block layout: ggml_half d (FP16 super-block scale, 2 B), scales[4] (eight 4-bit sub-scales, 4 B), qs[16] (lower 2 bits of 64 indices), qh[8] (upper 1 bit of 64 indices). sizeof == 30 bytes โ ~3.75 bits per weight. GGML type id: 40.
A 64-weight super-block split into 8 sub-blocks of 8 weights, with one 4-bit sub-scale per sub-block. It breaks the "QK_K = 256 geometry trap" (1856 = 29ร64, 2688 = 42ร64) and doubles the spatial resolution of local-outlier capture versus Q3_K/MXFP4.
The raw 4-bit sub-scale code sc โ {0..15} is shifted to mult = 4 + sc โ {4..19} to eliminate the dead codes {0..3} and raise effective sub-scale resolution by ~31% at zero bit cost.
Official
Weights are 3-bit non-linear indices decoded via the table {-14,-8,-4,-1,+1,+4,+8,+14} through VPSHUFB. The dot_q2_0_q8_0 dot-product kernel fuses sub-scale multiplication into a single AVX2 vpmaddwd instruction and uses variable shifts _mm256_sllv/srlv_epi32 โ a +55% speedup (86.5 ms โ 55.8 ms per 10M blocks).
Implementation
Fractal-64 files use the custom GGML_TYPE_Q2_0 (id 40) and deepseek4/Stockfish-PVS kernels. Unpatched llama.cpp, Ollama and LM Studio reject them with "unknown tensor type 40".
The fractal64 profile is designed to transcode from BF16 master weights. Building from an already-quantized Q4_0 (as the mxfp4 profile does) does not exploit the full information of the original.
Evolution
Block-64 Q2_0 (3.75 bpw) for the "cold" routed-MoE experts in NVIDIA-Nemotron-3.5-30B-A3B; GGUF 17.59 โ 14.35 GiB, 100/100 Quality Gate.
Transcoding dense MLA and indexer tensors (attn_q_b, attn_output_a/b, compressors) to block_q2_0; GGUF 76.87 โ 56.56 GiB, fitting a 284B MoE in 64 GB RAM.
Hyperparameters (configurable axes)
Weights per super-block. Fixed at 64 in Fractal-64 (8 sub-blocks of 8 weights).
Effective bits per weight from 30 bytes per 64-weight block.
Number of weights per 4-bit sub-scale.
GQA attention layers, ssm_out and output.weight stay in Q8_0, outside Fractal-64 quantization.
Computational complexity
Space complexity: ~3.75 bpw (30 B / 64 weights).
Compute bottleneck
CPU decode is memory-bandwidth bound โ the format cuts the bytes of weights streamed per token (~3.75 bpw) and isolates the L3 cache via _MM_HINT_NTA, which directly lifts decode throughput.
Execution paradigm
The mode refers to weight-block encode/decode, not to model-level sparsity.
Fractal-64 is a weight-encoding format (a dense block), not a routing mechanism. It is, however, applied primarily to routed-MoE expert tensors and to MLA tensors.
Hardware requirements
The format is built for CPU inference: the dot_q2_0_q8_0 kernel uses AVX2/AVX-512 (variable shifts _mm256_sllv/srlv_epi32, sub-scale fusion in vpmaddwd) and _MM_HINT_NTA cache isolation. The block-64 geometry fits CPU L1D/L2 caches.
Fractal-64 is a CPU-first format (3.75 bpw). GPU acceleration in the project targets mainly the sibling Fractal-MXFP4 format (native Vulkan shaders), not Fractal-64.