SwiGLU-H2
How it works
1) For each expert e and intermediate channel j, the paired ffn_gate_exps and ffn_up_exps tensors are dequantized (in parallel across OpenMP threads). 2) An analytical second-order importance I(e,j) = ‖W_gate^(e)[j,:]‖² · ‖W_up^(e)[j,:]‖² is computed, approximating the Hessian diagonal of the quantization error under isotropic unit-variance inputs. 3) The I(e,j) vector is fed to quantize_row_iq1_s as the importance vector, so bits go to the channels that most affect the SwiGLU output. 4) Highly sensitive boundary layers (e.g. {0,1,26,28,41,42}) are kept in IQ2_XXS and sensitive mid layers in IQ1_M, while the bulk of experts drop to IQ1_S (1.5625 bpw). The whole thing is data-free (no calibration) and computed once, at checkpoint transcoding time.
Problem solved
Naive uncalibrated IQ1_S (1.5625 bpw) quantization across all 256 MoE expert layers collapses SwiGLU gating accuracy because high-norm activation channels in W_gate ⊙ W_up are quantized with uniform MSE weight. SwiGLU-H2 fixes this with analytical, second-order channel-importance weighting — without needing any calibration dataset.
Components
I(e,j) = ‖W_gate^(e)[j,:]‖² · ‖W_up^(e)[j,:]‖². The product of the squared row norms of the paired Gate and Up slices approximates the channel's expected second moment (the Hessian diagonal of the error) under isotropic inputs. Computed analytically, with no calibration data.
Channel output: h = SiLU(W_gate·x) · (W_up·x), feeding ffn_down_exps. SwiGLU-H2 weights the down-projection rows by the I(e,j) importance of the paired Gate/Up blocks.
The standard IQ1_S (1.5625 bpw) quantizer from llama.cpp/ggml, to which SwiGLU-H2 supplies an analytical importance vector instead of uniform MSE weight.
Official
Selected boundary (early/late) layers are kept in IQ2_XXS (2.06 bpw) or IQ1_M (1.75 bpw) while the bulk of experts drop to IQ1_S.
Official
Implementation
Uncalibrated IQ1_S with uniform error weight collapses SwiGLU gating — high-norm channels lose precision.
Dropping all layers to IQ1_S without anchoring sensitive routing layers degrades quality.
Hyperparameters (configurable axes)
Quantization format for the bulk experts. SwiGLU-H2 is tuned for IQ1_S (1.5625 bpw), with higher precision for sensitive layers.
Set of layers kept at higher precision to protect routing and quality (e.g. early and late layers).
The closed-form derivation assumes isotropic unit-variance inputs; this reduces importance to the product of squared row norms.
Execution paradigm
The paradigm reflects the deployment context (sparse MoE); SwiGLU-H2 itself is a quantization scheme operating on SwiGLU expert blocks.
SwiGLU-H2 quantizes the FFN expert layers of a sparse MoE, where only top-k experts per token are activated by the router.
Parallelism
Computing the I(e,j) importance vector is fully parallel across experts and channels (OpenMP) since each channel is independent. The quantization itself runs once, offline, at transcoding time.
Hardware requirements
Designed for pure-CPU x86_64 MoE inference (AVX2 / AVX-512); the quantized formats (IQ1_S) are decoded with AVX2/AVX-512 kernels. Also runs on ARM64 (NEON/SVE).