1) After the linear projections to Q and K, each query and key vector is normalized along the head dimension (head_dim). The original uses L2 normalization; variants use LayerNorm (ViT-22B) or RMSNorm (Qwen3). 2) The normalized Q and K are multiplied (dot product). After L2 normalization the dot product is bounded to [-1, 1]. 3) Instead of dividing by √d, the result is scaled by a learnable parameter (temperature g) that controls the sharpness of the softmax. 4) The rest of attention (softmax, multiply by V) is unchanged.
Without normalization, attention logits (Q·K dot products) can grow to very large magnitudes as models scale, causing softmax saturation: attention weights collapse toward near one-hot, near-zero-entropy distributions, gradients vanish, and training of large models diverges. QK-Norm bounds the logit range and stabilizes training.
A step that normalizes query and key vectors along the head dimension before the dot product.
Official
A learnable parameter replacing the fixed 1/√d scaling; controls the sharpness of the softmax over the normalized Q·K product.
After L2 normalization the Q·K product is bounded to [-1,1]; without the learnable parameter g the logits are too small and the softmax too flat.
Normalization must be per-head along head_dim; the choice of variant (L2 vs LayerNorm vs RMSNorm) affects stability.
QK-Norm and rotary position embeddings (RoPE) operate on the same Q/K; the order of application affects the result.
Henry et al. propose QKNorm: L2 normalization of Q and K along the head dimension plus scaling by a learnable parameter instead of 1/√d; average BLEU gain of 0.928 on 5 low-resource pairs.
Google Research applies LayerNorm to Q and K in ViT-22B to prevent attention-logit divergence and near one-hot, zero-entropy attention distributions in 8B+ models.
Qwen3 removes the QKV bias from Qwen2 and introduces QK-Norm (RMSNorm) into the attention mechanism to ensure stable training.
Time complexity: O(n·d) dodatkowo.
L2 (original), LayerNorm (ViT-22B), or RMSNorm (Qwen3).
Parameter g scaling the Q·K product instead of 1/√d.
Per-head normalization along head_dim.
A modification of dense attention; all heads and paths remain active.
Normalization is independent per token and per head, so it is fully parallel.
A pointwise normalization runs on any accelerator without special requirements.
Transformers with QK-Norm are trained mainly on GPUs with Tensor Cores.
