For an input vector a with n elements, RMSNorm computes the statistic RMS(a) = sqrt((1/n) · Σ aᵢ²) and then normalizes and rescales each component: āᵢ = (aᵢ / RMS(a)) · gᵢ, where g is a learnable gain vector of dimension n. Unlike LayerNorm, there is no mean subtraction and (in the basic version) no bias/offset term. A small constant ε is added to the denominator for numerical stability. In the partial variant (pRMSNorm), the RMS statistic is estimated from only the first p% of the vector's components, further lowering cost without breaking the invariance properties.
LayerNorm requires two passes over the activation vector — computing the mean (re-centering) and the variance (re-scaling) — which adds computational overhead that is especially costly in recurrent networks and deep Transformers. RMSNorm removes the re-centering step, reducing the number of operations and the running time while preserving training stability.
The root mean square of the vector components: RMS(a) = sqrt((1/n)·Σ aᵢ²). It replaces LayerNorm's standard deviation and requires no mean computation.
A learnable parameter vector of dimension n, multiplied element-wise with the normalized activations. It provides adaptive re-scaling and implicit learning-rate adaptation.
A variant where RMS is estimated from only the first p% of the vector components, further reducing compute cost without breaking the invariance properties.
Official
Summing squares in fp16/bf16 can lead to overflow or loss of precision.
Adding ε inside vs. outside the square root yields different numerical behavior and differs across implementations.
Porting LayerNorm code and leaving in mean subtraction or a bias term changes RMSNorm's semantics.
Ba, Kiros, and Hinton introduce LayerNorm — the predecessor that RMSNorm simplifies.
Zhang and Sennrich show that dropping re-centering matches LayerNorm while reducing running time by 7–64%.
LLaMA uses RMSNorm in a pre-normalization setup, after which the technique becomes a standard in subsequent model families (Gemma, Mistral, Qwen).
Time complexity: O(n · d). Space complexity: O(d).
The operation is memory-bandwidth bound: an element-wise squaring and a reduction over dimension d, not a matrix multiplication. It does not exploit tensor cores.
Size of the activation vector being normalized; typically equal to the model hidden dimension.
A small constant added for numerical stability (to avoid division by zero).
Whether a learnable gain vector is used; in practice almost always enabled.
Fraction of inputs used to estimate RMS in the pRMSNorm variant.
A dense operation applied to all activations with no conditional routing.
Each token's normalization is independent of the others, so it is fully parallel across tokens; within a token there is a reduction over the hidden dimension.
An element-wise-plus-reduction operation, memory-bandwidth bound; runs efficiently on any accelerator and on CPU.
Does not use tensor cores (no matrix multiply); usually fused with neighboring ops for efficiency.
