The input x is projected by two independent matrices: W (gate) and V (value). The gate output Swish₁(xW) is multiplied element-wise by xV, and the result is projected by an output matrix W₂: FFN_SwiGLU(x) = (Swish₁(xW) ⊗ xV)·W₂, where Swish₁(z) = z·σ(z) (β=1, i.e. SiLU). To keep the three matrices (W, V, W₂) from increasing the parameter count relative to a two-matrix FFN, the hidden dimension d_ff is reduced by about 2/3 (e.g. to 2/3·4d in LLaMA). The layer usually omits bias terms.
The conventional Transformer FFN sublayer with ReLU or GELU leaves room to improve model quality without increasing the parameter count. SwiGLU provides a more expressive, gated transformation that raises representation quality and benchmark scores while keeping the parameter budget constant.
Linear projection of the input whose output is passed through Swish (β=1, SiLU) and acts as the gate.
The second linear projection of the input, multiplied element-wise by the gate output.
Hadamard product of the gate output Swish₁(xW) and the value projection xV.
Linear projection mapping the hidden layer back to the model dimension.
Adding the third (gate) matrix without shrinking d_ff by about 2/3 increases FFN parameters by ~50% relative to the conventional variant.
SwiGLU uses Swish with β=1 (SiLU); using a different β or making it trainable deviates from the paper's definition.
The 2/3·4d factor can yield a dimension not aligned to the multiples used by tensor cores.
Dauphin et al. introduce GLU for language modeling with gated convolutional networks using a sigmoid gate.
Introduction of the Swish activation (z·σ(βz)); its β=1 variant (SiLU) is used in the SwiGLU gate.
Noam Shazeer introduces SwiGLU as a GLU variant in the Transformer FFN and shows quality gains on T5/GLUE/SuperGLUE.
PaLM uses SwiGLU as the FFN activation in a 540B-parameter model.
LLaMA replaces ReLU with SwiGLU and uses a 2/3·4d dimension, popularizing SwiGLU across open LLMs.
Time complexity: O(n · d_model · d_ff). Space complexity: O(d_model · d_ff).
Performance dominated by three dense GEMM operations (W, V, W₂); no sparsity or routing.
FFN hidden dimension. Reduced by roughly 2/3 relative to a conventional FFN to offset the third matrix.
Parameter of the Swish function; set to 1 in SwiGLU (Swish₁ = SiLU).
Whether the projections use bias terms. In practice usually disabled (PaLM, LLaMA).
All parameters are active for every token (no conditional computation).
The layer is fully parallel across tokens and within matmuls; no sequential dependencies.
The three dense matrix multiplications map ideally onto GPU tensor cores.
Dense GEMMs use the TPU matrix units well; the original T5 experiments ran on TPUs.
