SwiGLU
How it works
The input x is projected by two independent matrices: W (gate) and V (value). The gate output Swish₁(xW) is multiplied element-wise by xV, and the result is projected by an output matrix W₂: FFN_SwiGLU(x) = (Swish₁(xW) ⊗ xV)·W₂, where Swish₁(z) = z·σ(z) (β=1, i.e. SiLU). To keep the three matrices (W, V, W₂) from increasing the parameter count relative to a two-matrix FFN, the hidden dimension d_ff is reduced by about 2/3 (e.g. to 2/3·4d in LLaMA). The layer usually omits bias terms.
Problem solved
The conventional Transformer FFN sublayer with ReLU or GELU leaves room to improve model quality without increasing the parameter count. SwiGLU provides a more expressive, gated transformation that raises representation quality and benchmark scores while keeping the parameter budget constant.
Components
Linear projection of the input whose output is passed through Swish (β=1, SiLU) and acts as the gate.
The second linear projection of the input, multiplied element-wise by the gate output.
Hadamard product of the gate output Swish₁(xW) and the value projection xV.
Linear projection mapping the hidden layer back to the model dimension.
Implementation
Adding the third (gate) matrix without shrinking d_ff by about 2/3 increases FFN parameters by ~50% relative to the conventional variant.
SwiGLU uses Swish with β=1 (SiLU); using a different β or making it trainable deviates from the paper's definition.
The 2/3·4d factor can yield a dimension not aligned to the multiples used by tensor cores.
Evolution
Dauphin et al. introduce GLU for language modeling with gated convolutional networks using a sigmoid gate.
Introduction of the Swish activation (z·σ(βz)); its β=1 variant (SiLU) is used in the SwiGLU gate.
Noam Shazeer introduces SwiGLU as a GLU variant in the Transformer FFN and shows quality gains on T5/GLUE/SuperGLUE.
PaLM uses SwiGLU as the FFN activation in a 540B-parameter model.
LLaMA replaces ReLU with SwiGLU and uses a 2/3·4d dimension, popularizing SwiGLU across open LLMs.
Hyperparameters (configurable axes)
FFN hidden dimension. Reduced by roughly 2/3 relative to a conventional FFN to offset the third matrix.
Parameter of the Swish function; set to 1 in SwiGLU (Swish₁ = SiLU).
Whether the projections use bias terms. In practice usually disabled (PaLM, LLaMA).
Computational complexity
Time complexity: O(n · d_model · d_ff). Space complexity: O(d_model · d_ff).
Compute bottleneck
Performance dominated by three dense GEMM operations (W, V, W₂); no sparsity or routing.
Execution paradigm
All parameters are active for every token (no conditional computation).
Parallelism
The layer is fully parallel across tokens and within matmuls; no sequential dependencies.
Hardware requirements
The three dense matrix multiplications map ideally onto GPU tensor cores.
Dense GEMMs use the TPU matrix units well; the original T5 experiments ran on TPUs.