The input hidden state h0 is replicated n times to form a hyper hidden matrix H0 of shape n×d. Each layer applies two connection types: depth-connections (learnable weights linking layer inputs and outputs) and width-connections (a matrix that exchanges information among the n streams). In the static variant (SHC) the weights are fixed after initialization; in the dynamic variant (DHC) they depend on the normalized input through tanh activations. Initialization is chosen so the setup is equivalent to a Pre-Norm network at the start of training. Typical expansion rates are n=1,2,4,8, with n=4 performing best (n=8 adds minimal benefit).
Residual connections and their variants (Pre-Norm, Post-Norm) suffer from a seesaw effect: configurations that mitigate gradient vanishing tend to cause representation collapse, while those that prevent collapse worsen gradient propagation. Hyper-connections let the network learn this trade-off instead of hard-coding it.
Learnable weights linking layer inputs and outputs across depths — a generalization of the scalar residual connection (weight 1) to a learnable weight matrix.
A weight matrix that exchanges information among the n parallel hidden-state streams within a layer.
The hidden state expanded into n copies (H in R^{n×d}); at the network input it is created by replicating the input vector n times.
In the dynamic variant, connection weights are predicted from the current normalized input using tanh activations instead of being fixed.
Official
Replicating the hidden state into n streams increases memory usage (~15% for n=2, up to ~26% for n=4), which can limit batch size.
Connection weights must be initialized so the setup is equivalent to a Pre-Norm network at training start; incorrect initialization harms training stability.
arXiv 2409.19606 introduces depth- and width-connections and the SHC/DHC variants as an alternative to residual connections; validated on OLMo, OLMoE, ViT and DiT.
Time complexity: O(n^2 · d) na warstwę (narzut width-connections). Space complexity: O(n · d) dla ukrytego stanu.
Number of parallel hidden-state streams. Tested values: 1, 2, 4, 8; n=4 is optimal, n=8 adds minimal benefit.
SHC (Static Hyper-Connections) — learned, fixed weights; DHC (Dynamic Hyper-Connections) — input-dependent weights. DHC yields the best results.
The mechanism is dense (all streams active) but works with both dense and sparse (MoE) models.
The dynamic variant (DHC) predicts connection weights from the input; all n streams remain active.
The n-stream (width-connection) computation is parallel, but propagation through layers remains sequential in depth, as with classic residual connections.
It is a purely architectural change with negligible FLOPs overhead (+0.15–0.2%); works on any accelerator that trains transformers.
