Architecture
manifold-constrained Hyper-Connections
ResearchPublished: 29 September 2026Updated: 29 September 2026Published
Key
innovation
A variant of Hyper-Connections in which the learnable connection weights are constrained to a manifold, aiming to stabilise training and limit overhead while retaining flexibility beyond a plain residual connection.
Category
Architecture
Abstraction level
Building block
Operation level
Architecture blockLayer
Use cases
Stabilising training of deep TransformersAlternative to residual connectionsReducing representation collapseLLM pretraining (dense and sparse)
How it works
Like Hyper-Connections, mHC replaces a single residual connection with a set of learnable connections among representations at different depths and widths. The difference is projecting/parametrising the weights onto a manifold satisfying imposed constraints (e.g. norm, orthogonality), which regulates the dynamics of signal and gradient propagation.
Problem solved
Unconstrained learnable Hyper-Connection weights can cause training instability and extra overhead. mHC aims to tame them via a manifold constraint.
Components
Manifold connection matrixCore mechanism
Learnable connection weights constrained to a manifold.
Manifold constraintStabilisation
A rule (e.g. norm/orthogonality) defining admissible weights.
Evolution
2024
Hyper-Connections — the base mechanism (a generalisation of residual connections)
Inflection point