MoVA extends MoE-style expert routing into the multi-head attention mechanism. Within the attention layer, many experts are available for computing values, and for each token a router activates only a small subset of them โ analogous to how an MoE router selects a subset of feed-forward experts. The remaining attention operations (computing attention weights and aggregating context) work as in a standard transformer, which is why MoVA remains compatible with FlashAttention, grouped-query attention, and sparse attention. In the 36B-A4B model this yields 36 billion total parameters with roughly 4 billion active per token. Within the K2 Horizon family, MoVA is combined with MoE feed-forward layers.
Conventional MoE architectures apply sparsity almost exclusively to feed-forward layers, leaving the attention layer dense. This limits how far model capacity can be scaled, because attention โ which determines how a transformer brings together information from across its context โ does not benefit from conditional expert activation. MoVA addresses this by introducing sparsity into attention as well, allowing total model capacity to grow while keeping the number of active parameters per token low.
They extend the MoE principle to attention: instead of a single dense value projection, many value experts are available and a subset is routed.
The MoE-style router moved into the attention layer; it drives conditional expert activation while keeping per-token compute fixed.
On 3 September 2026 IFM releases K2 Horizon; the 36B-A4B variant equipped with MoVA extends expert routing into the attention layer. Released under Apache 2.0.
Total number of parameters in the (sparse) model.
Number of parameters actually activated for a single token.
Native context length supported by the model.
A router activates only a subset of value experts per token, moving MoE-style conditional activation into the attention mechanism.
Recommended GPU serving: BF16, FlashAttention-3 backend, and tensor (TP) and expert (EP) parallelism.
