Architecture

MoVA

2026ExperimentalPublished: 4 September 2026Updated: 4 September 2026Published

Key innovation

Extends Mixture-of-Experts expert routing from feed-forward layers into the attention layer by routing values, adding a new axis of sparse capacity scaling.

How it works

MoVA extends MoE-style expert routing into the multi-head attention mechanism. Within the attention layer, many experts are available for computing values, and for each token a router activates only a small subset of them — analogous to how an MoE router selects a subset of feed-forward experts. The remaining attention operations (computing attention weights and aggregating context) work as in a standard transformer, which is why MoVA remains compatible with FlashAttention, grouped-query attention, and sparse attention. In the 36B-A4B model this yields 36 billion total parameters with roughly 4 billion active per token. Within the K2 Horizon family, MoVA is combined with MoE feed-forward layers.

Problem solved

Conventional MoE architectures apply sparsity almost exclusively to feed-forward layers, leaving the attention layer dense. This limits how far model capacity can be scaled, because attention — which determines how a transformer brings together information from across its context — does not benefit from conditional expert activation. MoVA addresses this by introducing sparsity into attention as well, allowing total model capacity to grow while keeping the number of active parameters per token low.

Components

Value expertsA set of experts computing value representations within the attention layer; the router activates only a subset of them per token.

They extend the MoE principle to attention: instead of a single dense value projection, many value experts are available and a subset is routed.

Attention routerA gating network that selects the active subset of value experts for each token.

The MoE-style router moved into the attention layer; it drives conditional expert activation while keeping per-token compute fixed.

Multi-head attention backboneThe base attention mechanism into which expert routing is integrated; remains compatible with FlashAttention, GQA, and sparse attention.

The standard transformer multi-head attention block: it computes attention weights and aggregates context. MoVA's expert routing over values is embedded into it, while remaining compatible with FlashAttention, GQA, and sparse attention.

MoE feed-forward layersSparse feed-forward layers used alongside MoVA in the K2 Horizon family.

Sparse MoE-style feed-forward layers used in the K2 Horizon family alongside MoVA. They contribute classic MoE sparsity in the feed-forward network, while MoVA contributes sparsity in the attention layer.

Implementation

Reference implementations

IFM/K2-Horizon-MoVA-36B-A4B (Hugging Face, model code)

Python · Institute of Foundation Models (IFM)

Official

Implementation pitfalls

Preserving router numerics (GEMM partitioning)High

Serving K2 Horizon MoVA requires the xllm_source_router_gemm_partitions override, which preserves the checkpoint's router numerics. Omitting it can alter expert-routing decisions and degrade output quality.

Fix:Use the validated serving recipe (SGLang: BF16, TP=2, FlashAttention-3, xllm_source_router_gemm_partitions override; vLLM: --enable-expert-parallel, --trust-remote-code).

Custom modeling code required (trust_remote_code)Medium

The model relies on custom architecture code (custom_code), so loading via HF libraries requires trust_remote_code=True. Without it, the MoVA architecture will not be built correctly.

Fix:Set trust_remote_code=True (transformers) or --trust-remote-code (vLLM/SGLang) when loading the model.

Evolution

Original paper · 2026 · IFM Blog · Institute of Foundation Models

Introducing K2 Horizon: Frontier Performance, Radically Open

Institute of Foundation Models

2026

MoVA introduced with the K2 Horizon family (IFM)

Inflection point

On 3 September 2026 IFM releases K2 Horizon; the 36B-A4B variant equipped with MoVA extends expert routing into the attention layer. Released under Apache 2.0.

Introducing K2 Horizon (paper)