MLA
How it works
MLA projects the input representation into a low-dimensional latent vector via a down-projection matrix (joint key-value compression). Only this compressed latent vector is stored in the KV cache (optionally alongside a small decoupled part carrying RoPE positional information). During attention, per-head keys and values are reconstructed from the latent vector through up-projection matrices, after which attention is computed normally. Queries can be compressed analogously to reduce activation memory during training. RoPE positional embeddings are handled in a decoupled path because rotary encodings do not compose directly with the latent compression. The result is a KV cache whose size scales with the latent dimension d_c, far smaller than the head-count times head-dimension product of standard MHA.
Problem solved
Standard Multi-Head Attention must store a full key-value cache for every token and head, growing linearly with context length and becoming the dominant memory and throughput constraint in long-context inference. MLA addresses this by compressing the KV cache into a small latent vector.
Components
Matrix projecting the input representation into a low-dimensional latent vector c_KV, the only quantity kept in the KV cache.
Buffer storing the latent vector c_KV per token; its size scales with the latent dimension d_c rather than the number of heads.
Up-projection matrices that reconstruct per-head keys and values from the latent vector at attention time.
A small, separated key/query component carrying rotary positional embeddings (RoPE), which cannot be folded directly into the latent compression.
Official
Implementation
Rotary positional embeddings cannot be folded directly into the latent compression; omitting the decoupled RoPE path breaks position encoding.
Too small a d_c degrades quality; too large limits the KV cache memory savings.
Evolution
DeepSeek-V2 (236B/21B active) introduces MLA, cutting KV cache by 93.3% and raising max generation throughput to 5.76x.
DeepSeek-V3 continues to use MLA as its attention mechanism.
Kimi K2 model cards (1T/32B active) explicitly list "Attention Mechanism: MLA", confirming adoption beyond DeepSeek.
Hyperparameters (configurable axes)
Dimension of the KV latent vector; the key trade-off between KV cache size and quality.
Latent dimension for query compression, reducing activation memory during training.
Size of the decoupled component carrying rotary positional embeddings.
Computational complexity
Time complexity: O(n² · d). Space complexity: O(n · d_c).
Compute bottleneck
In standard MHA the KV cache grows linearly with context length and head count, limiting memory and throughput; MLA moves this bottleneck onto a small latent vector.
Execution paradigm
MLA is a dense attention mechanism; all heads are active. It often co-occurs with a sparse (MoE) feed-forward, but MLA itself is dense.
Parallelism
Attention is computed in parallel across tokens; MLA adds no sequential dependency beyond standard attention.
Hardware requirements
MLA targets efficient LLM inference on GPUs; the KV cache reduction frees HBM and increases throughput.