Architecture
Sparse Attention
2019ActiveUpdated: 28 August 2026Published
Key
innovation
An attention mechanism that computes only a subset of token-token pairs instead of the full matrix, reducing the quadratic cost.
Category
Architecture
Abstraction level
Building block
Operation level
Architecture blockModel
Use cases
Long-context language modelsEfficient LLM inferenceMultimodal models (long video/documents)Reducing context-serving cost (e.g. QSA in Qwen, sparse attention in GLM-5)
How it works
The model chooses which token pairs to attend over — via fixed patterns (local/block/global) or dynamic, learned importance criteria — skipping the rest. This reduces operations and KV-memory use, enabling context scaling to hundreds of thousands and millions of tokens.
Problem solved
Standard (dense) attention scales quadratically with sequence length, making very long contexts expensive in compute and memory.