Architecture
Qwen Sparse Attention (QSA)
ActiveUpdated: 28 August 2026Published
Key
innovation
A sparse-attention mechanism developed by the Qwen team, applied at the micro-block level within a hybrid attention architecture.
Category
Architecture
Abstraction level
Building block
Operation level
Architecture blockModel
Use cases
Long-context serving in Qwen modelsEfficient LLM/VLM inferenceHybrid attention architecturesMultimodal models with large context windows
How it works
QSA restricts attention computation to relevant token connections and operates at the micro-block level in a hybrid architecture (alongside Gated DeltaNet and a 'Gated Residual' mechanism), lowering the cost of serving contexts of hundreds of thousands to a million tokens.
Problem solved
Dense attention is too costly for very long contexts; an efficient sparse-attention mechanism that preserves long-context quality is needed.