Architecture
CSA2
2026ResearchUpdated: 15 September 2026Published
Key
innovation
A sparse, compressed attention mechanism with KV states cached in FP4, radically reducing memory usage for very long contexts.
Category
Architecture
Abstraction level
Building block
Operation level
Architecture blockLayerInference
Use cases
Very long-context inferenceKV cache size reductionLanguage models with ~1M-token windows
How it works
CSA2 selects a sparse subset of attention connections instead of computing the full dense matrix, and stores key-value states in the compressed FP4 format. The SWA Bounded Replay technique reconstructs missing sliding-window KV states rather than keeping all of them in memory, making the persistent KV cache many times smaller. Together this lowers memory and bandwidth demands, speeding up inference on long contexts.
Problem solved
Standard dense attention requires memory that grows with context length, and a full KV cache at a 1M-token window is too large. CSA2 curbs this cost through sparsity and FP4 compression.
Evolution
Original paper · 2026
DeepSeek-V4.1-Flash (model card)
2026
CSA2 in DeepSeek-V4 models
Inflection pointSecond-generation compressed sparse attention with FP4 KV caching introduced in the DeepSeek-V4 family.