CSA2 selects a sparse subset of attention connections instead of computing the full dense matrix, and stores key-value states in the compressed FP4 format. The SWA Bounded Replay technique reconstructs missing sliding-window KV states rather than keeping all of them in memory, making the persistent KV cache many times smaller. Together this lowers memory and bandwidth demands, speeding up inference on long contexts.
Standard dense attention requires memory that grows with context length, and a full KV cache at a 1M-token window is too large. CSA2 curbs this cost through sparsity and FP4 compression.
Second-generation compressed sparse attention with FP4 KV caching introduced in the DeepSeek-V4 family.
