The model chooses which token pairs to attend over — via fixed patterns (local/block/global) or dynamic, learned importance criteria — skipping the rest. This reduces operations and KV-memory use, enabling context scaling to hundreds of thousands and millions of tokens.
Standard (dense) attention scales quadratically with sequence length, making very long contexts expensive in compute and memory.
