Coarse Semantic Blocks for Efficient Training-Free Sparse Attention

We introduce a training-free sparse attention method that builds a prompt-specific key index to accelerate both prefill and decoding. The method clusters the keys into large semantic blocks that group tokens from noncontiguous positions. Each query position selects blocks by scoring their centroids and attends to the original keys and values within them. Queries that select the same block are processed together, so prefill stays fast even though each query chooses its own blocks. Large blocks make centroid scoring and this grouped processing cheap, so each query can attend to more tokens in the same time. Larger blocks with a larger token budget can thus be both faster and more accurate on long-context tasks than smaller blocks with a smaller budget. We evaluate two hybrid LLMs, Qwen3.5-4B and 35B-A3B, on one NVIDIA H200. Including index construction and data movement, whole-model wall-clock speedups over dense attention are 2.3–2.4× for prefill and 1.7–2.0× for decoding at 512K tokens, and 2.9–3.0× for prefill at 1M. On five RULER tasks and LongBench-v2, the 35B-A3B model stays close to dense attention through 512K, and on MRCR through 256K. The 4B model loses accuracy mainly on multi-key retrieval, yet outperforms dense Qwen3.5-2B, whose latency is similar at 512K, on all three benchmarks.