Not All Tokens Deserve Equal Resolution: Score-Adaptive Indexing for Sparse Attention in Long-Context LLMs

Indexer-based sparse attention reduces the cost of the main attention operator by retrieving only a small subset of the context, but the indexing stage itself is expensive for long sequence lengths. Hierarchical indexing reduces this cost by representing contiguous regions with compact summaries and refining only selected candidates. However, fixed-size partitions assume that every region can be represented equally well by one summary. While mean pooling is reliable when nearby keys are similar, it can obscure important token-level variation when they are not. We introduce SCALE (Score-Adaptive Indexing for Long-Context LLMs), a training-free hierarchical index that improves summary fidelity under a fixed budget by adapting which tokens are grouped together. For a fixed chunk, the mean key minimizes squared reconstruction error. For the DSA scoring function used in our evaluation, we further show that key dispersion controls an upper bound on the score approximation error induced by mean pooling. SCALE therefore splits high-dispersion regions into more chunks and merges homogeneous neighbors, while retaining the original mean representation and token-level indexer for exact refinement. Using the same number of chunks, SCALE improves Top-$2,048$ candidate recall over fixed-block indexing by $0.32$–$2.37$ percentage points while consistently reducing score approximation error. With one summary per $64$ tokens on average, SCALE-64 improves pooled RULER accuracy from $65.3$ to $82.1$ and LongBench-v2 accuracy from $46.4$ to $50.4$ over fixed-block HISA-64, a recent strong fixed-block hierarchical baseline. On the 128K serving workload, the same configuration reduces end-to-end latency by $36\%$ relative to flat DSA (DeepSeek Sparse Attention), from $65.30$ s to $41.69$ s, while matching DSA on LongBench-v2 ($50.4$) and remaining close on RULER ($82.1$ versus $84.7$).