- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SCOPE, which replaces the query-dependent online block scoring used by FSA with stored, chunk-level offline priors. Chunk scores are built from a domain corpus and stored in a hash table; unseen chunks use a lightweight ridge predictor, followed by local refinement. The resulting scores are converted into FSA-compatible top-K block indices, allowing the existing sparse-attention kernel to remain unchanged.
Experiments on Qwen-3.5-2B, Llama-3.2-1B, and Qwen2.5-7B report roughly 11–15% latency reductions over FSA, with larger gains at longer contexts. The paper also evaluates perplexity, MedQA accuracy, domain transfer, and several components through ablations.
Critical strengths
-
The method targets a real bottleneck that remains after sparse-attention kernel optimization. Replacing repeated query-dependent scoring with a cheap reusable prior is simple and potentially practical, especially in domain-specific serving workloads.
-
The latency improvement is reasonably consistent and not merely a very small numerical gain. The reported average improvement is around 11–15%, becomes larger at longer sequence lengths, and is observed across multiple backbones. The paper also includes useful operator-level analysis and domain-shift experiments rather than reporting only one favorable configuration.
Critical weaknesses
-
The long-context quality evaluation is not strong enough for a method that replaces query-dependent attention routing with largely query-independent offline scores. This is exactly the setting where an important chunk may become relevant only for a particular query. However, the paper mainly relies on perplexity and a MedQA accuracy check, while standard long-context retrieval/reasoning benchmarks such as RULER and LongBench v2 are explicitly left to future work. This is consequential: good perplexity does not establish that SCOPE preserves retrieval of query-specific information from a long context. The relatively large PPL degradation on some Llama/MedQA settings also suggests that this deserves closer study.
-
The current latency measurements do not fully establish an end-to-end serving improvement. SCOPE moves the scoring path from the GPU to CPU-side chunking, hash lookup/regression, refinement, and host-to-device transfer, but the evaluation relies primarily on CUDA-event measurements and does not clearly report these CPU-side costs as end-to-end wall-clock overhead. The serving setup is also batch size 1, single process, without paged KV or a production serving stack. Given that the reported overall gain is mostly 10–15%, these omitted costs could materially affect the conclusion.
-
FSA is a sensible primary baseline, and the appendix includes HashAttention, which is useful. However, the HashAttention comparison is limited to median latency at 8K on one backbone and does not include matched quality or longer-context results. Other closely related online-routing methods discussed by the paper, such as SeerAttention and TokenButler, are not experimentally compared. I would not require comparison to every sparse-attention method, but a more complete comparison with the closest scoring alternatives would make the claimed Pareto advantage much more convincing.
Questions for authors
-
Can the authors evaluate SCOPE on RULER, LongBench v2, or another query-dependent long-context retrieval benchmark? I would particularly like to see cases where the relevant evidence occurs far from the query and has low prior importance independent of that query.
-
Can the authors report true end-to-end wall-clock latency, including CPU chunking, hash lookup/regression, refinement, host-to-device transfer, and synchronization? How much of the reported 11–15% gain remains under this measurement?
-
Could the authors provide a matched latency–quality comparison with HashAttention at 16K/32K, and preferably one additional closely related routing baseline? The current 8K latency-only comparison makes it difficult to judge the SOTA advantage.
-
How sensitive is SCOPE to domain shift in terms of quality, not only latency? The law-domain experiment already shows that a high hit ratio does not necessarily imply a speed advantage; it would be useful to understand when offline chunk importance ceases to approximate query-dependent routing well enough.
We thank Reviewer nN15 for asking for query-dependent long-context quality, complete serving-path latency, and closer routing baselines. On Llama-3.2-1B, SCOPE obtains 25.23% accuracy on 400/503 LongBench-v2 examples, compared with 24.58% for FSA (+0.65 percentage points). This reports task accuracy on non-repeated long-context examples rather than relying on perplexity alone.
On the Law transfer workload, synchronized end-to-end wall-clock latency includes CPU chunking, hash lookup/regression, refinement, H2D transfer, and synchronization. SCOPE reduces median latency from 811.0 to 755.8 ms (6.8%) and P99 from 898.1 to 851.4 ms (5.2%); the ChunkStore hit ratio is 93.5%.
At 16K, SCOPE's median/P99 latency is 8.5%/6.2% lower than HashAttention and 15.3%/12.1% lower than SeerAttention. We limit our conclusions to the measured workloads and report task quality separately from latency.