- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper separates two questions in sparse attention: whether a small set of keys captures most attention mass, and whether ANN indexes can retrieve them. Experiments on two 8B models across RULER, LongBench, and LongProc show that concentration depends on context length, task, and head, while ANN recall can misrepresent retained mass. Held-out generation experiments support calibrated head-specific budgets.
Critical strengths
-
Separating attention concentration from ANN retrieval quality helps distinguish insufficient sparsity from search failures, making the findings useful for diagnosing sparse-attention systems.
-
Held-out rank correlations around 0.97 across prompts and decoding steps support predictable head-specific allocation, providing an empirical basis for offline calibration.
-
Frozen head-aware budgets improve task scores by up to 7.3 points over uniform allocation at matched budgets. This connects the attention measurements to practical generation quality.
Critical weaknesses
-
Without explicitly retaining attention sinks, it is unclear whether ANN’s low coverage reflects content-retrieval failures or merely missing a few fixed positions.
-
High coverage does not consistently preserve task performance, limiting support for the paper’s broader coverage-based design guidelines.
-
The paper recommends retrieving up to a target attention mass, but does not explain how to estimate coverage or stop searching without computing the full softmax denominator. Could the authors clarify this mechanism? The exact top-k generation experiments validate budget allocation, but leave the efficiency and practicality of this recommendation untested.
Questions for authors
See Weaknesses
3 comments
Dear Reviewer,
This is a kind reminder that the discussion period will end in less than 24 hours. We would appreciate it if you could take a moment to review their responses and provide any further comments before the discussion period closes.
Thank you for your time and effort!
Thank you for the thorough rebuttal. The sink-preserved ANN tables, the coverage–performance analysis (including sparse-attention subsets), and the Section 8 calibration procedure address my main questions in W1–W3. I plan to increase my rating accordingly. Thank you again for the clarifications.
W1. Thanks for your comment! We added sink-preserved ANN attention evaluation here. Due to space limit, we show some selected datasets:
In each cell below: no sink → keeping sink.
2wikimqa
gov_report
hotpotqa
These provide a practical advice on the ANN attention. But this is not a common way in sparse attention to keep sinks, like DeepSeek v3.2 report[1] uses top-k without specifically retaining the sinks, so we do not evaluate this setting in this paper. We will include it in future work.
W2. Thanks for your comment! Figure 6 shows the correlation between coverage and task performance score. The Pearson correlation for 16K input is ~0.86 and for 32K is ~0.74. If we only calculate the sparse attention cases, the correlation coefficients are ~0.97 and ~0.98 for 16K and 32K. So the coverage does have significant correlation with the task performance, especially in TopK and ANN attention.
W3. Thanks for your comment! Section 8 introduces the mechanism: we select the parameter k with a calibration query subset for each head. Specifically, on a subset of 20 separate prompts, each head receives the smallest k∈{64,128,…,4096} whose step-averaged coverage (through dense decoding) reaches 90% on at least 80% of these prompts, or all visible keys if no k does. And these ks are then frozen in online inference. This mechanism relies on the head stability shown in Section 6. It adds no inference cost. In addition, there are existing online coverage estimation methods that can be used in our future work.
[1] Liu, Aixin, et al. "Deepseek-v3. 2: Pushing the frontier of open large language models." arXiv preprint arXiv:2512.02556 (2025).