- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a sparse attention method that approximates unselected tokens by grouping them. While retaining a small number of tokens for exact computation, it uses count-aware mean proxies to account for the contribution of the remaining tokens, reducing the error caused by directly discarding the tail. The method combines query-geometry-based clustering and selection with error-driven read-budget allocation. It maintains task performance close to dense attention at low read budgets and demonstrates attention-stage acceleration through a custom kernel.
Critical strengths
1.The paper addresses information loss from unselected tokens in sparse attention.
The paper does not focus solely on improving top-k selection accuracy. It also considers the collective contribution of unselected tokens and approximates this contribution using group-level proxies. I think this is a worthwhile problem to study, particularly at low read budgets, where the selector may miss important tokens and simply ignoring the remaining tokens may hurt performance on tasks that depend on the overall attention distribution.
2.The method performs well at low read budgets and shows potential for kernel-level acceleration.
On RULER-32K HARD with Llama-3.1-8B, OpenAttention achieves an average score of 86.30 at a 1% read budget, only 0.74 percentage points below the dense attention score of 87.04. This suggests that approximating the tail can preserve good task performance at low budgets. In addition, the custom kernel achieves approximately a 10× attention-stage speedup under the corresponding benchmark settings with a batch size of 32.
Critical weaknesses
1.The methodological novelty remains limited, which is my main concern.
From the perspective of sparse selection, QGS still computes approximate scores using compressed representations and then selects a small number of tokens for exact computation. Its main changes are the use of query statistics to construct the projection space and the sharing of this representation between selection and clustering. While I acknowledge the rationale behind this design, I see it more as an incremental improvement to existing approximate retrieval and sparse selection frameworks than as a fundamentally new sparse selection mechanism. The paper itself also discusses QGS in the context of prior work that uses query information to process keys.
The paper’s more central contribution is the use of proxies to compensate for information from unselected tokens. However, both compensating for the unread portion and using means or centroids to summarize groups of tokens have been explored in prior work.
2.The evaluation covers a limited range of model sizes.
The evaluated models include only Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, and Mistral-7B, with sizes concentrated in the 7–8B range. Although these results provide some evidence across different models, the lack of evaluation on larger models makes it difficult to determine whether the approximation quality of the tail proxies and their ability to preserve accuracy hold as model size increases.
Questions for authors
1.Could training further improve the method?
Could the authors explore a trainable variant of OpenAttention, for example by optimizing its token selection or proxy construction, and compare its downstream performance with the training-free version under the same read budget? In particular, would training help reduce the accuracy gap to dense attention at the 1% read budget?
2.Does the method generalize to larger models?
Could the authors provide results on models substantially larger than 7–8B, particularly comparing OpenAttention with dense attention at the 1% read budget? This would help establish whether the method maintains near-dense task performance as model size increases.
5 comments
Official Comment for Reviewer cj9z (2/2)
(W2,Q2) Generalization to Larger Models
Table M. Mean accuracy (%) on RULER-32K HARD at a 1% KV-read budget on larger models.
| Method | Llama-70B | Qwen2.5-32B |
|---|---|---|
| Dense | 88.56 | 82.09 |
| OpenAttention | 88.40 | 81.38 |
To assess generalization beyond the 7–8B range, we additionally evaluate Llama-3.1-70B-Instruct and Qwen2.5-32B-Instruct on RULER-HARD. As shown in Table M, OpenAttention retains 99.8% and 99.1% of dense accuracy at a 1% KV-read budget, respectively. These results demonstrate that OpenAttention preserves near-dense task performance on substantially larger models, supporting the effectiveness of its tail-proxy approximation as model size increases. We have incorporated these results into Table 10 and Section 4.9 of the revised manuscript.
(Q1) Exploring a Trainable Variant of OpenAttention
Table N. Accuracy (%) before and after training on RULER-32K HARD.
| Read budget | Training-free | Trained (150 steps) | Trained (300 steps) |
|---|---|---|---|
| 1% | 86.30 | 86.85 | 86.79 |
| 0.1% | 83.08 | 84.70 | 85.12 |
We thank the reviewer for the insightful suggestion to optimize a trainable variant of OpenAttention through additional training. We explore a trainable variant that optimizes the static query metric used by QGS and four per-head proxy coefficients controlling query–key scaling (), cluster-size scaling (), logit bias (), and value scaling (). We initialize and to reproduce the training-free method. The LLM weights remain frozen, and the attention kernel is unchanged. Training uses 4,752 examples, equally split between newly generated RULER examples and natural/synthetic long-context examples, with no overlap with evaluation instances. We use two equally weighted distillation losses, combining KL divergence between dense attention and the selector’s predicted distribution with relative MSE between sparse and dense attention outputs.
Table N shows that 300 training steps improve average accuracy on RULER-32K HARD from 86.30% to 86.79% at the same 1% read budget. This reduces the absolute gap to dense attention (87.04%) from 0.74% to 0.25%. Since the training-free method already achieves near-dense accuracy at 1%, we additionally evaluate a more challenging 0.1% read budget to examine the benefits of training under tighter constraints. In this setting, accuracy improves from 83.08% to 85.12%, reducing the gap from 3.96% to 1.92%. These results demonstrate that training can further improve accuracy under fixed read budgets, with greater benefits in the more challenging 0.1% setting. We present these results in Table 9 and Section 4.7 of the revised manuscript.
Follow-up on Our Response and Additional Experiments
Dear Reviewer cj9z,
Thank you again for your time and for reviewing our paper. As the discussion period draws to a close, we would like to kindly follow up on our response.
In brief, we clarified the distinction between allocating exact reads according to estimated proxy error and selecting whole clusters according to attention mass. Our comparison with a Double-P-based implementation shows comparable accuracy with approximately 102× fewer reads (Table L). We also evaluated Llama-3.1-70B-Instruct and Qwen2.5-32B-Instruct, retaining 99.8% and 99.1% of dense accuracy, respectively, at a 1% read budget (Table M). Finally, we explored a trainable variant with the LLM frozen, obtaining further accuracy improvements under the same read budgets, particularly at 0.1% read (Table N).
We would be grateful to know whether our response and the additional results address your concerns, and we would be happy to provide any further clarification.
Best regards,
Authors
Official Comment for Reviewer cj9z (1/2)
We sincerely appreciate the reviewer's thorough review and the insightful comments. We hope the clarification of our methodological novelty and the additional experiments on larger models and a trainable variant provide useful evidence to help address the reviewer's concerns.
(W1) Novelty Compared with Previous Clustering-Based Compensation Methods
Table L. Performance comparison with Double-P, which selects clusters based on attention mass.
We thank the reviewer for the constructive comments on novelty, which have helped us clarify the positioning of our work. We agree that clustering-based compensation has already been explored in sparse attention. The key distinction therefore concerns how clusters are selected and used. To address the reviewer’s concern, we examined Double-P [1], ClusterAttn [2], and ClusterAttention [3]. We explain our methodological differences below.
1. Cluster selection approach (mass vs proxy error)
Existing methods [1,2,3] select clusters to read based on attention mass or attention scores and compensate for the remaining clusters. In contrast, our proposed method, ErrorGate, selects clusters based on proxy error at the cluster level and performs compensation after extracting heavy-hitter tokens that cause substantial within-cluster variation. Thus, the objective and design of cluster selection are fundamentally different, as our goal is to reduce the error incurred when representing clusters with proxies, rather than simply meeting an attention-mass target. This design is also closely tied to enabling operation in the challenging low-read setting discussed below.
2. Design for challenging low-read budgets
Designing sparse attention for very low read budgets is challenging and important. Attention-mass-based approaches [1–3] prioritize mass preservation, but reading entire selected clusters makes budgets such as 1% or 0.1% difficult to satisfy.
At a 32K context length, a 0.1% budget allows approximately 32 token-equivalent reads, including proxies. Under whole-cluster reading, this budget must cover both unselected-cluster proxies and all tokens in selected clusters. With at most 32 roughly balanced clusters, each contains approximately 1,000 or more tokens, so even one cluster expansion exceeds the budget. In contrast, because ErrorGate is designed to reduce each cluster’s proxy error, it separately budgets proxies and heavy-hitter tokens extracted from clusters, enabling operation under this constraint.
To demonstrate this empirically, we implemented a Double-P-based baseline because official code was unavailable, following the paper’s top-p settings of (0.95,0.7). In Table L, the baseline achieves 83.86% accuracy at an average read fraction of 9.92%, whereas OpenAttention achieves 83.08% accuracy at a measured read fraction of 0.097% (a nominal 0.1% budget). Thus, OpenAttention retains approximately 99.1% of the baseline’s accuracy with approximately 102× fewer reads. This highlights the practical difference between attention-mass-based whole-cluster reading and proxy-error-based selection of heavy-hitter tokens.
We have incorporated this discussion and the corresponding results into Section 5 (“Clustering and centroid proxies”), Table 18, and Appendix H of the revised manuscript.
[1] Ni, Wentao, et al. "Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs." arXiv preprint arXiv:2602.05191 (2026).
[2] Zhang, Minwei, et al. "ClusterAttn: KV cache compression under intrinsic attention clustering." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
[3] Nordenram, Kasper, and Amelie Dittmann. "ClusterAttention: A training-free speedup of bidirectional attention." arXiv preprint arXiv:2608.26965 (2026).