acceptodds
Under review as a conference paper at ICLR 2027

OpenAttention: One Percent Is Enough with Tail Proxies

35 views

Abstract

Sparse attention reads only a small subset of the key-value cache, and how to treat the unread tail remains a key challenge. Discarding the tail biases the output, and sampling it as previous approaches do fails at reads of 5% and below, where the tail likely holds heavy hitters the selector missed. Our key observation is that the simplest possible proxy, one token that folds the entire tail into its count-aware mean, already yields lower output error than a sample at every read behind a practical selector. We further show in closed form that at low reads sampling nearly doubles the weight error of discarding while a single proxy stays below it. Building on these findings, we explore approximating the full attention distribution with a few proxies that target the tail. We propose OpenAttention, a pipeline of clustering, ErrorGate and top-k selection followed by a count-aware mean proxy. To implement clustering and selection both efficiently and effectively, we design training-free Query-Geometry Clustering (QGC) and a Query-Geometry Selector (QGS), which compress the keys while respecting the query directions. Additionally, unlike prior methods that rank clusters by centroid alone, ErrorGate allocates the exact read by the estimated weight error of each cluster. On RULER-32K HARD with Llama-3.1-8B, OpenAttention retains 99% of dense accuracy at a 1% read. It also achieves state-of-the-art accuracy across models, context lengths and datasets. Our customized attention kernel, OpenAttention.flash, runs the decode attention stage 10 faster than full attention at a 1% read.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attention, Efficient LLM, Inference Efficiency

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_cj9zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a sparse attention method that approximates unselected tokens by grouping them. While retaining a small number of tokens for exact computation, it uses count-aware mean proxies to account for the contribution of the remaining tokens, reducing the error caused by directly discarding the tail. The method combines query-geometry-based clustering and selection with error-driven read-budget allocation. It maintains task performance close to dense attention at low read budgets and demonstrates attention-stage acceleration through a custom kernel.

Critical strengths

1.The paper addresses information loss from unselected tokens in sparse attention.

The paper does not focus solely on improving top-k selection accuracy. It also considers the collective contribution of unselected tokens and approximates this contribution using group-level proxies. I think this is a worthwhile problem to study, particularly at low read budgets, where the selector may miss important tokens and simply ignoring the remaining tokens may hurt performance on tasks that depend on the overall attention distribution.

2.The method performs well at low read budgets and shows potential for kernel-level acceleration.

On RULER-32K HARD with Llama-3.1-8B, OpenAttention achieves an average score of 86.30 at a 1% read budget, only 0.74 percentage points below the dense attention score of 87.04. This suggests that approximating the tail can preserve good task performance at low budgets. In addition, the custom kernel achieves approximately a 10× attention-stage speedup under the corresponding benchmark settings with a batch size of 32.

Critical weaknesses

1.The methodological novelty remains limited, which is my main concern.

From the perspective of sparse selection, QGS still computes approximate scores using compressed representations and then selects a small number of tokens for exact computation. Its main changes are the use of query statistics to construct the projection space and the sharing of this representation between selection and clustering. While I acknowledge the rationale behind this design, I see it more as an incremental improvement to existing approximate retrieval and sparse selection frameworks than as a fundamentally new sparse selection mechanism. The paper itself also discusses QGS in the context of prior work that uses query information to process keys.

The paper’s more central contribution is the use of proxies to compensate for information from unselected tokens. However, both compensating for the unread portion and using means or centroids to summarize groups of tokens have been explored in prior work.

2.The evaluation covers a limited range of model sizes.

The evaluated models include only Llama-3.1-8B, DeepSeek-R1-Distill-Llama-8B, and Mistral-7B, with sizes concentrated in the 7–8B range. Although these results provide some evidence across different models, the lack of evaluation on larger models makes it difficult to determine whether the approximation quality of the tail proxies and their ability to preserve accuracy hold as model size increases.

Questions for authors

1.Could training further improve the method?

Could the authors explore a trainable variant of OpenAttention, for example by optimizing its token selection or proxy construction, and compare its downstream performance with the training-free version under the same read budget? In particular, would training help reduce the accuracy gap to dense attention at the 1% read budget?

2.Does the method generalize to larger models?

Could the authors provide results on models substantially larger than 7–8B, particularly comparing OpenAttention with dense attention at the 1% read budget? This would help establish whether the method maintains near-dense task performance as model size increases.

5 comments
AuthorsOpenReview ↗

Official Comment for Reviewer cj9z (1/2)

We sincerely appreciate the reviewer's thorough review and the insightful comments. We hope the clarification of our methodological novelty and the additional experiments on larger models and a trainable variant provide useful evidence to help address the reviewer's concerns.


(W1) Novelty Compared with Previous Clustering-Based Compensation Methods

Table L. Performance comparison with Double-P, which selects clusters based on attention mass.

MethodActual readqa_1qa_2vtfwemk_2mk_3mvAvg.
Dense100%79.0050.0095.2093.3395.00100.0096.7587.04
Double-P (0.95, 0.7)9.92%77.0050.5092.6091.3389.5090.5095.6283.86
OpenAttention (ours)0.097%75.5050.5089.7088.8390.0096.5090.5083.08

We thank the reviewer for the constructive comments on novelty, which have helped us clarify the positioning of our work. We agree that clustering-based compensation has already been explored in sparse attention. The key distinction therefore concerns how clusters are selected and used. To address the reviewer’s concern, we examined Double-P [1], ClusterAttn [2], and ClusterAttention [3]. We explain our methodological differences below.

1. Cluster selection approach (mass vs proxy error)

Existing methods [1,2,3] select clusters to read based on attention mass or attention scores and compensate for the remaining clusters. In contrast, our proposed method, ErrorGate, selects clusters based on proxy error at the cluster level and performs compensation after extracting heavy-hitter tokens that cause substantial within-cluster variation. Thus, the objective and design of cluster selection are fundamentally different, as our goal is to reduce the error incurred when representing clusters with proxies, rather than simply meeting an attention-mass target. This design is also closely tied to enabling operation in the challenging low-read setting discussed below.

2. Design for challenging low-read budgets

Designing sparse attention for very low read budgets is challenging and important. Attention-mass-based approaches [1–3] prioritize mass preservation, but reading entire selected clusters makes budgets such as 1% or 0.1% difficult to satisfy.

At a 32K context length, a 0.1% budget allows approximately 32 token-equivalent reads, including proxies. Under whole-cluster reading, this budget must cover both unselected-cluster proxies and all tokens in selected clusters. With at most 32 roughly balanced clusters, each contains approximately 1,000 or more tokens, so even one cluster expansion exceeds the budget. In contrast, because ErrorGate is designed to reduce each cluster’s proxy error, it separately budgets proxies and heavy-hitter tokens extracted from clusters, enabling operation under this constraint.

To demonstrate this empirically, we implemented a Double-P-based baseline because official code was unavailable, following the paper’s top-p settings of (0.95,0.7)(0.95, 0.7). In Table L, the baseline achieves 83.86% accuracy at an average read fraction of 9.92%, whereas OpenAttention achieves 83.08% accuracy at a measured read fraction of 0.097% (a nominal 0.1% budget). Thus, OpenAttention retains approximately 99.1% of the baseline’s accuracy with approximately 102× fewer reads. This highlights the practical difference between attention-mass-based whole-cluster reading and proxy-error-based selection of heavy-hitter tokens.

We have incorporated this discussion and the corresponding results into Section 5 (“Clustering and centroid proxies”), Table 18, and Appendix H of the revised manuscript.

[1] Ni, Wentao, et al. "Double-P: Hierarchical Top-P Sparse Attention for Long-Context LLMs." arXiv preprint arXiv:2602.05191 (2026).

[2] Zhang, Minwei, et al. "ClusterAttn: KV cache compression under intrinsic attention clustering." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

[3] Nordenram, Kasper, and Amelie Dittmann. "ClusterAttention: A training-free speedup of bidirectional attention." arXiv preprint arXiv:2608.26965 (2026).

AuthorsOpenReview ↗

Official Comment for Reviewer cj9z (2/2)

(W2,Q2) Generalization to Larger Models

Table M. Mean accuracy (%) on RULER-32K HARD at a 1% KV-read budget on larger models.

MethodLlama-70BQwen2.5-32B
Dense88.5682.09
OpenAttention88.4081.38

To assess generalization beyond the 7–8B range, we additionally evaluate Llama-3.1-70B-Instruct and Qwen2.5-32B-Instruct on RULER-HARD. As shown in Table M, OpenAttention retains 99.8% and 99.1% of dense accuracy at a 1% KV-read budget, respectively. These results demonstrate that OpenAttention preserves near-dense task performance on substantially larger models, supporting the effectiveness of its tail-proxy approximation as model size increases. We have incorporated these results into Table 10 and Section 4.9 of the revised manuscript.


(Q1) Exploring a Trainable Variant of OpenAttention

Table N. Accuracy (%) before and after training on RULER-32K HARD.

Read budgetTraining-freeTrained (150 steps)Trained (300 steps)
1%86.3086.8586.79
0.1%83.0884.7085.12

We thank the reviewer for the insightful suggestion to optimize a trainable variant of OpenAttention through additional training. We explore a trainable variant that optimizes the static query metric used by QGS and four per-head proxy coefficients controlling query–key scaling (α\alpha), cluster-size scaling (β\beta), logit bias (δ\delta), and value scaling (γ\gamma). We initialize α=β=γ=1\alpha=\beta=\gamma=1 and δ=0\delta=0 to reproduce the training-free method. The LLM weights remain frozen, and the attention kernel is unchanged. Training uses 4,752 examples, equally split between newly generated RULER examples and natural/synthetic long-context examples, with no overlap with evaluation instances. We use two equally weighted distillation losses, combining KL divergence between dense attention and the selector’s predicted distribution with relative MSE between sparse and dense attention outputs.

Table N shows that 300 training steps improve average accuracy on RULER-32K HARD from 86.30% to 86.79% at the same 1% read budget. This reduces the absolute gap to dense attention (87.04%) from 0.74% to 0.25%. Since the training-free method already achieves near-dense accuracy at 1%, we additionally evaluate a more challenging 0.1% read budget to examine the benefits of training under tighter constraints. In this setting, accuracy improves from 83.08% to 85.12%, reducing the gap from 3.96% to 1.92%. These results demonstrate that training can further improve accuracy under fixed read budgets, with greater benefits in the more challenging 0.1% setting. We present these results in Table 9 and Section 4.7 of the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Response and Additional Experiments

Dear Reviewer cj9z,

Thank you again for your time and for reviewing our paper. As the discussion period draws to a close, we would like to kindly follow up on our response.

In brief, we clarified the distinction between allocating exact reads according to estimated proxy error and selecting whole clusters according to attention mass. Our comparison with a Double-P-based implementation shows comparable accuracy with approximately 102× fewer reads (Table L). We also evaluated Llama-3.1-70B-Instruct and Qwen2.5-32B-Instruct, retaining 99.8% and 99.1% of dense accuracy, respectively, at a 1% read budget (Table M). Finally, we explored a trainable variant with the LLM frozen, obtaining further accuracy improvements under the same read budgets, particularly at 0.1% read (Table N).

We would be grateful to know whether our response and the additional results address your concerns, and we would be happy to provide any further clarification.

Best regards,

Authors

Reviewer_Cf3TOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces OpenAttention, which reduces KV-cache reads by computing attention exactly for a small set of tokens and representing the rest with cluster-based proxies. It combines query-aware clustering, a lightweight token selector, and error-based read allocation. Experiments across models and context lengths show accuracy close to dense attention, including about 99% of the dense score at a 1% read budget on RULER-32K HARD. A custom kernel achieves up to 10× attention-stage speedup, though full end-to-end costs are not measured.

Critical strengths
  1. A clear idea for handling unread tokens. Instead of dropping the tail or estimating it from a small sample, the method summarizes it with proxies. This directly addresses a key weakness of sparse attention, especially when the read budget is very small.

  2. Strong accuracy with very few reads. At a 1% read budget, OpenAttention scores 86.30 compared with 87.04 for dense attention on RULER-32K HARD, well ahead of the tested baselines. Results across different models and context lengths also suggest that the approach is broadly useful.

Critical weaknesses
  1. The practical speedup is not established.

The reported 10× gain only measures the attention kernel, leaving out selection, ErrorGate, proxy updates, and preprocessing. Saying that earlier methods also perform retrieval and selection does not justify excluding these costs, especially since OpenAttention adds a more complex pipeline with clustering, error-based allocation, and proxy maintenance.

  1. The evaluation does not fully support the breadth of the claims.

The strongest 1% result is on RULER-32K HARD. The 64K and 96K experiments use a higher 3% budget and increase the number of proxies with context length. Real-task evaluations are small, and the RULER evaluation covers seven tasks without reporting CWE. In my experience, several earlier sparse-attention methods show substantial performance drops on CWE, so I would particularly like to see OpenAttention’s results on this task.

  1. The benefit of ErrorGate seems small compared with its added complexity. With proxies and QGS already enabled, it improves the score from 86.73 to 86.82 at a 5% read budget. Results at the main 1% setting, including runtime costs, would help show whether the extra work is worthwhile.

  2. The architectural coverage is limited. The paper does not evaluate hybrid models such as Qwen3.5, which combines linear attention with full-attention layers.

Questions for authors

My main concerns are outlined in Weaknesses 1–3. Addressing Weakness 4 would be nice, but it’s not a deal-breaker if resources are tight.

5 comments
AuthorsOpenReview ↗

Official Comment for Reviewer Cf3T (1/2)

We sincerely appreciate the reviewer's thorough review and the insightful comments. We hope the additional experiments on practical speedup, OpenAttention's performance across additional settings, and the benefits of ErrorGate provide useful evidence to help address the reviewer's concerns.


(W1) Practical Speedup Comparison

Table E. End-to-end latency on H100 (batch size 1; sparse methods at 1% KV-read).

MethodE2E latency (s)Latency relative to ours
Dense (HF default SDPA)8.342.61×
OpenAttention (ours)3.191.00×
vAttention19.886.22×
MagicPIG23.437.34×

We agree that the 10× attention-kernel result alone does not establish an end-to-end inference speedup. In Table E, we therefore additionally benchmark the complete pipeline, including preprocessing, clustering, index scanning, selection, ErrorGate scoring, and proxy updates. On an NVIDIA H100 GPU with batch size 1 and a 1% KV-read budget, OpenAttention achieves 3.19 s, compared with 8.34 s for dense attention using default SDPA. This demonstrates a 2.6× end-to-end speedup with pipeline overheads included.

The evaluated vAttention and MagicPIG implementations require 19.88 s and 23.43 s, respectively. Their sparse-attention pipelines and selection stages are not jointly optimized within an integrated Triton kernel, resulting in overhead that makes them slower than dense attention in this setting. We included these results in Section 4.11 and Figure 9 of the revised manuscript.


(W2-1) Evaluation under Low Read Budgets at 64K and 96K Context Lengths

Table F. Evaluation at 64K and 96K context lengths with a 1% read budget.

Method (1% read)64K96K
Dense85.1575.61
MagicPIG (K10 L100)27.8018.89
HashAttention39.2517.38
vAttention (HAT)38.1627.93
OpenAttention (ours)84.2375.80

To address the reviewer’s concern about performance under low read budgets at 64K and 96K context lengths, we additionally evaluate OpenAttention under a 1% KV-read budget. As shown in Table F, OpenAttention retains at least 98.9% of dense accuracy across both context lengths and substantially outperforms the evaluated sparse-attention baselines. These results demonstrate that OpenAttention maintains near-dense performance with a low 1% read budget even for longer contexts up to 96K. We included these results in Table 12 and Section 4.10 of the revised manuscript.


(W2-2) Evaluation on the Full RULER Benchmark and CWE

Table G. Per-task accuracy (%) on all 13 RULER tasks

Method13-task Avg.HARD Avg.qa_1qa_2vtfwemk_2mk_3mvs_1s_2s_3mk_1mqcwe
Dense84.7587.0479.050.095.293.395.0100.096.898.099.098.094.096.257.2
HashAttention61.3348.2244.041.573.378.512.50.087.898.099.093.080.088.121.6
vAttention45.6836.2924.023.062.673.71.50.069.297.086.033.054.066.53.4
MagicPIG19.2010.5312.013.50.747.50.00.00.096.031.03.514.514.8816.05
OpenAttention (ours)85.4686.3077.550.096.093.595.097.095.198.099.098.094.596.2521.1

Table H. Comparison of CWE accuracy

Method16K CWE32K CWE
Dense78.007.20
HashAttention14.151.60
vAttention8.653.40
MagicPIG47.8016.05
OpenAttention67.5521.10

We thank the reviewer for recommending evaluation on the full RULER benchmark and especially for highlighting CWE as a challenging task for sparse attention. We have expanded our RULER-32K evaluation from the seven HARD tasks to all 13 tasks, including CWE. As shown in Table G, OpenAttention achieves an average score of 85.46% at a 1% KV-read budget, compared with 84.75% for dense attention.

Additionally, as shown in Table H, OpenAttention achieves 21.10% accuracy on CWE at 32K, compared with only 7.20% for dense attention. Since dense attention performs poorly at 32K, we also evaluate CWE at 16K using the same model to better assess accuracy preservation. At a 1% read budget, OpenAttention achieves 67.55%, compared with dense attention (78.00%), HashAttention (14.15%), vAttention (8.65%), and MagicPIG (47.80%). Although a 10.45 percentage-point gap to dense attention remains, OpenAttention preserves substantially more accuracy than the evaluated sparse-attention baselines. These results show that its advantages extend beyond the seven HARD tasks to CWE, which remains challenging and leaves room for further improvement. We have incorporated these results into Tables 5 and 13, Section 4.3, and Appendix C of the revised manuscript.

AuthorsOpenReview ↗

Official Comment for Reviewer Cf3T (2/2)

(W3) Effectiveness and Runtime Overhead of ErrorGate

Table I. Effectiveness of ErrorGate across top-k selectors and read budgets on RULER-32K HARD.

ProxiesErrorGateTop-k methodAvg.
✓✗QGS (0.1%)72.76
✓✓QGS (0.1%)83.08
✓✗HAT (5%)77.91
✓✓HAT (5%)85.90
✓✗Quest (5%)70.93
✓✓Quest (5%)84.49

Table J. Analysis of ErrorGate’s runtime overhead and additional cached memory.

MetricErrorGate offErrorGate onOn − off
E2E runtime (s)3.393.42+0.03
Cached memory excluding the KV cache (MiB)438504+66

We thank the reviewer for this comment. The small gain from ErrorGate with QGS should be viewed in light of the already near-dense performance of Proxy+QGS. At a 5% read budget, Proxy+QGS achieves 86.73 compared with 87.04 for dense attention, leaving limited room for further improvement. In Table I, we therefore evaluate ErrorGate under a more challenging 0.1% read budget and with different top-k selectors. With proxies enabled, ErrorGate improves QGS accuracy from 72.76 to 83.08 (+10.32%) at 0.1% read. At 5% read, it improves HashAttention from 77.91 to 85.90 (+7.99%) and Quest from 70.93 to 84.49 (+13.56%). These results demonstrate ErrorGate’s effectiveness under tighter read budgets and across different top-k selectors.

Regarding computational cost, as shown in Table J, profiling a single 32K-context request on Llama-3.1-8B at a 0.1% read budget shows that ErrorGate increases end-to-end runtime from 3.39 to 3.42 s, adding 0.03 s. Cached memory excluding the KV cache increases from 438 to 504 MiB (+66 MiB), while the KV cache remains unchanged. These results demonstrate that ErrorGate achieves meaningful accuracy improvements with very little runtime overhead. We have incorporated these results into Tables 7 and 17, Section 4.5, and Appendix G of the revised manuscript, and we hope they help address the reviewer’s concern.


(W4) Evaluation on Qwen3.5 with a Hybrid Architecture

Table K. Accuracy (%) on Qwen3.5 at 1% and 3% KV-read budgets.

Read budgetDenseOpenAttention (ours)vAttentionHashAttentionMagicPIG
1%84.1983.9130.8141.6754.05
3%84.1984.2451.3659.8482.88

We thank the reviewer for suggesting evaluation on hybrid architectures. To address this, we additionally evaluate Qwen3.5-9B on the seven RULER-HARD tasks. We apply OpenAttention and the sparse attention baselines to its eight gated full-attention layers while retaining the original 24 linear-attention layers. At a 1% KV-read budget in the gated full-attention layers, OpenAttention achieves 83.91% average accuracy versus 84.19% for the unmodified model, retaining 99.7% of its accuracy with an absolute decrease of only 0.28%. It also substantially outperforms MagicPIG (54.05%), HashAttention (41.67%), and vAttention with the HashAttention selector (30.81%). At a 3% read budget, OpenAttention achieves 84.24% accuracy, compared with 84.19% for the unmodified model. These results demonstrate that OpenAttention extends to hybrid architectures such as Qwen3.5, preserving model accuracy while substantially reducing KV reads in their gated full-attention layers. We included these results in Table 8 and Section 4.6 of the revised manuscript.

AuthorsOpenReview ↗

Follow-up on Our Response and Additional Experiments

Dear Reviewer Cf3T,

Thank you again for your time and for reviewing our paper. As the discussion period draws to a close, we would like to kindly follow up on our response.

In brief, we added end-to-end latency measurements demonstrating a 2.6× speedup over dense attention (Table E), along with further ErrorGate experiments showing substantial accuracy gains with little runtime overhead (Tables I and J). We extended the 1% read evaluation to 64K and 96K contexts, where OpenAttention maintains near-dense accuracy (Table F). We also evaluated all 13 RULER tasks and added CWE experiments at two context lengths, consistently outperforming the evaluated sparse-attention baselines on CWE (Tables G and H). Finally, our Qwen3.5-9B experiments demonstrate applicability to hybrid architectures while preserving near-original accuracy (Table K).

We would be grateful to know whether our response and the additional results address your concerns, and we would be happy to provide any further clarification.

Best regards,

Authors

Reviewer_Pg3EOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OpenAttention, which approximates unread KV-cache tokens using count-aware mean proxies instead of discarding or sampling them. It combines query-aware clustering, selection, and error-guided budget allocation, achieving near-dense accuracy at extremely low attention-read budgets.

Critical strengths

S1. Strong accuracy preservation at extreme sparsity. On RULER-32K HARD, OpenAttention achieves 86.30 accuracy at a 1% read, compared with 87.04 for dense attention and 48.22 for HashAttention. The large improvements over existing sparse-attention baselines are compelling, and the experiments across different models and context lengths further support the method's effectiveness.

S2. Good motivation together with thorough theoretical and empirical work. The analysis of residual sampling and proxy approximation provides a clear motivation for the approach. The paper also includes controlled ablations for proxy aggregation, query-aware clustering, and selection, along with an optimized fused attention kernel.

Critical weaknesses

W1. The reported speedups do not represent end-to-end inference speedups. The 10× speedup in Table 9 only corresponds to the attention computation part, and excludes index scanning, top-k selection, ErrorGate scoring, and residual-statistics updating. Additionally, this benchmark corresponds to the E=0 case and the main configuration at a 1% read employs error-guided cluster expansion.

W2. The additional gain of ErrorGate is questionable. Although the ablations demonstrate the effectiveness of tail proxies, the additional benefit of ErrorGate appears relatively small when QGS is already used. In Table 4, adding ErrorGate to Proxy+QGS improves accuracy from 86.73 to 86.82 at a 5% read, a gain of only 0.09 points. At a 1% read, the proposed error-aware ranking outperforms mean-logit ranking by 0.31 points (Table 3), while alternative error estimates achieve similar overall accuracy (Appendix A.1). Further evidence is needed to demonstrate if such improvement can warrant the additional calculation and complexity.

W3. Limited evaluation on realistic long-context tasks. The main results focus heavily on the seven synthetic RULER-HARD tasks. Besides designing experiments on LongBench, LooGLE, and LongBench-v2, the paper also conducts limited additional evaluation (i.e., only 30 examples are used in each summarization task and 60 examples on LongBench-v2). Broader evaluation would strengthen the generality of the conclusions.

Questions for authors

What is the complete decoding latency of the 1%-read configuration, including selection, ErrorGate, and statistics updates?

At a 1% read with QGS fixed, how much accuracy does ErrorGate add, and what is its runtime overhead?

Can the authors provide broader evaluations on realistic long-context tasks to confirm the improvements beyond RULER-HARD?

1 comment
AuthorsOpenReview ↗

Official Comment for Reviewer Pg3E

We sincerely appreciate the reviewer's thorough review and the insightful comments. We hope the additional experiments on end-to-end speedup, the gains from ErrorGate, and performance on long-context tasks provide useful evidence to help address the reviewer's concerns.


(W1,Q1) End-to-End Latency Comparison

Table A. End-to-end latency on H100 (batch size 1; sparse methods at 1% KV-read).

MethodE2E latency (s)Latency relative to ours
Dense (HF default SDPA)8.342.61×
OpenAttention (ours)3.191.00×
vAttention19.886.22×
MagicPIG23.437.34×

To address the reviewer’s concern regarding end-to-end inference performance, we additionally evaluate the complete pipeline using a 1% KV-read budget on an NVIDIA H100 GPU with batch size 1. As shown in Table A, OpenAttention achieves 3.19 s, compared with 8.34 s for dense attention using Hugging Face’s default SDPA, yielding a 2.6× end-to-end speedup. The evaluated vAttention and MagicPIG implementations require 19.88 s and 23.43 s, respectively. Their sparse-attention pipelines and selection stages are not jointly optimized within an integrated Triton kernel, resulting in overhead that makes them slower than dense attention in this setting. We included these results in Section 4.11 and Figure 9 of the revised manuscript.


(W2,Q2) Effectiveness and Runtime Overhead of ErrorGate

Table B. Effectiveness of ErrorGate across top-k selectors and read budgets on RULER-32K HARD.

ProxiesErrorGateTop-k methodAvg.
✓✗QGS (0.1%)72.76
✓✓QGS (0.1%)83.08
✓✗HAT (5%)77.91
✓✓HAT (5%)85.90
✓✗Quest (5%)70.93
✓✓Quest (5%)84.49

Table C. Analysis of ErrorGate’s runtime overhead and additional cached memory.

MetricErrorGate offErrorGate onOn − off
E2E runtime (s)3.393.42+0.03
Cached memory excluding the KV cache (MiB)438504+66

We thank the reviewer for the helpful comment encouraging a more thorough analysis of ErrorGate’s effectiveness. The Proxy+QGS method in Table 4 and the mean-logit baseline in Table 3 of our draft, both referenced by the reviewer, achieve 86.73 and 85.99, respectively. Both already achieve near-dense performance compared with 87.04 for dense attention, leaving limited room for further improvement. We therefore evaluate ErrorGate under a more challenging 0.1% read budget and with different top-k selectors. As shown in Table B, ErrorGate improves QGS accuracy from 72.76 to 83.08 at 0.1% read, a gain of 10.32%. At 5% read, it improves HashAttention from 77.91 to 85.90 (+7.99%) and Quest from 70.93 to 84.49 (+13.56%). These results demonstrate ErrorGate’s effectiveness under challenging read budgets and across different top-k selectors.

To address the reviewer’s concern, we measured runtime and additional cached memory and report the results in Table C. Profiling a single 32K-context request on Llama-3.1-8B at the 0.1% read setting shows a small increase in end-to-end runtime from 3.39 to 3.42 s. This corresponds to an overhead of 0.03 s, or approximately 0.9%. Under the same setting, enabling ErrorGate leaves the KV cache unchanged and increases cached memory excluding the KV cache from 438 to 504 MiB, adding 66 MiB. These results demonstrate that ErrorGate achieves meaningful accuracy improvements with very little runtime overhead. We have incorporated these results into Tables 7 and 17, Section 4.5, and Appendix G of the revised manuscript, and we hope they help address the reviewer’s concern.


(W3,Q3) Comparison of Sparse Attention Methods on Long-Context Tasks

Table D. Performance comparison on LongBench, LooGLE, and LongBench v2.

MethodLongBenchLooGLELongBench v2
dense42.890.75929.40
HashAttention27.970.67226.00
vAttention11.120.63321.10
MagicPIG26.050.65825.40
OpenAttention42.670.75829.20

In Table D, we expanded our evaluation on realistic long-context benchmarks using the full official evaluation sets, covering 21 tasks with 4,750 samples for LongBench, four tasks with 6,448 samples for LooGLE, and one task with 503 samples for LongBench-v2. At a 1% KV-read budget, OpenAttention retains over 99% of dense attention performance across all three benchmarks while consistently outperforming the compared sparse attention methods. These results provide further evidence that OpenAttention preserves near-dense performance on realistic long-context tasks beyond synthetic RULER-HARD. We present these results in Table 4 and Section 4.2 of the revised manuscript and thank the reviewer for suggesting experiments that help highlight the effectiveness of our method.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (Revision summary)

We sincerely thank all reviewers for their constructive feedback. We have revised the manuscript and added experiments and analyses to address the reviewers’ concerns. Unless otherwise specified, table and section numbers below refer to the revised manuscript.


Summary of revisions

  • Practical efficiency through end-to-end latency measurements (Pg3E, Cf3T). We measured end-to-end latency covering the entire inference process, demonstrating a 2.6× speedup over dense attention.

    • Revised manuscript. We added these results to Sec. 4.11 and Figure 9.
  • Effectiveness and computational overhead of ErrorGate (Pg3E, Cf3T). We demonstrated substantial accuracy gains from ErrorGate under tighter read budgets and across different token selectors. Additional profiling showed that these gains come with little runtime overhead.

    • Revised manuscript. We incorporated these results into Tables 7 and 17, Sec. 4.5, and Appendix G.
  • Broader evaluation on realistic long-context tasks (Pg3E). We expanded the experiments to the full evaluation sets of LongBench, LooGLE, and LongBench-v2. OpenAttention retains over 99% of dense performance across all three benchmarks at a 1% read budget.

    • Revised manuscript. We added these results to Table 4 and Sec. 4.2.
  • Evaluation at longer contexts under a 1% read budget (Cf3T). We extended the 1% read evaluation to 64K and 96K contexts, demonstrating that OpenAttention maintains near-dense accuracy at longer context lengths.

    • Revised manuscript. We added these results to Table 12 and Sec. 4.10.
  • Evaluation on all RULER tasks and additional CWE analysis (Cf3T). We expanded the evaluation to all 13 RULER tasks and added CWE experiments at two context lengths. OpenAttention achieves strong overall performance and consistently outperforms the evaluated sparse-attention baselines on CWE.

    • Revised manuscript. We incorporated these results into Tables 5 and 13, Sec. 4.3, and Appendix C.
  • Generalization to hybrid architectures (Cf3T). We added experiments on Qwen3.5-9B, demonstrating that OpenAttention extends to hybrid architectures while matching the original model’s accuracy and substantially outperforming existing sparse-attention methods.

    • Revised manuscript. We added these results to Table 8 and Sec. 4.6.
  • Methodological distinction and comparison with clustering-based methods (cj9z). We clarified the distinction between our proxy-error-based read allocation and attention-mass-based whole-cluster reading. A comparison with our Double-P-based implementation demonstrates comparable accuracy with approximately 102× fewer reads.

    • Revised manuscript. We expanded the discussion in Sec. 5 and added the comparison to Table 18 and Appendix H.
  • Generalization to substantially larger models (cj9z). We added experiments on Llama-3.1-70B-Instruct and Qwen2.5-32B-Instruct. OpenAttention retains 99.8% and 99.1% of dense accuracy, respectively, at a 1% read budget, demonstrating generalization beyond the originally evaluated 7–8B models.

    • Revised manuscript. We added these results to Table 10 and Sec. 4.9.
  • Exploration of a trainable variant (cj9z). We explored a trainable variant while keeping the LLM frozen. Training further improves accuracy under the same read budgets, with larger gains in the more challenging 0.1% setting.

    • Revised manuscript. We added these results to Table 9 and Sec. 4.7.

We hope these revisions address the reviewers’ concerns, and we would be glad to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.