acceptodds
Under review as a conference paper at ICLR 2027

Sparse Attention with Entropy-Triggered KV Repair

Abstract

Sparse attention reduces the per-step decoding cost of language models trained with full attention. However, it can lengthen generations and lower accuracy, offsetting the per-step savings with additional tokens. We find that sparse-attention errors are most detrimental at steps where the full-attention next-token entropy is high: applying sparse attention only at these steps degrades accuracy far more than applying it at an equal number of random steps. We further find that at these high-entropy steps, most of the degradation stems from errors accumulated into the key-value (KV) cache during earlier sparse decoding, so even attending to the entire cache at such steps results in performance degradation. Motivated by these findings, we propose **HERA** (**H**igh-**E**ntropy-triggered KV **R**ep**A**ir), a method which uses sparse next-token entropy to trigger a KV repair — a full-attention pass that recomputes and replaces the cached KV states to yield the dense next-token distribution for sampling. We evaluate **HERA** on three reasoning models across four benchmarks. On MATH-500 with Qwen3-8B, **HERA** improves accuracy by 18.8 percentage points over the sparse-decoding baseline (i.e., Quest) with 25.7% fewer tokens, and by 5.6 points over a KV repair at fixed intervals baseline (i.e., ReSA) with 7.4% fewer tokens and 13.7% fewer repair calls.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse attention, entropy-guided decoding, large reasoning models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WnV9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Motivated by two findings, (1) sparse-attention errors are most detrimental at steps where the full-attention next-token entropy is high, and applying sparse attention only at these steps degrades performance, (2) the error from KV cache from past sparse decoding steps accounts for most of the degradation, a method named HERA which uses sparse next-token entropy to trigger a KV repair — a full-attention pass that recomputes and replaces the cached KV states to yield the dense next-token distribution for sampling is proposed in this paper. Experimental results on three reasoning models across four benchmarks show that HERA consistently improves accuracy and reduces generation length over the sparse-decoding baseline Quest, with accuracy gains of up to 18.8 percentage points and token reductions of up to 49.1%.

Critical strengths
  • The proposed method is interesting and well-motivated. It shows that sparse attention drifts from dense attention at high-entropy steps, and checks the causal direction by intervening at high-entropy versus random steps, then separates inherited KV-history error from local attention error.
  • The results are strong against Quest, HERA improves or matches Quest accuracy while generating fewer tokens across all settings. Against ReSA at matched repair frequency, HERA wins most of the accuracy comparisons.
  • The diagnosis and ablations are thorough, including causal high-entropy versus random interventions, inherited versus local error isolation, proxy validation, threshold sweeps, per-repair distributional analysis, trigger analysis, case studies, and reported truncation rates.
Critical weaknesses
  • Mentioned by the authors, efficiency is estimated only and end-to-end latency left as future work, it can’t support better time or memory efficiency without real kernels, especially since each repair re-encodes a variable amount of history.
  • The setup is a bit narrow to call this a general repair policy yet. Only Quest is tested, ReSA is a reimplementation rather than the original system, so this is a scheduling-policy comparison, not a systems comparison with published ReSA.
Questions for authors
  • Most of your rescues against Quest (88/90) and ReSA-16 (16/20) involve baseline trajectories that hit the generation limit. What are the accuracies if truncated trajectories are excluded, so we can separate genuine per-step quality improvement from simply generating fewer tokens and avoiding truncation?
  • How sensitive are results to a single fixed threshold across all settings, instead of per-setting frequency matching to ReSA?
5 comments
AuthorsOpenReview ↗

Author Response #1

We sincerely appreciate your encouraging and detailed review. We are especially grateful that you recognized the high-entropy intervention analysis, the isolation of inherited versus local KV errors, and the consistent improvements over Quest and matched-frequency periodic repair, as well as the breadth of our existing diagnostic ablations.


W #1. Measured efficiency and variable-length KV repair

We evaluate Qwen3-8B on Math87, a subset of 87 MATH-500 problems whose dense-decoding generations exceed 8,000 tokens. This subset allows us to evaluate efficiency on naturally long reasoning workloads. The subset is selected based solely on dense-decoding generation length, independently of HERA's performance, and all methods are evaluated on the same problems. All methods use a batch size of 14, and total wall-clock time is averaged over two runs.

Table 1. End-to-end efficiency on Math87 with Qwen3-8B. Total wall-clock time is averaged over two executions on a single NVIDIA RTX PRO 6000 (batch size = 14).

MethodSparse BudgetAcc. (%) ↑Time (min) ↓Truncated ↓
DenseFull75.8645.894
Quest4489.2061.9379
Quest102464.3745.1220
Quest204872.4138.999
Quest307270.1140.416
Quest409672.4140.454
ReSA-1644848.2856.4738
ReSA-16102472.4138.202
ReSA-16204874.7138.092
HERA (h=0.5)44865.5243.8620
HERA (h=0.5)51268.9742.9814
HERA (h=0.5)102475.8636.930
HERA (h=0.5)204875.8635.071

HERA achieves the same accuracy as dense decoding with lower end-to-end latency. At a sparse budget of 1024, HERA matches dense accuracy (75.86%) while reducing latency by 19.5%. Increasing the budget to 2048 further reduces latency by 23.6% (1.31× speedup) while maintaining the same accuracy.

The end-to-end results above integrate over entire trajectories, including the early decoding steps where the context is shorter than the sparse budget and sparse attention provides no savings. Following the measurement protocol of ReSA, which reports kernel-level latency at fixed context lengths, we additionally evaluate decoding efficiency under a fixed 32K context length and batch size of 8, including model computation and dense KV repair. Our optimized implementation achieves a 1.96× per-token GPU execution speedup over dense decoding.

The results also show that simply increasing the sparse-attention budget does not recover the same accuracy–efficiency trade-off as HERA in our evaluated configurations. Even with a budget of 4096, Quest achieves 72.41% accuracy in 40.45 minutes, whereas HERA achieves 75.86% accuracy in 35.07 minutes using only a 2048-token budget.

We further compare HERA against periodic repair using the same optimized kernel implementation, varying only the repair scheduling policy. At a 2048-token budget, HERA achieves 75.86% accuracy in 35.07 minutes, compared with 74.71% accuracy in 38.09 minutes for ReSA-16, reducing end-to-end latency by 7.9%. A similar advantage is observed at a 1024-token budget.

These results provide direct wall-clock evidence that HERA can improve practical inference efficiency, beyond the analytical KV-memory-access estimates reported in our original submission. They also demonstrate that selectively repairing accumulated KV errors can provide a more favorable accuracy–latency trade-off than either increasing the sparse-attention budget alone or applying periodic repair.

Regarding variable-length repair, HERA re-encodes the unrepaired suffix in chunks of up to 64 tokens, typically executed in the same batches as sparse decoding. Our profiling estimates that the incremental cost of dense KV repair accounts for 6.0% of end-to-end wall-clock time (Appendix A.11). This cost is included in the wall-clock results above.

AuthorsOpenReview ↗

Author Response #2

W #2. & Q #2. Robustness to a fixed entropy threshold across settings & Generality of the repair policy and comparison with original ReSA

(1) Robustness and generality. To further evaluate the robustness and generality of entropy-triggered repair, we conducted additional experiments sweeping HERA's entropy thresholds and periodic-repair intervals with both Quest and TidalDecode. For each sparse-attention method, we keep its token-selection mechanism fixed and vary only the repair policy to enable a controlled comparison across different repair frequencies.

Table 2. Results with Quest on Qwen3-8B / MATH-500 (3 seeds / 32,768-token generation limit). Metrics are defined in Sec. 5.1 of the manuscript.

MethodParameterAcc. (%) ↑Tokens (k) ↓Repairs ↓KV Access ↓
Dense–86.35.410.01.000×
Quest (no repair)–67.013.270.01.176×
Periodic repairr=6474.010.81168.30.996×
Periodic repairr=3277.89.63300.40.918×
Periodic repairr=1681.08.15508.80.822×
Periodic repairr=883.77.50936.90.864×
HERAh=0.980.98.25230.00.747×
HERAh=0.781.97.67305.90.699×
HERAh=0.583.96.93392.90.633×
HERAh=0.384.26.82445.20.630×
HERAh=0.284.86.75478.90.628×

Table 3. Additional results with TidalDecode on Qwen3-8B / MATH-500 (500 questions × 3 seeds; 32,768-token generation limit).

MethodParameterAcc. (%) ↑Tokens (k) ↓Repairs ↓KV Access ↓
Dense–86.35.410.01.000×
TidalDecode (no repair)–64.212.810.01.356×
Periodic repairr=3279.28.03250.40.800×
Periodic repairr=1680.57.57472.60.804×
Periodic repairr=883.16.95868.00.810×
HERAh=0.880.57.86303.90.803×
HERAh=0.781.17.57349.80.777×
HERAh=0.582.86.93438.20.708×
HERAh=0.382.96.65480.50.676×
HERAh=0.284.06.49501.40.655×

The additional results demonstrate that HERA achieves a more favorable accuracy–efficiency Pareto frontier than periodic repair with both sparse-attention methods. Notably, even compared with more frequent periodic repair (r=8r=8), HERA (h=0.2h=0.2) achieves higher accuracy with fewer generated tokens, fewer repair calls, and lower estimated KV access under both sparse-attention methods.

These results show that the advantage of entropy-triggered repair is not limited to a particular repair frequency or to Quest's token-selection mechanism. We have also added the complete Pareto-frontier plots to Appendix A.10 (Figures 7 and 8) of the revised manuscript.

(2) A fixed threshold for every setting. Per-setting frequency matching is used only to compare HERA with ReSA at similar repair counts. HERA itself does not require it. As shown in Tables 2 and 3, accuracy and generation length change smoothly with h, and HERA achieves a more favorable trade-off than periodic repair across thresholds from 0.2 to 0.9. Moreover, a single threshold h=0.5 is already shared across five model–benchmark settings in our main results (Qwen3-8B on MATH-500, AIME 2025, and GPQA-Diamond; Qwen3-4B on MATH-500 and AIME 2025; Tables 2–3 of the paper), where HERA achieves higher accuracy than ReSA-16 in all five. The same h=0.5 also achieves better performance in our wall-clock experiments at the same attention budgets (Table 1).

(3) Comparison with original ReSA. The released ReSA code targets Qwen2/Qwen2.5 architectures and requires adaptation for Qwen3. We adapted it to Qwen3-8B and compared it with our implementation on all 87 Math87 problems. We use a fixed 1K-token budget for our implementation, which is comparable in scale to ReSA's dynamic 10%-of-context budget on these trajectories. Since the ReSA paper reports kernel-level latency at fixed context lengths rather than workload-level wall-clock time, we measure the generation time of both implementations ourselves.

Table 4. Comparison with the original ReSA implementation on Math87 (Qwen3-8B). Time is the total generation time, excluding model loading.

ImplementationBudgetAcc. (%)Time (min) ↓
Original ReSA-1610% of context71.3191.1
Our ReSA-16102472.435.3

Our ReSA-16 implementation matches the original implementation's accuracy while completing the same workload 5.4× faster.

Although the original ReSA paper discusses compatibility with continuous batching and chunked prefill (Sec. 2.2), its released evaluation implementation does not include these serving optimizations. Our engine instead builds on FlashInfer, paged KV caching, continuous batching, and CUDA Graphs. Importantly, HERA and periodic repair share this optimized engine, allowing us to isolate the practical benefits of entropy-guided repair scheduling from differences in the underlying implementation.

AuthorsOpenReview ↗

Author Response #3

Q #1. Accuracy when truncated trajectories are excluded

We conducted trajectory-level analyses on Qwen3-4B and DeepSeek-R1-Distill-Llama-8B using the MATH-500 benchmark with a 32,768-token generation limit.

(1) Accuracy analysis.

Table 5. Accuracy and truncation analysis on Qwen3-4B / MATH-500. Results are aggregated over three seeds. The jointly completed subset contains 1,123 question–seed pairs on which all four methods complete.

MethodOverall Acc. (%)Trunc. (%)Completed-only Acc. (%)Jointly Completed Acc. (%)
Dense86.730.2786.97 (1301/1496)89.76 (1008/1123)
Quest67.9324.6089.57 (1013/1131)89.49 (1005/1123)
ReSA-1682.208.0789.12 (1229/1379)89.67 (1007/1123)
HERA (h=0.5)85.473.5388.39 (1279/1447)89.76 (1008/1123)

Table 6. Accuracy and truncation analysis on DeepSeek-R1-Distill-Llama-8B / MATH-500.

MethodOverall Acc. (%)Trunc. (%)Completed-only Acc. (%)Jointly Completed Acc. (%)
Dense83.000.6083.50 (830/994)87.99 (762/866)
Quest75.9012.7086.71 (757/873)86.84 (752/866)
ReSA-1682.501.9084.10 (825/981)87.30 (756/866)
HERA (h=0.7)83.701.1084.63 (837/989)88.45 (766/866)

Completed-only accuracy is computed over each method's own non-truncated trajectories. Since these subsets differ, we additionally report accuracy on identical question–seed pairs where all four methods complete. On jointly completed trajectories, HERA matches or slightly exceeds dense decoding in accuracy across both models. On the full benchmark, HERA also substantially reduces truncation and increases the number of correct responses compared with Quest and ReSA-16.

Much of this accuracy improvement is associated with baseline-truncated trajectories. On DeepSeek-R1-Distill-Llama-8B, HERA correctly answers 78 more question–seed pairs than Quest, of which 63 (81%) come from trajectories where Quest reaches the generation limit. Importantly, reducing excessively prolonged reasoning is itself a central objective of our work, as longer generations can substantially offset the computational savings of sparse attention.

(2) Understanding the improvements.

Our trajectory analysis reveals that excessively prolonged reasoning under sparse attention can manifest in diverse failure patterns, including repeated verification of intermediate results, oscillation between competing interpretations, conflicting derivations, instability in previously established results, and extensive token repetition. These observations suggest that sparse attention can disrupt the progression of reasoning in different ways, rather than producing a single, uniform failure pattern.

HERA addresses these diverse failure patterns through entropy-guided KV repair, without relying on any particular observable failure behavior. As shown in Appendix A.3, repair changes the top-1 token in 39–53% of repair events, demonstrating its impact on intermediate decoding rather than merely shortening generation.

Reviewer_PwzUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the degradation of long-form reasoning under sparse attention. It finds that sparse-attention errors are particularly harmful at high-entropy decoding steps and attributes much of this degradation to errors accumulated in the KV cache. Based on these observations, the authors propose HERA, which uses sparse next-token entropy to trigger dense re-encoding of tokens generated since the previous repair.

Critical strengths
  1. The paper addresses a practical problem: sparse decoding can lengthen generations and reduce accuracy, offsetting its per-step savings. The distinction between local and inherited errors provides a clear motivation for repairing the KV history.
  2. The controlled interventions and matched-random repair baseline provide useful evidence that repair timing matters. HERA achieves meaningful improvements over periodic repair in several settings, particularly on MATH-500.
Critical weaknesses
  1. The practical efficiency benefit remains unverified. The paper reports estimated KV-memory access rather than wall-clock latency or throughput. This estimate does not capture the full cost of dense re-encoding or irregular repair scheduling. Moreover, the ReSA baseline uses a shared Quest backend rather than its original optimized implementation. These comparisons isolate scheduling effects but do not establish a practical advantage over existing systems.
  2. The methodological contribution is limited. Dense KV repair is already introduced by ReSA, while the importance of high-entropy tokens is also established in prior work. HERA mainly replaces a fixed repair interval with an entropy threshold. The paper needs stronger evidence that this rule offers a robust advantage over other inexpensive triggers and well-tuned periodic repair. Evaluation is restricted to one sparse-attention backend and a narrow set of attention budgets.
  3. The intervention study does not fully establish greater sensitivity at high-entropy positions. The controls match intervention rates, but not the magnitude of the resulting distributional perturbations. Since the paper already shows larger sparse–dense discrepancies at high-entropy positions, the greater degradation could reflect larger perturbations rather than greater sensitivity to comparable errors. Similarly, predicting dense entropy does not directly establish that sparse entropy identifies the positions with the greatest repair benefit.
Questions for authors
  1. Does HERA improve end-to-end latency at comparable accuracy when all repair and scheduling costs are included? How does it compare with the original ReSA implementation?
  2. Do the advantages persist across attention budgets, generation limits, and alternative triggers such as maximum token probability or the top-two probability margin?
  3. Can the authors separate the effect of perturbation magnitude from positional sensitivity, and quantify how much of the accuracy improvement comes from reduced truncation rather than improved correctness of completed responses?
7 comments
AuthorsOpenReview ↗

Author Response #1

We are glad that the reviewer finds our study practical, and improvements of our method meaningful. We hope to address concerns below:


W #1 & Q #1. End-to-end efficiency and comparison with the original optimized ReSA

(1) Measured end-to-end efficiency. We evaluate Qwen3-8B on Math87, a subset of 87 MATH-500 problems whose dense-decoding generations exceed 8,000 tokens. This subset allows us to evaluate efficiency on naturally long reasoning workloads. The subset is selected based solely on dense-decoding generation length, independently of HERA's performance, and all methods are evaluated on the same problems. All methods use a batch size of 14, and total wall-clock time is averaged over two runs.

Table 1. End-to-end efficiency on Math87 with Qwen3-8B. Total wall-clock time is averaged over two executions on a single NVIDIA RTX PRO 6000 (batch size = 14).

MethodSparse BudgetAcc. (%) ↑Time (min) ↓Truncated ↓
DenseFull75.8645.894
Quest4489.2061.9379
Quest102464.3745.1220
Quest204872.4138.999
Quest307270.1140.416
Quest409672.4140.454
ReSA-1644848.2856.4738
ReSA-16102472.4138.202
ReSA-16204874.7138.092
HERA (h=0.5)44865.5243.8620
HERA (h=0.5)51268.9742.9814
HERA (h=0.5)102475.8636.930
HERA (h=0.5)204875.8635.071

HERA achieves the same accuracy as dense decoding with lower end-to-end latency. At a sparse budget of 1024, HERA matches dense accuracy (75.86%) while reducing latency by 19.5%. Increasing the budget to 2048 further reduces latency by 23.6% (1.31× speedup) while maintaining the same accuracy.

The end-to-end results above integrate over entire trajectories, including the early decoding steps where the context is shorter than the sparse budget and sparse attention provides no savings. Following the measurement protocol of ReSA, which reports kernel-level latency at fixed context lengths, we additionally evaluate decoding efficiency under a fixed 32K context length and batch size of 8, including model computation and dense KV repair. Our optimized implementation achieves a 1.96× per-token GPU execution speedup over dense decoding.

(2) Comparison with larger sparse budgets and periodic repair. The results also show that simply increasing the sparse-attention budget does not recover the same accuracy–efficiency trade-off as HERA in our evaluated configurations. Even with a budget of 4096, Quest achieves 72.41% accuracy in 40.45 minutes, whereas HERA achieves 75.86% accuracy in 35.07 minutes using only a 2048-token budget.

We further compare HERA against periodic repair using the same optimized kernel implementation, varying only the repair scheduling policy. At a 2048-token budget, HERA achieves 75.86% accuracy in 35.07 minutes, compared with 74.71% accuracy in 38.09 minutes for ReSA-16, reducing end-to-end latency by 7.9%. A similar advantage is observed at a 1024-token budget.

These results provide direct wall-clock evidence that HERA can improve practical inference efficiency, beyond the analytical KV-memory-access estimates reported in our original submission. They also demonstrate that selectively repairing accumulated KV errors can provide a more favorable accuracy–latency trade-off than either increasing the sparse-attention budget alone or applying periodic repair.

(3) Comparison with original ReSA. The released ReSA code targets Qwen2/Qwen2.5 architectures and requires adaptation for Qwen3. We adapted it to Qwen3-8B and compared it with our implementation on all 87 Math87 problems. We use a fixed 1K-token budget for our implementation, which is comparable in scale to ReSA's dynamic 10%-of-context budget on these trajectories. Since the ReSA paper reports kernel-level latency at fixed context lengths rather than workload-level wall-clock time, we measure the generation time of both implementations ourselves.

Table 2. Comparison with the original ReSA implementation on Math87 (Qwen3-8B). Time is the total generation time, excluding model loading.

ImplementationBudgetAcc. (%)Time (min) ↓
Original ReSA-1610% of context71.3191.1
Our ReSA-16102472.435.3

Our ReSA-16 implementation matches the original implementation's accuracy while completing the same workload 5.4× faster.

Although the original ReSA paper discusses compatibility with continuous batching and chunked prefill (Sec. 2.2), its released evaluation implementation does not include these serving optimizations. Our engine instead builds on FlashInfer, paged KV caching, continuous batching, and CUDA Graphs. Importantly, HERA and periodic repair share this optimized engine, allowing us to isolate the practical benefits of entropy-guided repair scheduling from differences in the underlying implementation.

AuthorsOpenReview ↗

Author Response #2

W #1 & Q #1. (4) Runtime breakdown and overhead analysis. We profile HERA (h=0.5h=0.5, 2048-token budget) on Qwen3-8B/Math87 using an NVIDIA RTX PRO 6000 GPU, as shown in Table 3.

Table 3. Kernel-level runtime breakdown of HERA. Percentages are relative to total GPU kernel execution time.

ComponentGPU Time (%)
Model computation (decode-only batches)33.3
Model computation (mixed decode/repair batches)35.4
Sparse attention16.8
Sampling4.1
Dense attention (mixed batches)3.9
Quest page selection3.4
KV-cache writes0.7
Standalone repair forward0.3
Entropy evaluation and trigger checking0.25

Incremental repair overhead. Since dense repair is often executed together with sparse decoding in mixed batches, the standalone repair-forward kernels do not capture its full cost. We therefore estimate the incremental repair cost by comparing sampled batch executions with and without repair operations under identical initial KV states. Dense KV repair incurs an estimated 6.0% of end-to-end wall-clock time, while entropy computation and associated checking account for 0.7%.

Scheduling overhead. We examine whether irregular KV repair causes additional GPU waiting. Across two GPU allocations, GPU idle time accounts for 4.6–10.2% of wall-clock time, including all CPU-side scheduling and kernel-launch delays. Importantly, steps involving repair do not show a consistent increase in GPU idle time compared with ordinary decoding steps. This suggests that irregular repair scheduling does not introduce substantial additional GPU stalls in our evaluated setting.

Importantly, all these costs are included in our end-to-end wall-clock measurements, where HERA achieves 23.6% lower execution time than dense decoding and 7.9% lower time than periodic repair at the same 2048-token budget. These results provide direct evidence of practical efficiency improvements beyond the analytical KV-memory-access estimates in our original submission.


W #2 (a). Methodological Contribution

The methodological contribution is limited.

While ReSA introduced periodic KV repair and prior work has highlighted the importance of high-entropy tokens in reasoning, understanding when and why sparse attention degrades long-form reasoning remains an important open question.

In particular, it remains unclear where sparse attention is most detrimental to downstream reasoning, and whether the degradation stems primarily from local errors introduced by current-step sparse attention or from errors accumulated in historical KV states.

Our analysis in Sec. 3 aims to bridge this gap. First, to the best of our knowledge, our work is the first to establish the connection between high-entropy decoding positions and reasoning degradation under sparse attention. Through controlled interventions (Sec. 3.2), we show that applying sparse attention at high-entropy positions causes substantially greater downstream accuracy degradation than approximately rate-matched random interventions (10.8 vs. 1.4 percentage points).

Second, we identify the primary source of this degradation. By disentangling local errors from current-step sparse attention and inherited errors accumulated in historical KV states (Sec. 3.3), we find that inherited errors account for most of the degradation at high-entropy positions (8.0 vs. 1.2 percentage points).

These findings directly motivate HERA, which uses sparse next-token entropy to identify consequential decoding steps and selectively repair accumulated KV errors, rather than performing dense repair at fixed intervals. Beyond addressing reasoning degradation under sparse attention, HERA also provides practical validation of our findings on when and why such degradation occurs.

AuthorsOpenReview ↗

Author Response #3

W #2(b). Robustness across repair schedules and sparse-attention methods

Evaluation is restricted to one sparse-attention backend and a narrow set of attention budgets.

To further evaluate the robustness and generality of entropy-triggered repair, we conducted additional experiments sweeping HERA's entropy thresholds and periodic-repair intervals with both Quest and TidalDecode. For each sparse-attention method, we keep its token-selection mechanism fixed and vary only the repair policy to enable a controlled comparison across different repair frequencies.

Table 4. Results with Quest on Qwen3-8B / MATH-500 (32K limit). Metrics are defined in our Sec 5.1

MethodParameterAcc. (%) ↑Tokens (k) ↓Repairs ↓KV Access ↓
Dense–86.35.410.01.000×
Quest (no repair)–67.013.270.01.176×
Periodic repairr=6474.010.81168.30.996×
Periodic repairr=3277.89.63300.40.918×
Periodic repairr=1681.08.15508.80.822×
Periodic repairr=883.77.50936.90.864×
HERAh=0.980.98.25230.00.747×
HERAh=0.781.97.67305.90.699×
HERAh=0.583.96.93392.90.633×
HERAh=0.384.26.82445.20.630×
HERAh=0.284.86.75478.90.628×

Table 5. Additional results with TidalDecode on Qwen3-8B / MATH-500 (32K limit).

MethodParameterAcc. (%) ↑Tokens (k) ↓Repairs ↓KV Access ↓
Dense–86.35.410.01.000×
TidalDecode (no repair)–64.212.810.01.356×
Periodic repairr=3279.28.03250.40.800×
Periodic repairr=1680.57.57472.60.804×
Periodic repairr=883.16.95868.00.810×
HERAh=0.880.57.86303.90.803×
HERAh=0.781.17.57349.80.777×
HERAh=0.582.86.93438.20.708×
HERAh=0.382.96.65480.50.676×
HERAh=0.284.06.49501.40.655×

The additional results demonstrate that HERA achieves an accuracy–efficiency Pareto frontier than periodic repair with both sparse-attention methods. We have also added the complete Pareto-frontier plots to Appendix A.10 (Figures 7 and 8) of the revised manuscript.


Q #2. Robustness across attention budgets, generation limits, and repair triggers

(1) Attention budgets. As shown in our response to W #1, HERA achieves a better accuracy–latency trade-off than Quest and periodic repair across attention budgets on Math87.

(2) Generation limits. We examine HERA's robustness across generation limits of 4K, 8K, 16K, and 32K tokens on Qwen3-4B/MATH-500.

Table 6. Accuracy and generation length under different generation limits. Each cell reports accuracy (%) / mean generated tokens.

Method4K8K16K32K
Dense60.2/3,18078.4/4,29784.8/4,95486.2/5,213
Quest38.2/3,46254.4/5,60563.0/8,56467.6/13,025
ReSA-1649.6/3,32166.0/4,87077.8/6,52082.6/8,225
HERA (h=0.5)52.6/3,23573.0/4,61382.6/5,82185.6/6,624
ReSA-3247.0/3,29664.0/4,98575.4/6,88480.6/9,121
HERA (h=0.9)51.4/3,28869.4/4,77579.0/6,29082.6/7,663

HERA consistently achieves higher accuracy with fewer generated tokens than Quest and the corresponding periodic-repair baselines across all generation limits.

(3) Alternative repair triggers.

Motivation for using entropy. We follow Qwen3's recommendation to use stochastic rather than greedy decoding in thinking mode, with temperature 0.6 and top-p 0.95 (Sec. 5.1). Under stochastic decoding, token selection depends on the full sampling distribution rather than only its most likely candidate. While maximum token probability and the top-two probability margin characterize uncertainty through the leading candidates, entropy captures uncertainty across the entire predictive distribution, making it a natural signal for our setting.

Empirical evidence. We further compare entropy, maximum token probability, and the top-two probability margin in identifying positions with large sparse–dense distributional discrepancies. On Qwen3-8B / MATH-500, we analyze 150 Quest-generated trajectories and rank decoding positions using each signal.

Table 7. Recall of high-discrepancy positions across different selection rates.

Selected positionsEntropy recallMax. prob. recallTop-two margin recall
Top 5%32%22%20%
Top 10%49%43%41%
Top 20%86%80%79%

Recall measures the fraction of positions with sparse–dense total variation distance exceeding 0.5 identified at each selection rate. Entropy consistently identifies more high-discrepancy positions than the alternative signals. Moreover, all three signals can be computed from existing next-token logits, and our runtime profiling shows that entropy evaluation and trigger checking account for only 0.25% of GPU kernel time, with an estimated total entropy-related overhead of 0.7% of wall-clock time. Together, these results support entropy as an effective and computationally inexpensive signal for guiding KV repair.

Reviewer_c8SXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates accuracy degradation in sparse attention during long-context decoding. The authors find that the drop concentrates at positions with high attention entropy, and through controlled studies attribute it mainly to accumulated errors in historical KV states rather than approximation error in the current attention step. They propose HERA, an entropy-guided KV repair mechanism that corrects accumulated errors at high-entropy positions. Compared with Quest on mathematical and coding reasoning tasks, HERA mitigates accuracy loss during long-context decoding.

Critical strengths
  1. A useful empirical observation with solid supporting analysis.
    The paper shows that accuracy degradation at high-entropy positions is chiefly linked to accumulated errors in historical KV states, not to current-step sparsification. This offers a useful perspective on the failure modes of sparse attention. Targeted experiments that disentangle historical KV errors from current-step sparsification make the finding a substantive contribution.
Critical weaknesses
  1. The efficiency case is not adequately established.
    HERA is motivated by efficient long-context inference, yet no latency or throughput results are reported. Since HERA adds entropy computation and KV repair overhead, it is unclear whether the accuracy gains justify the extra cost. A runtime breakdown is also needed to show whether HERA offers a better accuracy–efficiency trade-off than simply increasing the sparse attention budget.

  2. Practical value is not yet demonstrated because validation is confined to a single sparse attention design.
    HERA is evaluated only on Quest. Its applicability to mainstream architectures such as DSA by DeepSeek remains uncertain, since different sparsity patterns and selection mechanisms may require nontrivial modifications. Effectiveness or feasibility on representative architectures should be shown to establish practical significance.

Questions for authors
  1. Please report end-to-end latency and throughput for HERA, with a breakdown of the overhead from entropy computation and KV repair, and compare against simply increasing the sparse attention budget under a comparable computational cost.
  2. Please discuss the feasibility and required modifications for integrating HERA into mainstream sparse attention architectures such as DSA, and whether the KV repair mechanism remains applicable under different sparsity patterns and selection strategies.
4 comments
AuthorsOpenReview ↗

Author Response (1/2)

We sincerely appreciate your constructive review. We are especially grateful that you recognized the empirical finding that accumulated errors in historical KV states are the main source of degradation at high-entropy positions, and that our controlled interventions help isolate this failure mode.


W #1 & Q #1. Practical efficiency: end-to-end latency, overhead, and larger sparse-attention budgets

no latency or throughput results are reported.

We appreciate the reviewer's concern regarding practical inference efficiency. To address this, we implemented an optimized inference engine and conducted additional end-to-end wall-clock experiments, including comparisons with larger sparse-attention budgets and periodic KV repair.

(1) Optimized implementation. Our optimized engine builds on widely adopted LLM serving components, including FlashInfer paged attention, paged KV caching, continuous batching, and CUDA Graphs, facilitating integration with modern inference frameworks. HERA and periodic repair share the same optimized engine, enabling a controlled comparison that isolates the effect of repair scheduling. Further implementation details are provided in Sec. 5.5 and Appendix A.11 of the revised manuscript.

(2) Measured end-to-end efficiency. We evaluate Qwen3-8B on Math87, a subset of 87 MATH-500 problems whose dense-decoding generations exceed 8,000 tokens. This subset allows us to evaluate efficiency on naturally long reasoning workloads. The subset is selected based solely on dense-decoding generation length, independently of HERA's performance, and all methods are evaluated on the same problems. All methods use a batch size of 14, and total wall-clock time is averaged over two runs.

Table 1. End-to-end efficiency on Math87 with Qwen3-8B. Accuracy and truncation are measured over 87 problems. Total wall-clock time is averaged over two executions (batch size = 14).

MethodSparse BudgetAcc. (%) ↑Time (min) ↓Truncated ↓
DenseFull75.8645.894
Quest4489.2061.9379
Quest102464.3745.1220
Quest204872.4138.999
Quest307270.1140.416
Quest409672.4140.454
ReSA-1644848.2856.4738
ReSA-16102472.4138.202
ReSA-16204874.7138.092
HERA (h=0.5)44865.5243.8620
HERA (h=0.5)51268.9742.9814
HERA (h=0.5)102475.8636.930
HERA (h=0.5)204875.8635.071

HERA achieves the same accuracy as dense decoding with lower end-to-end latency. At a sparse budget of 1024, HERA matches dense accuracy (75.86%) while reducing latency by 19.5%. Increasing the budget to 2048 further reduces latency by 23.6% (1.31× speedup, i.e., 1.31× higher request throughput) while maintaining the same accuracy.

The end-to-end results above integrate over entire trajectories, including the early decoding steps where the context is shorter than the sparse budget and sparse attention provides no savings. Following the measurement protocol of ReSA, which reports kernel-level latency at fixed context lengths, we additionally evaluate decoding efficiency under a fixed 32K context length and batch size of 8, including model computation and dense KV repair. Our optimized implementation achieves a 1.96× per-token GPU execution speedup over dense decoding.

(3) Comparison with larger sparse budgets and periodic repair. The results also show that simply increasing the sparse-attention budget does not recover the same accuracy–efficiency trade-off as HERA in our evaluated configurations. Even with a budget of 4096, Quest achieves 72.41% accuracy in 40.45 minutes, whereas HERA achieves 75.86% accuracy in 35.07 minutes using only a 2048-token budget.

We further compare HERA against periodic repair using the same optimized kernel implementation, varying only the repair scheduling policy. At a 2048-token budget, HERA achieves 75.86% accuracy in 35.07 minutes, compared with 74.71% accuracy in 38.09 minutes for ReSA-16, reducing end-to-end latency by 7.9%. A similar advantage is observed at a 1024-token budget.

These results provide direct wall-clock evidence that HERA can improve practical inference efficiency, beyond the analytical KV-memory-access estimates reported in our original submission. They also demonstrate that selectively repairing accumulated KV errors can provide a more favorable accuracy–latency trade-off than either increasing the sparse-attention budget alone or applying periodic repair.

AuthorsOpenReview ↗

Author Response (2/2)

(4) Runtime breakdown and overhead analysis. We profile HERA (h=0.5h=0.5, 2048-token budget) on Qwen3-8B/Math87 using an NVIDIA RTX PRO 6000 GPU, as shown in Table 3.

Table 3. Kernel-level runtime breakdown of HERA. Percentages are relative to total GPU kernel execution time.

ComponentGPU Time (%)
Model computation (decode-only batches)33.3
Model computation (mixed decode/repair batches)35.4
Sparse attention16.8
Sampling4.1
Dense attention (mixed batches)3.9
Quest page selection3.4
KV-cache writes0.7
Standalone repair forward0.3
Entropy evaluation and trigger checking0.25

Incremental repair overhead. Since dense repair is often executed together with sparse decoding in mixed batches, the standalone repair-forward kernels do not capture its full cost. We therefore estimate the incremental repair cost by comparing sampled batch executions with and without repair operations under identical initial KV states. Dense KV repair incurs an estimated 6.0% of end-to-end wall-clock time, while entropy computation and associated checking account for 0.7%.

Scheduling overhead. We examine whether irregular KV repair causes additional GPU waiting. Across two GPU allocations, GPU idle time accounts for 4.6–10.2% of wall-clock time, including all CPU-side scheduling and kernel-launch delays. Importantly, steps involving repair do not show a consistent increase in GPU idle time compared with ordinary decoding steps. This suggests that irregular repair scheduling does not introduce substantial additional GPU stalls in our evaluated setting. Importantly, all these costs are included in our end-to-end measurements, where HERA achieves 23.6% lower wall-clock time than dense decoding and 7.9% lower time than periodic repair at the same 2048-token budget. Detailed profiling methodology is provided in Appendix A.11 of the revised manuscript.


W #2 & Q #2. Applicability to other sparse-attention architectures

Beyond Quest, we conducted experiments using TidalDecode as another sparse-attention method with a different token-selection mechanism. We compare our HERA (entropy-triggered repair) with periodic repair by sweeping entropy thresholds and repair intervals, while keeping the sparse-attention method fixed for a fair comparison.

Table 3. Results on Qwen3-8B / MATH-500 with TidalDecode (500 questions × 3 seeds). Tokens and Repairs are averaged per trajectory. KV Access is estimated and normalized by dense decoding.

MethodParameterAcc. (%) ↑Tokens (k) ↓Repairs ↓KV Access ↓
Dense–86.35.410.01.000×
TidalDecode (no repair)–64.212.810.01.356×
Periodic repairr=3279.28.03250.40.800×
Periodic repairr=1680.57.57472.60.804×
Periodic repairr=883.16.95868.00.810×
HERAh=0.880.57.86303.90.803×
HERAh=0.781.17.57349.80.777×
HERAh=0.582.86.93438.20.708×
HERAh=0.382.96.65480.50.676×
HERAh=0.284.06.49501.40.655×

The results demonstrate that HERA retains its Pareto advantage over periodic repair with TidalDecode. The corresponding Pareto-frontier plots are provided in Appendix A.10 (Figure 8) of the revised manuscript.

Applicability to DSA. A direct evaluation on DeepSeek Sparse Attention (DSA) exceeds our current GPU resources, as DeepSeek-V3.2 has 671B parameters. Nevertheless, HERA's entropy-based trigger is independent of the sparse-attention selection mechanism and is conceptually applicable to DSA. Dense KV repair would leave the lightning indexer and its selection rule unchanged, but would need to be adapted to DSA's MLA-based cache, including refreshing the indexer's cached keys for the re-encoded tokens. Importantly, DSA is trained with sparse attention, unlike the dense-trained models studied in our work, so the effectiveness of dense KV repair requires further validation. Given our finding that sparse-attention errors are most consequential at high-entropy steps (Sec. 3), it may also be worth exploring whether these positions deserve particular attention during sparse-attention training. Our TidalDecode experiments already demonstrate generalization beyond Quest, while extending HERA to DSA remains future work.

Thank you for your response. However, my main concern about the limited applicability and significance of the work remains unresolved, so I will maintain my original score.

AuthorsOpenReview ↗

Additional Evaluation on Learned Sparse Attention (SeerAttention-R).

We thank the reviewer for raising the question of generalization to learned sparse attention. Given the computational cost of evaluating large-scale DSA models, we additionally evaluate HERA with SeerAttention-R (ICLR 2026)[1], a learned sparse attention method designed for long-form reasoning.

We conduct experiments on MATH87 (a subset of 87 MATH-500 problems whose dense-decoding generations exceed 8,000 tokens) using Qwen3-4B (A100, batch size 8, sparse budget 512). Under the same budget, HERA achieves 75.86% accuracy (66/87), outperforming both the original SeerAttention-R (51.72%, 45/87) and periodic repair with interval 16 (70.11%, 61/87). HERA also reduces average generation length from 23,919.6 to 13,552.1 tokens compared to the original method, while achieving lower inference time than periodic repair (100.68 vs. 107.90 minutes).

Together with our comprehensive evaluations of QUEST and TidalDecode across two models, these results further demonstrate HERA's applicability across different sparse attention mechanisms, including both training-free and learned approaches.

[1] Yizhao Gao et al. "Sparse Attention Adaptation for Long Reasoning." ICLR 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers and AC for their time and efforts. We would like to summarize the core contributions of our work and the additional evidence provided during the rebuttal.

1. Core Contributions and Novelty.

Our work goes beyond observing prolonged generations and accuracy degradation under sparse attention by identifying where and why sparse attention disrupts long-form reasoning.

(1) where: By analyzing the divergence between dense and sparse predictions and conducting controlled interventions, we show that applying sparse attention at high-entropy positions causes disproportionately greater downstream reasoning degradation.

(2) why: We further isolate two sources of sparse-attention error: local errors from current-step sparsification and inherited errors accumulated in historical KV states. We identify that the latter accounts for most of the degradation at these critical positions.

Motivated by these findings, we propose HERA, which adaptively triggers KV repair at high-entropy positions rather than applying dense correction at fixed intervals, achieving better accuracy–efficiency trade-offs.

2. Additional Evidence Addressing Reviewer Concerns.

  • Practical efficiency (c8SX, PwzU, WnV9): We developed an optimized GPU inference engine and evaluated actual wall-clock performance. HERA matches dense decoding accuracy with 23.6% lower inference time and runs 7.9% faster than periodic repair at the same attention budget. Simply increasing Quest's attention budget does not achieve the same accuracy–runtime trade-off. We also provide kernel-level profiling and repair overhead analysis.

  • General applicability and robustness (c8SX, PwzU, WnV9): We show that HERA achieves accuracy–efficiency Pareto trade-offs over periodic repair across different entropy thresholds and repair frequencies, using two sparse attention methods: Quest [1] and TidalDecode [2]. We further observe positive results with SeerAttention-R [3], a learned sparse attention method, where HERA improves accuracy from 51.72% to 75.86%, outperforming periodic repair (70.11%).

  • Generation length and reasoning degradation (PwzU, WnV9): Our analyses suggest that prolonged generation is an important source of degradation under sparse attention, as additional tokens can offset its per-step efficiency gains. For example, Quest [1] can even be slower than dense decoding (61.93 vs. 45.89 minutes). HERA addresses this problem, achieving higher overall accuracy and lower runtime than Quest [1]. Importantly, even on matched trajectories where both methods complete and answer correctly, HERA reduces generation length by 41.7% and 24.5% relative to Quest [1] across two models, showing that its benefits extend beyond avoiding truncation.

Overall, We believe our work contributes both an understanding of where and why sparse attention disrupts long-form reasoning and a practical approach to addressing this degradation by improving reasoning accuracy and inference efficiency across different sparse attention mechanisms.

References

[1] Jiaming Tang et al. "QUEST: Query-aware Sparsity for Efficient Long-context LLM Inference." ICML 2024.

[2] Lijie Yang et al. "TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention." ICLR 2025.

[3] Yizhao Gao et al. "Sparse Attention Adaptation for Long Reasoning." ICLR 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.