- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Motivated by two findings, (1) sparse-attention errors are most detrimental at steps where the full-attention next-token entropy is high, and applying sparse attention only at these steps degrades performance, (2) the error from KV cache from past sparse decoding steps accounts for most of the degradation, a method named HERA which uses sparse next-token entropy to trigger a KV repair — a full-attention pass that recomputes and replaces the cached KV states to yield the dense next-token distribution for sampling is proposed in this paper. Experimental results on three reasoning models across four benchmarks show that HERA consistently improves accuracy and reduces generation length over the sparse-decoding baseline Quest, with accuracy gains of up to 18.8 percentage points and token reductions of up to 49.1%.
Critical strengths
- The proposed method is interesting and well-motivated. It shows that sparse attention drifts from dense attention at high-entropy steps, and checks the causal direction by intervening at high-entropy versus random steps, then separates inherited KV-history error from local attention error.
- The results are strong against Quest, HERA improves or matches Quest accuracy while generating fewer tokens across all settings. Against ReSA at matched repair frequency, HERA wins most of the accuracy comparisons.
- The diagnosis and ablations are thorough, including causal high-entropy versus random interventions, inherited versus local error isolation, proxy validation, threshold sweeps, per-repair distributional analysis, trigger analysis, case studies, and reported truncation rates.
Critical weaknesses
- Mentioned by the authors, efficiency is estimated only and end-to-end latency left as future work, it can’t support better time or memory efficiency without real kernels, especially since each repair re-encodes a variable amount of history.
- The setup is a bit narrow to call this a general repair policy yet. Only Quest is tested, ReSA is a reimplementation rather than the original system, so this is a scheduling-policy comparison, not a systems comparison with published ReSA.
Questions for authors
- Most of your rescues against Quest (88/90) and ReSA-16 (16/20) involve baseline trajectories that hit the generation limit. What are the accuracies if truncated trajectories are excluded, so we can separate genuine per-step quality improvement from simply generating fewer tokens and avoiding truncation?
- How sensitive are results to a single fixed threshold across all settings, instead of per-setting frequency matching to ReSA?
5 comments
Author Response #2
W #2. & Q #2. Robustness to a fixed entropy threshold across settings & Generality of the repair policy and comparison with original ReSA
(1) Robustness and generality. To further evaluate the robustness and generality of entropy-triggered repair, we conducted additional experiments sweeping HERA's entropy thresholds and periodic-repair intervals with both Quest and TidalDecode. For each sparse-attention method, we keep its token-selection mechanism fixed and vary only the repair policy to enable a controlled comparison across different repair frequencies.
Table 2. Results with Quest on Qwen3-8B / MATH-500 (3 seeds / 32,768-token generation limit). Metrics are defined in Sec. 5.1 of the manuscript.
| Method | Parameter | Acc. (%) ↑ | Tokens (k) ↓ | Repairs ↓ | KV Access ↓ |
|---|---|---|---|---|---|
| Dense | – | 86.3 | 5.41 | 0.0 | 1.000× |
| Quest (no repair) | – | 67.0 | 13.27 | 0.0 | 1.176× |
| Periodic repair | r=64 | 74.0 | 10.81 | 168.3 | 0.996× |
| Periodic repair | r=32 | 77.8 | 9.63 | 300.4 | 0.918× |
| Periodic repair | r=16 | 81.0 | 8.15 | 508.8 | 0.822× |
| Periodic repair | r=8 | 83.7 | 7.50 | 936.9 | 0.864× |
| HERA | h=0.9 | 80.9 | 8.25 | 230.0 | 0.747× |
| HERA | h=0.7 | 81.9 | 7.67 | 305.9 | 0.699× |
| HERA | h=0.5 | 83.9 | 6.93 | 392.9 | 0.633× |
| HERA | h=0.3 | 84.2 | 6.82 | 445.2 | 0.630× |
| HERA | h=0.2 | 84.8 | 6.75 | 478.9 | 0.628× |
Table 3. Additional results with TidalDecode on Qwen3-8B / MATH-500 (500 questions × 3 seeds; 32,768-token generation limit).
| Method | Parameter | Acc. (%) ↑ | Tokens (k) ↓ | Repairs ↓ | KV Access ↓ |
|---|---|---|---|---|---|
| Dense | – | 86.3 | 5.41 | 0.0 | 1.000× |
| TidalDecode (no repair) | – | 64.2 | 12.81 | 0.0 | 1.356× |
| Periodic repair | r=32 | 79.2 | 8.03 | 250.4 | 0.800× |
| Periodic repair | r=16 | 80.5 | 7.57 | 472.6 | 0.804× |
| Periodic repair | r=8 | 83.1 | 6.95 | 868.0 | 0.810× |
| HERA | h=0.8 | 80.5 | 7.86 | 303.9 | 0.803× |
| HERA | h=0.7 | 81.1 | 7.57 | 349.8 | 0.777× |
| HERA | h=0.5 | 82.8 | 6.93 | 438.2 | 0.708× |
| HERA | h=0.3 | 82.9 | 6.65 | 480.5 | 0.676× |
| HERA | h=0.2 | 84.0 | 6.49 | 501.4 | 0.655× |
The additional results demonstrate that HERA achieves a more favorable accuracy–efficiency Pareto frontier than periodic repair with both sparse-attention methods. Notably, even compared with more frequent periodic repair (), HERA () achieves higher accuracy with fewer generated tokens, fewer repair calls, and lower estimated KV access under both sparse-attention methods.
These results show that the advantage of entropy-triggered repair is not limited to a particular repair frequency or to Quest's token-selection mechanism. We have also added the complete Pareto-frontier plots to Appendix A.10 (Figures 7 and 8) of the revised manuscript.
(2) A fixed threshold for every setting. Per-setting frequency matching is used only to compare HERA with ReSA at similar repair counts. HERA itself does not require it. As shown in Tables 2 and 3, accuracy and generation length change smoothly with h, and HERA achieves a more favorable trade-off than periodic repair across thresholds from 0.2 to 0.9. Moreover, a single threshold h=0.5 is already shared across five model–benchmark settings in our main results (Qwen3-8B on MATH-500, AIME 2025, and GPQA-Diamond; Qwen3-4B on MATH-500 and AIME 2025; Tables 2–3 of the paper), where HERA achieves higher accuracy than ReSA-16 in all five. The same h=0.5 also achieves better performance in our wall-clock experiments at the same attention budgets (Table 1).
(3) Comparison with original ReSA. The released ReSA code targets Qwen2/Qwen2.5 architectures and requires adaptation for Qwen3. We adapted it to Qwen3-8B and compared it with our implementation on all 87 Math87 problems. We use a fixed 1K-token budget for our implementation, which is comparable in scale to ReSA's dynamic 10%-of-context budget on these trajectories. Since the ReSA paper reports kernel-level latency at fixed context lengths rather than workload-level wall-clock time, we measure the generation time of both implementations ourselves.
Table 4. Comparison with the original ReSA implementation on Math87 (Qwen3-8B). Time is the total generation time, excluding model loading.
| Implementation | Budget | Acc. (%) | Time (min) ↓ |
|---|---|---|---|
| Original ReSA-16 | 10% of context | 71.3 | 191.1 |
| Our ReSA-16 | 1024 | 72.4 | 35.3 |
Our ReSA-16 implementation matches the original implementation's accuracy while completing the same workload 5.4× faster.
Although the original ReSA paper discusses compatibility with continuous batching and chunked prefill (Sec. 2.2), its released evaluation implementation does not include these serving optimizations. Our engine instead builds on FlashInfer, paged KV caching, continuous batching, and CUDA Graphs. Importantly, HERA and periodic repair share this optimized engine, allowing us to isolate the practical benefits of entropy-guided repair scheduling from differences in the underlying implementation.
Author Response #3
Q #1. Accuracy when truncated trajectories are excluded
We conducted trajectory-level analyses on Qwen3-4B and DeepSeek-R1-Distill-Llama-8B using the MATH-500 benchmark with a 32,768-token generation limit.
(1) Accuracy analysis.
Table 5. Accuracy and truncation analysis on Qwen3-4B / MATH-500. Results are aggregated over three seeds. The jointly completed subset contains 1,123 question–seed pairs on which all four methods complete.
| Method | Overall Acc. (%) | Trunc. (%) | Completed-only Acc. (%) | Jointly Completed Acc. (%) |
|---|---|---|---|---|
| Dense | 86.73 | 0.27 | 86.97 (1301/1496) | 89.76 (1008/1123) |
| Quest | 67.93 | 24.60 | 89.57 (1013/1131) | 89.49 (1005/1123) |
| ReSA-16 | 82.20 | 8.07 | 89.12 (1229/1379) | 89.67 (1007/1123) |
| HERA (h=0.5) | 85.47 | 3.53 | 88.39 (1279/1447) | 89.76 (1008/1123) |
Table 6. Accuracy and truncation analysis on DeepSeek-R1-Distill-Llama-8B / MATH-500.
| Method | Overall Acc. (%) | Trunc. (%) | Completed-only Acc. (%) | Jointly Completed Acc. (%) |
|---|---|---|---|---|
| Dense | 83.00 | 0.60 | 83.50 (830/994) | 87.99 (762/866) |
| Quest | 75.90 | 12.70 | 86.71 (757/873) | 86.84 (752/866) |
| ReSA-16 | 82.50 | 1.90 | 84.10 (825/981) | 87.30 (756/866) |
| HERA (h=0.7) | 83.70 | 1.10 | 84.63 (837/989) | 88.45 (766/866) |
Completed-only accuracy is computed over each method's own non-truncated trajectories. Since these subsets differ, we additionally report accuracy on identical question–seed pairs where all four methods complete. On jointly completed trajectories, HERA matches or slightly exceeds dense decoding in accuracy across both models. On the full benchmark, HERA also substantially reduces truncation and increases the number of correct responses compared with Quest and ReSA-16.
Much of this accuracy improvement is associated with baseline-truncated trajectories. On DeepSeek-R1-Distill-Llama-8B, HERA correctly answers 78 more question–seed pairs than Quest, of which 63 (81%) come from trajectories where Quest reaches the generation limit. Importantly, reducing excessively prolonged reasoning is itself a central objective of our work, as longer generations can substantially offset the computational savings of sparse attention.
(2) Understanding the improvements.
Our trajectory analysis reveals that excessively prolonged reasoning under sparse attention can manifest in diverse failure patterns, including repeated verification of intermediate results, oscillation between competing interpretations, conflicting derivations, instability in previously established results, and extensive token repetition. These observations suggest that sparse attention can disrupt the progression of reasoning in different ways, rather than producing a single, uniform failure pattern.
HERA addresses these diverse failure patterns through entropy-guided KV repair, without relying on any particular observable failure behavior. As shown in Appendix A.3, repair changes the top-1 token in 39–53% of repair events, demonstrating its impact on intermediate decoding rather than merely shortening generation.
Author Response #1
We sincerely appreciate your encouraging and detailed review. We are especially grateful that you recognized the high-entropy intervention analysis, the isolation of inherited versus local KV errors, and the consistent improvements over Quest and matched-frequency periodic repair, as well as the breadth of our existing diagnostic ablations.
W #1. Measured efficiency and variable-length KV repair
We evaluate Qwen3-8B on Math87, a subset of 87 MATH-500 problems whose dense-decoding generations exceed 8,000 tokens. This subset allows us to evaluate efficiency on naturally long reasoning workloads. The subset is selected based solely on dense-decoding generation length, independently of HERA's performance, and all methods are evaluated on the same problems. All methods use a batch size of 14, and total wall-clock time is averaged over two runs.
Table 1. End-to-end efficiency on Math87 with Qwen3-8B. Total wall-clock time is averaged over two executions on a single NVIDIA RTX PRO 6000 (batch size = 14).
HERA achieves the same accuracy as dense decoding with lower end-to-end latency. At a sparse budget of 1024, HERA matches dense accuracy (75.86%) while reducing latency by 19.5%. Increasing the budget to 2048 further reduces latency by 23.6% (1.31× speedup) while maintaining the same accuracy.
The end-to-end results above integrate over entire trajectories, including the early decoding steps where the context is shorter than the sparse budget and sparse attention provides no savings. Following the measurement protocol of ReSA, which reports kernel-level latency at fixed context lengths, we additionally evaluate decoding efficiency under a fixed 32K context length and batch size of 8, including model computation and dense KV repair. Our optimized implementation achieves a 1.96× per-token GPU execution speedup over dense decoding.
The results also show that simply increasing the sparse-attention budget does not recover the same accuracy–efficiency trade-off as HERA in our evaluated configurations. Even with a budget of 4096, Quest achieves 72.41% accuracy in 40.45 minutes, whereas HERA achieves 75.86% accuracy in 35.07 minutes using only a 2048-token budget.
We further compare HERA against periodic repair using the same optimized kernel implementation, varying only the repair scheduling policy. At a 2048-token budget, HERA achieves 75.86% accuracy in 35.07 minutes, compared with 74.71% accuracy in 38.09 minutes for ReSA-16, reducing end-to-end latency by 7.9%. A similar advantage is observed at a 1024-token budget.
These results provide direct wall-clock evidence that HERA can improve practical inference efficiency, beyond the analytical KV-memory-access estimates reported in our original submission. They also demonstrate that selectively repairing accumulated KV errors can provide a more favorable accuracy–latency trade-off than either increasing the sparse-attention budget alone or applying periodic repair.
Regarding variable-length repair, HERA re-encodes the unrepaired suffix in chunks of up to 64 tokens, typically executed in the same batches as sparse decoding. Our profiling estimates that the incremental cost of dense KV repair accounts for 6.0% of end-to-end wall-clock time (Appendix A.11). This cost is included in the wall-clock results above.