- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses KV-cache eviction for long chain-of-thought reasoning under a fixed memory budget. Its argument is that the usefulness of a reasoning state is set-conditional and changes as reasoning proceeds. It therefore avoids scoring every cached entry and uses sparse, revisable semantic constraints instead. The method is called SPARE.
SPARE splits the generated trace online into episodes of 128–256 tokens. An external “System One” model (Jev, called through an API) periodically audits the episodes and labels each one PROTECT, OBSOLETE, or UNCONSTRAINED. Eviction then follows a tiered uniform lottery. Protected episodes are kept up to a capped budget. Obsolete episodes are evicted first. The rest are sampled uniformly, as in the Random Attention baseline. Audits run asynchronously.
The experiments cover three models (Qwen3-8B, Qwen3-4B, Phi-4-reasoning) and four math benchmarks (MATH-500, AIME 2024, AIME 2025/26, HMMT 2025) at about 4× compression. Baselines are Full attention, SnapKV, R-KV, VaSE and Random Attention. The paper also reports:
-
Accuracy: SPARE is best or tied-best in most cells of Table 2.
-
Efficiency: the final KV footprint falls 3.8× with an eviction-path overhead of 0.2 s per trace over Random Attention, plus an external audit cost of about $0.05 per trace.
-
Motivating analyses: re-reference rates of low-scored episodes, a constructed redundancy probe, and correction-relation probes.
Critical strengths
-
A simple, well-posed mechanism built on a strong baseline, with a fair shared protocol. SPARE reduces exactly to Random Attention when the controller abstains, and it adds only a tier offset to the same lottery. All compared methods share the same trigger, recency buffer, prompt pinning, seeds and forced-closure protocol, and the score-based baselines are reimplemented on the same hooks (Appendix A). The efficiency reporting is also straightforward. Asynchronous auditing adds 0.2 s per trace over Random Attention, while the blocking variant costs +26 s (Table 5). The results also hold as the budget shrinks (Fig. 2).
-
Evidence that the semantic signal carries information about later use. Low-scoring or oldest episodes are re-referenced 36–50% of the time after at least 1K tokens (Table 3), which supports the claim that attention-based and age-based eviction discard states that are later used. The protected episodes in Table 4b are about 18× more costly to remove than random unconstrained episodes (−0.063 vs. −0.0035 per-token log-probability). This suggests the PROTECT labels identify load-bearing content.
Critical weaknesses
-
The gains cannot be attributed to the semantic labels, and the comparison is not resource-matched. SPARE differs from Random Attention in more than the labels. It moves from token-level uniform sampling to episode-level protection of contiguous spans under a tiered lottery. It also uses an external LLM that reads the full trace, with inputs up to about 24K tokens, at about $0.05 per trace. The baselines get no such extra computation or information. The paper has no controls that separate these factors. Missing controls include random PROTECT/OBSOLETE labels at the same rates, protecting the most recent episodes, a PROTECT-only vs. OBSOLETE-only ablation, a cheap non-LLM rule using the revision cues already used in the audit trigger, and a different or smaller controller. Table 1b also shows direct deletion based on semantic judgments is much worse than random deletion (59.6 vs. 70.2), which suggests the labels are not reliable importance signals on their own. Only Jev is evaluated, even though CLM and general LLMs are named as alternatives. The settings in Table 1 (model, benchmark, sample size) are not specified, and the effects in Table 4b are tiny with no intervals (−0.0070 vs. −0.0038).
-
The effect sizes are small relative to evaluation noise, and some reported numbers cannot be reproduced. Many gains over Random Attention are about one point or less, for example Qwen3-4B AIME 2024 (+0.8) and Phi-4 AIME 2024 (−0.7), on 30–60-problem sets where one problem is worth 3.3 or 1.7 points, with only 2–4 seeds. Table 2 reports no intervals, and the paired bootstrap and Wilcoxon results mentioned in Sec. 3.1 are not shown in the main text. The text says Qwen3-8B average accuracy rises from 75.9% (Random Attention) to 81.6% (SPARE). Averaging the four Table 2 columns for Qwen3-8B gives about 65.0 vs. 68.1 (about 82.4 vs. 84.4 weighted by problem count), so I could not reproduce those figures. R-KV (84.9) and VaSE (85.1) fall more than five points below Random Attention (91.0) on MATH-500, which is hard to reconcile with their published claims. The paper does not show that its reimplementations reproduce reported results. Reasoning length and forced-termination rate are said to be measured but are not reported in the main text. The largest gain over Random Attention is a single cell, Qwen3-8B AIME 2025/26 (+7.1).
Questions for authors
-
Can you add controls with the same budget and episode structure but random labels, protect-recent-episodes labels, PROTECT-only, OBSOLETE-only, and a non-LLM heuristic using revision cues? Can you report results with an open-weight or smaller controller and with other thresholds (ρ_P = 0.5, the 0.85 confidence gate, T, episode lengths)?
-
Can you report paired intervals and p-values for every cell of Table 2, with per-seed results? Can you explain how the Qwen3-8B averages of 75.9% and 81.6% were computed? Can you extend the evaluation to more problems or other domains (e.g., science QA or code), since the AIME and HMMT sets are small?
-
Do your reimplementations of SnapKV, R-KV and VaSE reproduce the published numbers under their original protocols? Why are R-KV and VaSE more than five points below Random Attention on MATH-500?
-
What are the settings behind Table 1 (model, benchmark, n)? How is “re-reference” defined in Table 3, and what are the populations and intervals behind Table 4b?
-
How reproducible is the pipeline given a proprietary endpoint? With asynchronous audits, the labels in force depend on API latency. How much do results vary across repeated runs, endpoint versions, and a synchronous-audit variant?
4 comments
[W2, Q2] Statistical reliability, corrected averages, and evaluation scope
Thank you for catching the typo in Section 3.2. We have corrected the reported Qwen3-8B averages in the revised manuscript; the per-benchmark results in Table 2 remain unchanged.
We have also strengthened the statistical evaluation and added a detailed paired analysis of Qwen3-8B on AIME 2025/26 to Appendix C, using 60 problems and four shared seeds (240 paired runs).
SPARE-Jev improves accuracy from 54.2% to 61.3% (+7.1 percentage points), with a problem-level bootstrap 95% CI of [+0.8, +13.3], sign-flip permutation p=.042, and Wilcoxon signed-rank p=.040. Both tests support nominal significance at the 5% level for this benchmark. We do not interpret this as a family-wise significance claim across all model–benchmark comparisons.
We also expanded selected evaluations to eight seeds. The corresponding P+U improvements over RA are +3.1 points for Jev (95% CI [−1.0, +6.9]) and +5.0 points for Decisions (95% CI [+1.7, +8.3]). These results highlight seed sensitivity and motivate reporting paired uncertainty rather than relying solely on individual accuracy differences. We will extend the paired statistical analysis to the remaining model–benchmark settings in the final revision.
To further assess generalization beyond mathematical reasoning, we additionally evaluated SPARE on GPQA Diamond, a graduate-level science reasoning benchmark covering physics, chemistry, and biology. Under the same KV-cache budget (K=4096) and inference protocol, SPARE-Jev achieves 61.1% accuracy compared with 56.6% for Random Attention (RA), an improvement of +4.5 percentage points, with an average of 13.5 audits per trace at an external cost of approximately $0.016 per trace.
These preliminary results provide encouraging evidence beyond mathematical reasoning. We will further evaluate three additional random seeds and report the complete four-seed results with paired statistical analysis in the final revision.
[W1, Q3] Baseline implementations and evaluation fairness
We appreciate the request to clarify the comparison with SnapKV, R-KV, and VaSE.
Our primary objective was to compare eviction policies under a shared inference protocol, with matched physical KV budgets, eviction triggers, recency buffers, and generation settings.
Importantly, we explicitly applied full prompt protection to all baselines, following the findings of Random Attention. As demonstrated in that work, allowing prompt tokens to be evicted can substantially degrade reasoning accuracy, while preserving them can improve the performance of existing eviction methods. We therefore incorporated this additional safeguard into SnapKV, R-KV, and VaSE, rather than evaluating them under potentially less favorable prompt-retention conditions.
Even with full prompt protection, R-KV and VaSE achieve 84.9% and 85.1% on Qwen3-8B MATH-500, compared with 91.0% for Random Attention. Thus, the remaining performance gaps cannot be attributed to prompt eviction.
We emphasize that these results reflect a controlled comparison under our shared protocol, rather than an exact reproduction of each method's originally published settings. The implementation details and evaluation configurations are provided in Appendix A.
[W1, Q4] Experimental settings and interpretation of Tables 1, 3, and 4
We appreciate the request for additional experimental details. We revisited the underlying intervention and auditing logs to clarify the evaluation settings, populations, and statistical uncertainty.
Table 1 (causal sensitivity and direct semantic deletion). Table 1a uses Qwen3-4B in thinking mode, with 42 fixed reasoning traces from AIME 2024/2025 and MATH-500 (level 5). Among 1,525 candidate spans, 22 were identified as harmful ( nat) and 613 as harmless (), with ambiguous cases excluded. The AUROC analysis uses these 635 labeled spans. The learned MLP detector was evaluated with five-fold trace-grouped out-of-fold predictions, while the attention- and age-based scores were directly evaluated without training. Since this analysis uses development traces, we regard it as exploratory evidence motivating the method.
Table 1b separately evaluates Qwen3-8B on 47 paired problem–seed runs. Direct deletion of controller-identified OBSOLETE spans achieves 59.6% accuracy, versus 70.2% for size-matched random deletion. The paired difference is −10.6 percentage points, with 95% CI [−27.7, +6.4]. Thus, the observed direction motivates caution against unconditional semantic deletion, although the difference is not statistically established at this sample size.
Table 3 (re-reference). We analyzed 106 full-attention Qwen3-8B AIME 2025/26 traces, avoiding eviction-induced survival bias. Re-reference is measured as the later reappearance of a numerical value first introduced in an episode, excluding values already present in the problem statement.
The analysis includes 98,611 UNCONSTRAINED episode–audit pairs, of which 15,611 contain eligible numerical values. Re-reference rates are computed over these eligible pairs at delays of at least 1K, 2K, and 4K generated tokens. For example, the oldest quartile shows a 49.6% re-reference rate after 1K tokens, while the lowest-confidence quartile shows 36.2%. A shuffled-future negative control yields only 1–3%.
We emphasize that numerical reappearance is an observable proxy for subsequent reuse, rather than direct evidence that the model retrieved the corresponding KV entries.
Table 4b (deletion sensitivity). The analysis uses 148 verified correction relations from 33 reasoning traces. At the first post-correction checkpoint, deleting PROTECT episodes produces a substantially larger mean next-window log-probability decrease (−0.0627, n=134) than size-matched random UNCONSTRAINED controls (−0.0035).
The paired PROTECT-minus-random difference is −0.059, with 95% CI [−0.076, −0.044] and a trace-cluster bootstrap CI of [−0.086, −0.035]. These results provide evidence that PROTECT identifies computationally sensitive reasoning states, even when accounting for correlations within traces.
The approximately 18× difference reflects a ratio of mean deletion effects. Since the distribution is heavy-tailed, we also examined median effects (−0.0063 for PROTECT versus −0.0008 for random controls). The smaller correction-related differences show greater uncertainty in their mean effects and are substantially smaller in magnitude than the PROTECT contrast.
These findings reinforce the distinction motivating SPARE: semantic supersession does not guarantee causal dispensability, while selectively protected reasoning states exhibit greater sensitivity to KV removal.
[W1, Q5] Controller reproducibility, asynchronous auditing, and inference costs
SPARE specifies the controller version (Jev 1.13), input construction, decision criteria, label conversion, and audit schedule in Appendices A–B. During asynchronous execution, eviction uses the latest completed audit, with measured label staleness of approximately 40 tokens (p95: 72). We retain episode-level decisions and applied labels for reproducibility.
We measured complete inference costs on an isolated RTX PRO 6000 GPU:
| Metric | RA | Jev (sync) | Jev (async) |
|---|---|---|---|
| Mean e2e time (s/trace) | 349 | 365 | 324 |
| Audit calls/trace | 0 | 29.9 | 28.2 |
| API cost/trace | $0 | $0.056 | $0.052 |
Asynchronous auditing substantially reduces blocking overhead while maintaining an external cost of approximately $0.05 per trace. The alternative Decisions controller exhibits comparable synchronous/asynchronous latency (366/324 s/trace), with approximately 2.2× higher synchronous API cost. End-to-end timing differences also reflect variations in reasoning length.
Since live controller responses may vary across repeated executions, we record controller outputs and applied constraints rather than assuming deterministic behavior under fixed decoding seeds.
Summary of Addressed Concerns
Thank you again for your detailed feedback. We summarize the additional experiments, analyses, and supporting evidence below.
| # | Issue | Response Summary | Evidence |
|---|---|---|---|
| 1 | [W1, Q1] Semantic contributions and resource-matched controls | Added matched random, recent, revision-cue, P+U/O+U, and forced-selection controls with two controllers. Full SPARE improves over RA, while forced selection underperforms, highlighting selective semantic control and abstention. | [W1, Q1], Appendix A.9 |
| 2 | [W2, Q2] Statistical validation and broader evaluation | Added paired CI and statistical tests (+7.1 pp, CI [+0.8, +13.3], p=.042/.040), extended to eight seeds, and evaluated GPQA Diamond (61.1% vs. 56.6% RA). | [W2, Q2], Appendix C |
| 3 | [W1, Q3] Baseline fairness and reproduction | Clarified the shared evaluation protocol, including full prompt protection for SnapKV, R-KV, and VaSE, and distinguished it from original-protocol reproduction. | [W1, Q3], Appendix A |
| 4 | [W1, Q4] Experimental settings and causal evidence | Clarified Tables 1, 3, and 4 with sample sizes, operational definitions, and uncertainty estimates. PROTECT − random: −0.059, trace-cluster 95% CI [−0.086, −0.035]. | [W1, Q4] |
| 5 | [W1, Q5] Controller reproducibility and inference cost | Measured label staleness (40 tokens, p95 72), end-to-end latency (324 s/trace async vs. 349 RA), and external cost ($0.052/trace). | [W1, Q5], Appendices A–B |
We sincerely thank the reviewer for the detailed and constructive feedback, particularly for recognizing the motivation behind relational semantic memory control, the simplicity of our eviction mechanism, and the evidence supporting the evolving utility of reasoning states. We have conducted additional experiments and analyses to clarify the contributions of semantic guidance, strengthen statistical evaluation, and address reproducibility and computational costs.
[W1, Q1] Semantic contributions and resource-matched controls
We evaluated resource-matched ablations on Qwen3-8B (K=4096, r=64, pinned prompt KV) with the original Jev controller and Decisions, an OpenAI GPT-6-Luna-based controller that produces structured semantic decisions without free-form reasoning generation.
Beyond PROTECT-only (P+U), OBSOLETE-only (O+U), and Full SPARE (P+O+U), we evaluated matched random episode protection, recency-based protection, revision-cue heuristics, and forced semantic selection baselines.
AIME 2025/26 accuracy (%), K=4096
For the non-LLM controls (matched random, recent-episode, and revision-cue protection), the Jev and Decisions columns represent separate matching targets rather than different heuristic controllers. Each control matches the per-audit protection-token targets of its corresponding Full SPARE run under the same problem and seed. Audit timing follows each control's own trajectory, and exact protection counts may differ slightly due to whole-episode selection. Matched random protection additionally matches protection turnover, whereas the recency and revision-cue heuristics do not. The differences between the two matching conditions were not statistically significant for either heuristic.
These experiments clarify three aspects of the proposed protocol.
First, selective PROTECT is the primary source of accuracy improvements. Unlike Random Attention, matched random protection preserves randomly selected episodes with a matched protection budget before applying stochastic eviction to the remainder. While this structural protection already provides improvements over RA, selective P+U achieves higher accuracy with both controllers (61.3% and 63.3%). Recent-episode and revision-cue heuristics are also competitive, but remain below selective P+U in these comparisons. We acknowledge that the differences between semantic selection and individual heuristic controls are not statistically decisive in every setting.
Second, OBSOLETE offers a modest additional benefit depending on the controller. Jev reports the same accuracy for P+U and Full SPARE (61.3%), whereas Decisions improves from 63.3% to 64.6% when OBSOLETE is included. This indicates that the usefulness of semantic retirement may be sensitive to the quality of supersession judgments.
Third, forced semantic selection highlights the importance of abstention. Forced P-only uses a binary protection-oriented prompt to select the entire retention budget, while Forced O-only uses a binary obsolescence-oriented prompt to select all eviction candidates. Both underperform the selective policies, particularly Forced O-only. Since these baselines also change the decision prompts, we interpret the comparison as evidence for selective semantic control rather than an isolated causal estimate of abstention alone.
We further examined sensitivity to the protection cap (ρP=0.125,0.25,0.5), OBSOLETE confidence gate (0, 0.5, 0.85), and audit interval (T=512,1024,2048), with the default configurations specified in Appendix A.9. In our Qwen3-8B AIME 2025/26 sensitivity runs, none of the tested alternatives showed a statistically significant improvement over the default (e.g., ρP=0.25: −3.3 pp, p=.55; ρP=0.125: −2.5 pp, p=.69; OBSOLETE gate 0.5: −1.7 pp, p=.83).