acceptodds
Under review as a conference paper at ICLR 2027

KV Cache Eviction as System One Memory Control for System Two Reasoning

Abstract

Long-chain reasoning expands the key–value (KV) cache, forcing memory-allocation decisions before the future roles of intermediate reasoning states are clear. We frame KV cache eviction as System One memory control for System Two reasoning: satisfying a fixed memory budget while avoiding premature semantic commitments. We propose a framework that uses relations within the evolving reasoning trajectory to selectively guide retention, rather than assigning a fine-grained importance score to every cached entry. It identifies active dependencies and explicitly superseded reasoning, while withholding semantic preferences when their roles remain unresolved. These judgments serve as revisable constraints on budget-driven eviction rather than direct deletion commands, with a simple stochastic policy resolving the remaining allocation decisions. Semantic audit runs asynchronously with generation, keeping the eviction path lightweight. Experiments on mathematical reasoning benchmarks across multiple reasoning models show higher mean accuracy than score-based and prompt-protected random eviction at matched cache budgets. An efficiency evaluation further shows reduced cache memory relative to full attention, with little additional eviction-path overhead over random eviction. These results suggest that selective, revisable semantic guidance can improve memory allocation for extended reasoning without requiring exhaustive importance estimation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, long reasoning, cache eviction, efficient inference, reasoning-time memory management, semantic memory control, System-One models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SLc7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses KV-cache eviction for long chain-of-thought reasoning under a fixed memory budget. Its argument is that the usefulness of a reasoning state is set-conditional and changes as reasoning proceeds. It therefore avoids scoring every cached entry and uses sparse, revisable semantic constraints instead. The method is called SPARE.

SPARE splits the generated trace online into episodes of 128–256 tokens. An external “System One” model (Jev, called through an API) periodically audits the episodes and labels each one PROTECT, OBSOLETE, or UNCONSTRAINED. Eviction then follows a tiered uniform lottery. Protected episodes are kept up to a capped budget. Obsolete episodes are evicted first. The rest are sampled uniformly, as in the Random Attention baseline. Audits run asynchronously.

The experiments cover three models (Qwen3-8B, Qwen3-4B, Phi-4-reasoning) and four math benchmarks (MATH-500, AIME 2024, AIME 2025/26, HMMT 2025) at about 4× compression. Baselines are Full attention, SnapKV, R-KV, VaSE and Random Attention. The paper also reports:

  • Accuracy: SPARE is best or tied-best in most cells of Table 2.

  • Efficiency: the final KV footprint falls 3.8× with an eviction-path overhead of 0.2 s per trace over Random Attention, plus an external audit cost of about $0.05 per trace.

  • Motivating analyses: re-reference rates of low-scored episodes, a constructed redundancy probe, and correction-relation probes.

Critical strengths
  1. A simple, well-posed mechanism built on a strong baseline, with a fair shared protocol. SPARE reduces exactly to Random Attention when the controller abstains, and it adds only a tier offset to the same lottery. All compared methods share the same trigger, recency buffer, prompt pinning, seeds and forced-closure protocol, and the score-based baselines are reimplemented on the same hooks (Appendix A). The efficiency reporting is also straightforward. Asynchronous auditing adds 0.2 s per trace over Random Attention, while the blocking variant costs +26 s (Table 5). The results also hold as the budget shrinks (Fig. 2).

  2. Evidence that the semantic signal carries information about later use. Low-scoring or oldest episodes are re-referenced 36–50% of the time after at least 1K tokens (Table 3), which supports the claim that attention-based and age-based eviction discard states that are later used. The protected episodes in Table 4b are about 18× more costly to remove than random unconstrained episodes (−0.063 vs. −0.0035 per-token log-probability). This suggests the PROTECT labels identify load-bearing content.

Critical weaknesses
  1. The gains cannot be attributed to the semantic labels, and the comparison is not resource-matched. SPARE differs from Random Attention in more than the labels. It moves from token-level uniform sampling to episode-level protection of contiguous spans under a tiered lottery. It also uses an external LLM that reads the full trace, with inputs up to about 24K tokens, at about $0.05 per trace. The baselines get no such extra computation or information. The paper has no controls that separate these factors. Missing controls include random PROTECT/OBSOLETE labels at the same rates, protecting the most recent episodes, a PROTECT-only vs. OBSOLETE-only ablation, a cheap non-LLM rule using the revision cues already used in the audit trigger, and a different or smaller controller. Table 1b also shows direct deletion based on semantic judgments is much worse than random deletion (59.6 vs. 70.2), which suggests the labels are not reliable importance signals on their own. Only Jev is evaluated, even though CLM and general LLMs are named as alternatives. The settings in Table 1 (model, benchmark, sample size) are not specified, and the effects in Table 4b are tiny with no intervals (−0.0070 vs. −0.0038).

  2. The effect sizes are small relative to evaluation noise, and some reported numbers cannot be reproduced. Many gains over Random Attention are about one point or less, for example Qwen3-4B AIME 2024 (+0.8) and Phi-4 AIME 2024 (−0.7), on 30–60-problem sets where one problem is worth 3.3 or 1.7 points, with only 2–4 seeds. Table 2 reports no intervals, and the paired bootstrap and Wilcoxon results mentioned in Sec. 3.1 are not shown in the main text. The text says Qwen3-8B average accuracy rises from 75.9% (Random Attention) to 81.6% (SPARE). Averaging the four Table 2 columns for Qwen3-8B gives about 65.0 vs. 68.1 (about 82.4 vs. 84.4 weighted by problem count), so I could not reproduce those figures. R-KV (84.9) and VaSE (85.1) fall more than five points below Random Attention (91.0) on MATH-500, which is hard to reconcile with their published claims. The paper does not show that its reimplementations reproduce reported results. Reasoning length and forced-termination rate are said to be measured but are not reported in the main text. The largest gain over Random Attention is a single cell, Qwen3-8B AIME 2025/26 (+7.1).

Questions for authors
  1. Can you add controls with the same budget and episode structure but random labels, protect-recent-episodes labels, PROTECT-only, OBSOLETE-only, and a non-LLM heuristic using revision cues? Can you report results with an open-weight or smaller controller and with other thresholds (ρ_P = 0.5, the 0.85 confidence gate, T, episode lengths)?

  2. Can you report paired intervals and p-values for every cell of Table 2, with per-seed results? Can you explain how the Qwen3-8B averages of 75.9% and 81.6% were computed? Can you extend the evaluation to more problems or other domains (e.g., science QA or code), since the AIME and HMMT sets are small?

  3. Do your reimplementations of SnapKV, R-KV and VaSE reproduce the published numbers under their original protocols? Why are R-KV and VaSE more than five points below Random Attention on MATH-500?

  4. What are the settings behind Table 1 (model, benchmark, n)? How is “re-reference” defined in Table 3, and what are the populations and intervals behind Table 4b?

  5. How reproducible is the pipeline given a proprietary endpoint? With asynchronous audits, the labels in force depend on API latency. How much do results vary across repeated runs, endpoint versions, and a synchronous-audit variant?

4 comments

We sincerely thank the reviewer for the detailed and constructive feedback, particularly for recognizing the motivation behind relational semantic memory control, the simplicity of our eviction mechanism, and the evidence supporting the evolving utility of reasoning states. We have conducted additional experiments and analyses to clarify the contributions of semantic guidance, strengthen statistical evaluation, and address reproducibility and computational costs.

[W1, Q1] Semantic contributions and resource-matched controls

We evaluated resource-matched ablations on Qwen3-8B (K=4096, r=64, pinned prompt KV) with the original Jev controller and Decisions, an OpenAI GPT-6-Luna-based controller that produces structured semantic decisions without free-form reasoning generation.

Beyond PROTECT-only (P+U), OBSOLETE-only (O+U), and Full SPARE (P+O+U), we evaluated matched random episode protection, recency-based protection, revision-cue heuristics, and forced semantic selection baselines.

AIME 2025/26 accuracy (%), K=4096

MethodJevDecisions
Random Attention54.254.2
Matched Random Protection57.558.8
Recent Episode Protection56.760.0
Revision-Cue Heuristic58.360.0
Selective P+U61.363.3
Selective O+U57.960.4
Full SPARE61.364.6
Forced P-only52.959.2
Forced O-only52.149.2

For the non-LLM controls (matched random, recent-episode, and revision-cue protection), the Jev and Decisions columns represent separate matching targets rather than different heuristic controllers. Each control matches the per-audit protection-token targets of its corresponding Full SPARE run under the same problem and seed. Audit timing follows each control's own trajectory, and exact protection counts may differ slightly due to whole-episode selection. Matched random protection additionally matches protection turnover, whereas the recency and revision-cue heuristics do not. The differences between the two matching conditions were not statistically significant for either heuristic.

These experiments clarify three aspects of the proposed protocol.

First, selective PROTECT is the primary source of accuracy improvements. Unlike Random Attention, matched random protection preserves randomly selected episodes with a matched protection budget before applying stochastic eviction to the remainder. While this structural protection already provides improvements over RA, selective P+U achieves higher accuracy with both controllers (61.3% and 63.3%). Recent-episode and revision-cue heuristics are also competitive, but remain below selective P+U in these comparisons. We acknowledge that the differences between semantic selection and individual heuristic controls are not statistically decisive in every setting.

Second, OBSOLETE offers a modest additional benefit depending on the controller. Jev reports the same accuracy for P+U and Full SPARE (61.3%), whereas Decisions improves from 63.3% to 64.6% when OBSOLETE is included. This indicates that the usefulness of semantic retirement may be sensitive to the quality of supersession judgments.

Third, forced semantic selection highlights the importance of abstention. Forced P-only uses a binary protection-oriented prompt to select the entire retention budget, while Forced O-only uses a binary obsolescence-oriented prompt to select all eviction candidates. Both underperform the selective policies, particularly Forced O-only. Since these baselines also change the decision prompts, we interpret the comparison as evidence for selective semantic control rather than an isolated causal estimate of abstention alone.

We further examined sensitivity to the protection cap (ρP=0.125,0.25,0.5\rho_P=0.125, 0.25, 0.5), OBSOLETE confidence gate (0, 0.5, 0.85), and audit interval (T=512,1024,2048T=512, 1024, 2048), with the default configurations specified in Appendix A.9. In our Qwen3-8B AIME 2025/26 sensitivity runs, none of the tested alternatives showed a statistically significant improvement over the default (e.g., ρP=0.25\rho_P=0.25: −3.3-3.3 pp, p=.55p=.55; ρP=0.125\rho_P=0.125: −2.5-2.5 pp, p=.69p=.69; OBSOLETE gate 0.50.5: −1.7-1.7 pp, p=.83p=.83).

[W2, Q2] Statistical reliability, corrected averages, and evaluation scope

Thank you for catching the typo in Section 3.2. We have corrected the reported Qwen3-8B averages in the revised manuscript; the per-benchmark results in Table 2 remain unchanged.

We have also strengthened the statistical evaluation and added a detailed paired analysis of Qwen3-8B on AIME 2025/26 to Appendix C, using 60 problems and four shared seeds (240 paired runs).

SPARE-Jev improves accuracy from 54.2% to 61.3% (+7.1 percentage points), with a problem-level bootstrap 95% CI of [+0.8, +13.3], sign-flip permutation p=.042, and Wilcoxon signed-rank p=.040. Both tests support nominal significance at the 5% level for this benchmark. We do not interpret this as a family-wise significance claim across all model–benchmark comparisons.

We also expanded selected evaluations to eight seeds. The corresponding P+U improvements over RA are +3.1 points for Jev (95% CI [−1.0, +6.9]) and +5.0 points for Decisions (95% CI [+1.7, +8.3]). These results highlight seed sensitivity and motivate reporting paired uncertainty rather than relying solely on individual accuracy differences. We will extend the paired statistical analysis to the remaining model–benchmark settings in the final revision.

To further assess generalization beyond mathematical reasoning, we additionally evaluated SPARE on GPQA Diamond, a graduate-level science reasoning benchmark covering physics, chemistry, and biology. Under the same KV-cache budget (K=4096) and inference protocol, SPARE-Jev achieves 61.1% accuracy compared with 56.6% for Random Attention (RA), an improvement of +4.5 percentage points, with an average of 13.5 audits per trace at an external cost of approximately $0.016 per trace.

These preliminary results provide encouraging evidence beyond mathematical reasoning. We will further evaluate three additional random seeds and report the complete four-seed results with paired statistical analysis in the final revision.


[W1, Q3] Baseline implementations and evaluation fairness

We appreciate the request to clarify the comparison with SnapKV, R-KV, and VaSE.

Our primary objective was to compare eviction policies under a shared inference protocol, with matched physical KV budgets, eviction triggers, recency buffers, and generation settings.

Importantly, we explicitly applied full prompt protection to all baselines, following the findings of Random Attention. As demonstrated in that work, allowing prompt tokens to be evicted can substantially degrade reasoning accuracy, while preserving them can improve the performance of existing eviction methods. We therefore incorporated this additional safeguard into SnapKV, R-KV, and VaSE, rather than evaluating them under potentially less favorable prompt-retention conditions.

Even with full prompt protection, R-KV and VaSE achieve 84.9% and 85.1% on Qwen3-8B MATH-500, compared with 91.0% for Random Attention. Thus, the remaining performance gaps cannot be attributed to prompt eviction.

We emphasize that these results reflect a controlled comparison under our shared protocol, rather than an exact reproduction of each method's originally published settings. The implementation details and evaluation configurations are provided in Appendix A.

[W1, Q4] Experimental settings and interpretation of Tables 1, 3, and 4

We appreciate the request for additional experimental details. We revisited the underlying intervention and auditing logs to clarify the evaluation settings, populations, and statistical uncertainty.

Table 1 (causal sensitivity and direct semantic deletion). Table 1a uses Qwen3-4B in thinking mode, with 42 fixed reasoning traces from AIME 2024/2025 and MATH-500 (level 5). Among 1,525 candidate spans, 22 were identified as harmful (Δlog⁡p≥0.3\Delta \log p \geq 0.3 nat) and 613 as harmless (Δlog⁡p≤0\Delta \log p \leq 0), with ambiguous cases excluded. The AUROC analysis uses these 635 labeled spans. The learned MLP detector was evaluated with five-fold trace-grouped out-of-fold predictions, while the attention- and age-based scores were directly evaluated without training. Since this analysis uses development traces, we regard it as exploratory evidence motivating the method.

Table 1b separately evaluates Qwen3-8B on 47 paired problem–seed runs. Direct deletion of controller-identified OBSOLETE spans achieves 59.6% accuracy, versus 70.2% for size-matched random deletion. The paired difference is −10.6 percentage points, with 95% CI [−27.7, +6.4]. Thus, the observed direction motivates caution against unconditional semantic deletion, although the difference is not statistically established at this sample size.

Table 3 (re-reference). We analyzed 106 full-attention Qwen3-8B AIME 2025/26 traces, avoiding eviction-induced survival bias. Re-reference is measured as the later reappearance of a numerical value first introduced in an episode, excluding values already present in the problem statement.

The analysis includes 98,611 UNCONSTRAINED episode–audit pairs, of which 15,611 contain eligible numerical values. Re-reference rates are computed over these eligible pairs at delays of at least 1K, 2K, and 4K generated tokens. For example, the oldest quartile shows a 49.6% re-reference rate after 1K tokens, while the lowest-confidence quartile shows 36.2%. A shuffled-future negative control yields only 1–3%.

We emphasize that numerical reappearance is an observable proxy for subsequent reuse, rather than direct evidence that the model retrieved the corresponding KV entries.

Table 4b (deletion sensitivity). The analysis uses 148 verified correction relations from 33 reasoning traces. At the first post-correction checkpoint, deleting PROTECT episodes produces a substantially larger mean next-window log-probability decrease (−0.0627, n=134) than size-matched random UNCONSTRAINED controls (−0.0035).

The paired PROTECT-minus-random difference is −0.059, with 95% CI [−0.076, −0.044] and a trace-cluster bootstrap CI of [−0.086, −0.035]. These results provide evidence that PROTECT identifies computationally sensitive reasoning states, even when accounting for correlations within traces.

The approximately 18× difference reflects a ratio of mean deletion effects. Since the distribution is heavy-tailed, we also examined median effects (−0.0063 for PROTECT versus −0.0008 for random controls). The smaller correction-related differences show greater uncertainty in their mean effects and are substantially smaller in magnitude than the PROTECT contrast.

These findings reinforce the distinction motivating SPARE: semantic supersession does not guarantee causal dispensability, while selectively protected reasoning states exhibit greater sensitivity to KV removal.


[W1, Q5] Controller reproducibility, asynchronous auditing, and inference costs

SPARE specifies the controller version (Jev 1.13), input construction, decision criteria, label conversion, and audit schedule in Appendices A–B. During asynchronous execution, eviction uses the latest completed audit, with measured label staleness of approximately 40 tokens (p95: 72). We retain episode-level decisions and applied labels for reproducibility.

We measured complete inference costs on an isolated RTX PRO 6000 GPU:

MetricRAJev (sync)Jev (async)
Mean e2e time (s/trace)349365324
Audit calls/trace029.928.2
API cost/trace$0$0.056$0.052

Asynchronous auditing substantially reduces blocking overhead while maintaining an external cost of approximately $0.05 per trace. The alternative Decisions controller exhibits comparable synchronous/asynchronous latency (366/324 s/trace), with approximately 2.2× higher synchronous API cost. End-to-end timing differences also reflect variations in reasoning length.

Since live controller responses may vary across repeated executions, we record controller outputs and applied constraints rather than assuming deterministic behavior under fixed decoding seeds.

Summary of Addressed Concerns

Thank you again for your detailed feedback. We summarize the additional experiments, analyses, and supporting evidence below.

#IssueResponse SummaryEvidence
1[W1, Q1] Semantic contributions and resource-matched controlsAdded matched random, recent, revision-cue, P+U/O+U, and forced-selection controls with two controllers. Full SPARE improves over RA, while forced selection underperforms, highlighting selective semantic control and abstention.[W1, Q1], Appendix A.9
2[W2, Q2] Statistical validation and broader evaluationAdded paired CI and statistical tests (+7.1 pp, CI [+0.8, +13.3], p=.042/.040), extended to eight seeds, and evaluated GPQA Diamond (61.1% vs. 56.6% RA).[W2, Q2], Appendix C
3[W1, Q3] Baseline fairness and reproductionClarified the shared evaluation protocol, including full prompt protection for SnapKV, R-KV, and VaSE, and distinguished it from original-protocol reproduction.[W1, Q3], Appendix A
4[W1, Q4] Experimental settings and causal evidenceClarified Tables 1, 3, and 4 with sample sizes, operational definitions, and uncertainty estimates. PROTECT − random: −0.059, trace-cluster 95% CI [−0.086, −0.035].[W1, Q4]
5[W1, Q5] Controller reproducibility and inference costMeasured label staleness (40 tokens, p95 72), end-to-end latency (324 s/trace async vs. 349 RA), and external cost ($0.052/trace).[W1, Q5], Appendices A–B
Reviewer_Q9RYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SPARE which uses Jev as a backbone model to build a KV-cache eviction framework for long-chain reasoning.An auxiliary controller identifies active dependencies and superseded reasoning, while abstaining when evidence is insufficient. These revisable semantic constraints guide stochastic eviction, with audits running asynchronously. Experiments across three reasoning models on mathematical benchmarks show improved average accuracy over compared eviction baselines at matched cache budgets.

Critical strengths

1.The paper recognizes that reasoning states can remain computationally useful even after being semantically superseded, motivating a separation between semantic judgments and physical eviction. 2.Revisable semantic constraints guide stochastic eviction, while asynchronous auditing keeps the online eviction path lightweight. 3.Experiments cover three reasoning models under matched cache budgets and shared prompt protection, showing improved average accuracy over the compared eviction baselines.

Critical weaknesses

1.Figure 1 is not a pdf or svg. 2.The paper does not isolate the contributions of protection, obsolescence, abstention, and label revision. 3.The Qwen3-8B average accuracies stated in Section 3.2 do not match the macro averages from Table 2. Confidence intervals and significance-test results promised in Section 3.1 are also not presented.

Questions for authors

1.How much does each component—protection, obsolescence, abstention, and label revision—contribute to the accuracy gains? Could you provide corresponding ablations? 2.What are the end-to-end latency and total resource costs when Jev’s auditing context and computation are included, compared with Random Attention? 3.How are the Qwen3-8B average accuracies in Section 3.2 computed, given their apparent discrepancy with Table 2? Could you also provide the confidence intervals and significance-test results mentioned in Section 3.1?

3 comments

We thank the reviewer for recognizing the motivation behind separating semantic judgments from physical eviction, as well as the benefits of revisable semantic constraints and asynchronous auditing. We have conducted additional experiments and analyses to address the reviewer's concerns.

[W1] Figure 1 format

Thank you for pointing this out. Figure 1 has been updated to vector PDF format in the revised manuscript.


[W2, Q1] Component contributions and semantic abstention

We conducted extensive component ablations under matched KV-cache budgets (K=4096, r=64, pinned prompt KV) on Qwen3-8B using both the original Jev controller and an alternative OpenAI GPT-6-Luna-based typed-decision controller (Decisions). The latter produces structured semantic decisions rather than free-form generated reasoning.

The experiments distinguish selective PROTECT (P+U), selective OBSOLETE (O+U), and the complete SPARE protocol (P+O+U). We additionally evaluate matched random episode protection and forced semantic selection baselines. Unlike Random Attention, matched random protection protects randomly selected episodes under a matched protection budget, with stochastic eviction applied to the remainder.

AIME 2025/26 accuracy (%), K=4096

MethodJevDecisions
Random Attention54.254.2
Matched Random Protection57.558.8
Selective P+U61.363.3
Selective O+U57.960.4
Full SPARE61.364.6
Forced P-only52.959.2
Forced O-only52.149.2

The results clarify the respective contributions:

First, selective PROTECT provides the largest accuracy improvements, outperforming Random Attention by +7.1 and +9.2 percentage points with Jev and Decisions, respectively. Matched random episode protection also provides gains, but remains below selective semantic protection in these comparisons, suggesting that both episode-level protection and semantic selection can contribute.

Second, OBSOLETE provides a smaller, controller-dependent contribution. O+U is less effective than P+U for both controllers. The reported Jev P+U and Full accuracies are both 61.3%, while Decisions shows a modest improvement from 63.3% to 64.6% (+1.3 percentage points). This suggests that the additional benefit of OBSOLETE may depend on the controller's ability to identify reliable, settled supersession relations.

Third, the forced semantic baselines highlight the importance of selective decisions and abstention. Forced P-only selects the entire retention budget using semantic protection scores, while Forced O-only fills the required eviction budget using obsolescence scores. Both underperform the selective policies, indicating that imposing semantic decisions on unresolved reasoning states can be detrimental. These baselines also use task-specific binary prompts, so their differences should be interpreted as evidence for selective control rather than an isolated causal estimate of abstention alone.

We also examined label revision through a separate no-revision ablation. Under the evaluated two-seed setting, both variants achieved comparable accuracy (Δ=0.0 percentage points, 95% CI [−5.8, +5.8]), suggesting that the measurable effect of revision is limited in this setting. Nevertheless, revisability remains part of the protocol for adapting semantic constraints as reasoning evolves.

We will include these additional ablations and clarify the individual roles of semantic protection, retirement, abstention, and revision.


[W3, Q3] Average accuracy and statistical significance

Thank you for catching the typo in Section 3.2. We have corrected the reported Qwen3-8B averages to 65.0% (RA) and 68.1% (SPARE), computed as equal-weight macro-averages across the four benchmarks. The individual results in Table 2 remain unchanged.

We have also added a paired statistical analysis in Appendix C for Qwen3-8B on AIME 2025/26 (60 problems × 4 seeds). SPARE-Jev improves accuracy from 54.2% to 61.3% (+7.1 percentage points), with a 95% bootstrap CI of [+0.8, +13.3], permutation p=.042, and Wilcoxon signed-rank p=.040. Both tests indicate nominal significance at the 5% level for this benchmark. We will extend this analysis to the remaining model–benchmark settings in the final revision.

[Q2] End-to-end latency and total auditing costs

We measured end-to-end inference time, including semantic auditing, on an isolated RTX PRO 6000 GPU. Random Attention requires 349 s/trace, compared with 365 s/trace for synchronous Jev and 324 s/trace for asynchronous Jev. The asynchronous variant incurs approximately 28.2 audit calls, 1.24M input tokens, and $0.052 in external API costs per trace.

The alternative Decisions controller shows comparable mean latency (366 s/trace synchronous; 324 s/trace asynchronous), although its measured synchronous API cost is approximately 2.2× that of Jev (\0.121versus0.121 versus \\0.056 per trace). These results account for the external auditing cost while demonstrating the benefit of overlapping audits with generation. Differences in generated reasoning lengths also affect the measured end-to-end times.

Summary of Addressed Concerns

Thank you again for your feedback. We summarize the key revisions and supporting evidence below.

#IssueResponse SummaryEvidence
1[W1] Figure 1 formatUpdated Figure 1 to vector PDF format in the revised manuscript.[W1], Revised PDF
2[W2, Q1] Component and revision ablationsEvaluated P+U, O+U, Full SPARE, matched random, and forced selection with two controllers. Full SPARE improves over RA, while forced selection underperforms. Label revision was also evaluated.[W2, Q1]
3[W3, Q3] Statistical validationAdded paired confidence intervals, permutation tests, and Wilcoxon tests, supporting the AIME improvement (+7.1 pp, CI [+0.8, +13.3], p=.042/.040).[W3, Q3], Appendix C
4[Q2] End-to-end auditing costsMeasured full inference latency and external auditing costs. Async Jev: 324 s/trace vs. 349 for RA, with 28.2 audits and $0.052/trace.[Q2]
Reviewer_qGFyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV cache eviction during extended reasoning, where the relevance of intermediate states may change as reasoning progresses. The authors introduce SPARE, which employs an auxiliary semantic controller to classify reasoning episodes as PROTECT, OBSOLETE, or UNCONSTRAINED. These judgments guide a stochastic eviction policy while preserving a fixed KV budget. Semantic auditing runs asynchronously with generation. Experiments on three reasoning models and math reasoning benchmarks report improved accuracy over existing eviction strategies, including Random Attention, under matched cache budgets.

Critical strengths
  1. The distinction between semantic relevance and causal importance is worth examining. In particular, Table 1 shows that direct semantic deletion can perform considerably worse than random eviction, while Table 4 investigates how the impact of removing reasoning states changes with subsequent deliberation. These observations motivate a more selective intervention strategy.
  2. SPARE builds upon a simple stochastic eviction policy and introduces semantic constraints only where the controller identifies dependencies or superseded reasoning. This design avoids exhaustive token importance estimation and provides a clear implementation of the proposed idea.
  3. The evaluation uses matched KV budgets across three reasoning models. Specifically, the improvement over Random Attention is particularly visible on Qwen3 8B for AIME 2025/26, where accuracy increases from 54.2% to 61.3%. The shrinking budget experiment also suggests that semantic guidance becomes more beneficial under stronger memory constraints.
Critical weaknesses
  1. Although SPARE combines protection, obsolescence detection, and semantic abstention, the experiments do not establish how much each component contributes to the final improvement. Table 4 investigates the causal sensitivity of selected memory states, but it does not determine whether the complete three-state protocol is necessary. Given the existing work on semantic and reasoning aware KV management, this distinction is central to assessing the methodological contribution. A controlled comparison of PROTECT only, PROTECT with OBSOLETE, and the complete SPARE policy, ideally accompanied by random episode protection under a matched budget, would address this question directly. Overall, the contribution of the proposed semantic control protocol remains insufficiently isolated.

  2. Section 3.1 states that paired bootstrap confidence intervals and Wilcoxon signed-rank tests are reported, yet these statistics are not presented in the available results. Several improvements over Random Attention are relatively modest, making their uncertainty important to the interpretation. Furthermore, Section 3.2 reports an average improvement from 75.9% to 81.6% for Qwen3 8B, whereas the four task results in Table 2 yield approximately 65.0% and 68.1% under equal task weighting. If another aggregation is intended, its definition should be provided.

  3. Table 5 reports an external auditing cost of approximately $0.052 per reasoning trace, while asynchronous SPARE achieves 58.2 tokens/s compared with 59.6 tokens/s for Random Attention. Although asynchronous execution reduces blocking overhead, the additional semantic computation remains part of the complete system. Less clear to me is whether the resulting accuracy improvement justifies this cost, particularly if a simpler protection heuristic could achieve comparable performance. The existing audit call statistics and the complete inference workflow might need additional accounting.

Questions for authors

Q1. How much of the improvement comes from PROTECT alone? Can the authors compare this with PROTECT plus OBSOLETE and the complete SPARE policy under the same budget?

Q2. Can the authors provide the paired statistical results mentioned in Section 3.1 and reconcile the average accuracy reported in Section 3.2 with Table 2?

Q3. Would audit call counts and complete inference cost measurements be available?

3 comments

We sincerely thank the reviewer for the constructive feedback and for recognizing the motivation behind separating semantic relevance from causal importance. To further clarify the contributions of individual semantic constraints, we have conducted additional component analyses and more detailed evaluations of statistical uncertainty and inference costs.

[W1, Q1] Component contributions and the necessity of selective semantic control

We agree that the respective roles of PROTECT, OBSOLETE, and UNCONSTRAINED require controlled evaluation.

We therefore conducted component ablations under identical KV-cache budgets (K=4096, recency buffer r=64, and pinned prompt KV) on Qwen3-8B using both the original Jev controller and an alternative OpenAI GPT-6-Luna-based typed-decision controller (Decisions). The latter produces structured semantic decisions rather than free-form generated reasoning.

We distinguish three selective policies:

  • P+U: Retain only PROTECT constraints; all other episodes follow stochastic eviction.
  • O+U: Apply only confident OBSOLETE constraints; all other episodes follow stochastic eviction.
  • Full SPARE: Apply PROTECT, OBSOLETE, and UNCONSTRAINED as originally proposed.

We additionally evaluate matched random episode protection and forced semantic selection baselines. Unlike Random Attention, which randomly selects individual eligible KV entries, matched random protection first protects randomly selected episodes with a matched protection budget and applies stochastic eviction to the remainder.

AIME 2025/26 accuracy (%), K=4096

MethodJevDecisions
Random Attention54.254.2
Matched Random Protection57.558.8
Selective P+U61.363.3
Selective O+U57.960.4
Full SPARE61.364.6
Forced P-only52.959.2
Forced O-only52.149.2

These results provide three insights.

First, selective PROTECT is the primary contributor to accuracy improvements. P+U improves over Random Attention by 7.1 percentage points with Jev and 9.2 percentage points with Decisions. Matched random episode protection also improves performance, but remains below selective P+U in these four-seed comparisons, suggesting that both episode-level protection and semantic selection can contribute.

Second, OBSOLETE has a more limited, controller-dependent contribution. O+U is less effective than P+U for both controllers. The reported Jev P+U and Full accuracies are both 61.3%, while Decisions shows a modest improvement from 63.3% to 64.6% (+1.3 percentage points). This suggests that the additional benefit of OBSOLETE may depend on the controller's ability to identify reliable, settled supersession relations.

Third, we directly investigate why semantic abstention matters. Unlike SPARE, forced semantic baselines require the controller to fill the entire retention or eviction budget using semantic rankings. We test both protection-oriented (Forced P-only) and obsolescence-oriented (Forced O-only) selection, using corresponding binary decision prompts.

As shown in the table, both forced approaches underperform selective P+U and Full SPARE with both controllers. In particular, forcing the controller to select all eviction candidates based on obsolescence judgments substantially degrades accuracy. Although these forced baselines also change the decision prompts, the results support the importance of avoiding premature semantic commitments rather than assigning retention or eviction preferences to every reasoning state.

These findings support the distinction underlying SPARE: semantic guidance is most useful when applied selectively, while forcing semantic preferences onto unresolved reasoning states can substantially impair performance.

We will add the ablation definitions, complete results, and controller-specific analysis to the revised manuscript.

[W2, Q2] Statistical significance and the average-accuracy discrepancy

Thank you for catching the typo in Section 3.2. The figures 75.9% and 81.6% were inadvertently misstated in the narrative text and have now been corrected to 65.0% (Random Attention) and 68.1% (SPARE), respectively, representing the equal-weight macro-averages across the four benchmarks in Table 2. The per-benchmark results in Table 2 remain unchanged. We appreciate the correction.

We have also expanded the statistical evaluation using paired problem–seed outcomes, problem-level bootstrap 95% confidence intervals, and paired significance tests. We have added a detailed statistical analysis of Qwen3-8B on AIME 2025/26 in Appendix C, including accuracy differences, confidence intervals, permutation tests, and Wilcoxon signed-rank tests.

Specifically, the originally reported SPARE-Jev improvement (54.2% to 61.3%) corresponds to a paired improvement of +7.1 percentage points, with 95% CI [+0.8, +13.3], permutation p=.042, and Wilcoxon signed-rank p=.040. The confidence interval excludes zero, and both paired tests reach nominal significance at the 5% level, providing consistent statistical support for the observed improvement on this benchmark.

We will further extend the paired statistical analysis to the remaining model–benchmark settings in the final revision.


[W3, Q3] Audit calls, end-to-end latency, and overall inference cost

We measured the complete inference workflow on an isolated RTX PRO 6000 GPU, including external audit calls, input-token usage, API cost, and end-to-end latency.

MetricRandom AttentionSPARE-Jev (sync)SPARE-Jev (async)
Mean e2e time (s/trace)349365324
Median e2e time (s/trace)287311272
p95 e2e time (s/trace)638696615
Audit calls/trace029.928.2
Audit input tokens/trace01.33M1.24M
API cost/trace$0$0.056$0.052

Asynchronous Jev reduces the measured mean end-to-end latency from 365 to 324 s/trace, while maintaining an external audit cost of approximately $0.05 per trace. Random Attention takes 349 s/trace on average, although differences in reasoning lengths also affect these measurements.

The alternative Decisions controller exhibits nearly identical synchronous and asynchronous mean end-to-end latency to Jev (366/324 vs. 365/324 s/trace). Its measured synchronous external API cost is approximately 2.2× higher (\0.121vs.0.121 vs. \\0.056 per trace).

We will include these complete measurements in the revised efficiency analysis.

Summary of Addressed Concerns

Thank you again for your constructive feedback. We briefly summarize the additional evidence addressing your comments.

#IssueRebuttal SummaryEvidence
1[W1, Q1] Component contributions and semantic abstentionEvaluated P+U, O+U, Full SPARE, matched random protection, and forced selection. Full SPARE improves over RA (61.3%/64.6% vs. 54.2%), while forced selection underperforms, supporting selective semantic control and abstention.[W1, Q1]
2[W2, Q2] Statistical validationAdded paired bootstrap 95% CI, permutation, and Wilcoxon tests. SPARE-Jev achieves +7.1 pp, CI [+0.8, +13.3], with p=.042/.040.[W2, Q2], Appendix C
3[W3, Q3] End-to-end inference costMeasured complete latency, audit calls, and API costs. Async Jev: 324 s/trace vs. 349 for RA, at $0.052/trace.[W3, Q3]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.