acceptodds
Under review as a conference paper at ICLR 2027

Many Caches, Same Reasoning: Rethinking Importance Selection for Reasoning-Time KV Compression

Abstract

KV-cache compression for long-form reasoning is usually framed as an importance-ranking problem: score past states, retain those judged most useful, and discard the rest. We ask whether successful retention policies actually agree on what those states are. Holding the full prefill cache fixed, we compare a signal-free page-level random retention baseline with R-KV, VaSE, and TriAttention across four reasoning models and three domains. Every compressed method therefore solves the same generated-state selection problem. Page-level random retention reaches 69.58% macro accuracy, defined as the unweighted mean across the 12 model-task cells, versus 70.45% for TriAttention, while exceeding R-KV and VaSE by 3.02 and 1.71 percentage points, respectively; its accuracy is within three percentage points of TriAttention in 11/12 model-task settings. More strikingly, independently successful random runs do not converge on the same cache. Across 2,000 multi-seed trajectories, their chance-adjusted overlap is (95% CI ) at comparable states, indistinguishable from independent random selection. Strong importance-based selectors likewise retain substantially different states, with adjusted pairwise overlaps of only 0.123–0.183; matched shared-core interventions reveal at most partial, asymmetric core effects rather than one common solution. These results suggest that reasoning-time KV compression is governed less by recovery of one privileged subset than by a broad space of distinct cache configurations that can preserve successful computation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, long-form reasoning, cache eviction, inference efficiency, large language models, memory management, test-time compute, paged KV memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6uHzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether successful reasoning-time KV cache compression requires identifying a uniquely important subset of states. With the prompt cache preserved and generated-state budgets matched, random page retention performs competitively with importance-based methods across four models and three reasoning domains. Analyses of successful random runs, selector overlap, and shared-core interventions suggest that multiple distinct cache configurations can support successful reasoning, although the mechanistic evidence is local to the evaluated states.

Critical strengths
  1. The research question is insightful. Beyond comparing accuracy under compression, the paper asks whether successful reasoning depends on a uniquely important cache subset, offering a useful perspective on the role of importance-based selection.

  2. The main experiments are well controlled. All compression methods preserve the full prompt and receive the same generated-state budget, reducing confounding from protection policies and cache capacity. The evaluation covers four models and three reasoning domains.

  3. The paper provides complementary mechanistic evidence. It combines multiple retention seeds, same-state cache overlap analysis, and shared-core interventions, while correcting for overlap expected from random sampling. This provides stronger support for solution multiplicity than accuracy comparisons alone.

Critical weaknesses
  1. Substantial overlap with Random Attention requires clarification. The proposed retention strategy closely resembles that of Random Attention [1]. The paper only employ block-level random selection instead of token-level ones. Moreover, all six shared Full-KV accuracy scores are identical, and the reported mean generation lengths for Qwen3-14B coincide across all three shared tasks. These observations do not establish inappropriate reuse, but the authors should clarify the relationship between the papers, disclose any shared experimental results or infrastructure, and explain the distinct contributions of this work.

  2. Presentation quality needs improvement. The budget table in Section 4.2 lacks a caption. The main overview figure appears to be rasterized; a vector version would improve readability and presentation quality.

  3. Experimental coverage and reproducibility details are limited. The capacity sweep and retention-granularity ablations are restricted to a single model–task setting. The paper also lacks end-to-end latency, peak memory, and throughput measurements, leaving the practical efficiency implications unquantified. Important implementation details, including the recent-window size and eviction-triggering rules, should be specified.

[1] Random Attention: Rethinking KV Cache Eviction for Efficient Reasoning

Questions for authors

Seeing Weaknesses. The authors should expain the relationship between the paper with Random Attention.

Reviewer_qTX6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper shows a random baseline for token-level KV cache compression only in decoding. It compares a content-free, page-level random retention policy with R-KV, VaSE and TriAttention across four reasoning models and three tasks. Random retention reaches 69.58% macro accuracy versus 70.45% for TriAttention.

Critical strengths
  • The unified protocol is useful in its own right. The paper shows that prefill handling alone can change accuracy by about 35 points, which is an important caution for comparisons in this area.
  • A strong random baseline is a valuable sanity check for the field.
Critical weaknesses
  1. Compression depth of correct samples.

Table 5 reports only whether random retention ever triggered eviction. On DeepSeek GPQA and LCB, with budgets of 4,048 and 10,640, 27% and 39% of samples are never evicted, yet these cells count in the macro average; the mean lengths also include long incorrect samples that hit the 32K cap (53% on DeepSeek LCB). Among samples that Full KV answers correctly, what fraction finish within the budget? Please report the distribution of output length / budget for Full-correct samples, and compare random retention with TriAttention within each range.

  1. No StreamingLLM-style baseline.

All methods pin the full prompt and share a fixed recent window that counts toward B (p4, p6); only the remaining discretionary part differs. With the prompt acting as the attention sink, this setting is close to StreamingLLM, which is cited (p2) but not compared. The recent-window length is not given anywhere, and the ablations are "intentionally narrow" (p7), covering only page size and prefill. Appendix D.2 shows only 32 discretionary pages (512 tokens) at the first eviction. What fraction of B is the recent window? Please add a baseline that keeps only the pinned prompt and the recent window, and ablate the window length, to separate "selection does not matter" from "recency carries most of the accuracy".

  1. The conclusion is broader than the methods tested.

The title and conclusion address "importance selection" in general, but only three token-level heuristic scorers are tested. Learned or head-level methods, several of which report strong importance signals, are not tested: DuoAttention and ForesightKV/SeerAttention-R are cited, while RazorAttention, HeadKV, reasoning-oriented head selection (e.g., Du et al., 2025) and Fast KVzip are not. The relation to native sparse attention with learned top-k selection (NSA, DSA) is not discussed. Can the claim be narrowed to heuristic token-level scoring in this budget regime, or can at least one learned or head-level method be added?

Questions for authors

Please address the weaknesses above.

Reviewer_akmSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at whether KV-cache compression really needs to keep a specific set of important states. Across four reasoning models and three benchmarks, randomly keeping cache pages works nearly as well as the strongest method tested, as long as the prompt cache is kept. Successful runs also keep quite different states, suggesting there is more than one way to retain enough information for a correct answer.

Critical strengths
  1. The random baseline is simple but works surprisingly well. This makes it worth asking how much the more complicated methods actually add.
  2. The comparison seems fair, since all methods have the same cache budget and can remove the same types of states.
Critical weaknesses
  1. Could different successful subsets preserve the same intermediate information? Comparing their content or attention outputs at the same generation point would help explain the low overlap.

  2. Importance-based methods do not necessarily need to find one unique subset. It is unclear whether the results challenge importance ranking itself or mainly show that random selection is a strong baseline.

  3. How would the methods compare on tasks where a specific intermediate result must be remembered and reused much later?

  4. The tighter-budget comparison is only shown for Qwen3-4B on MATH-500. Does the same pattern hold for other models, especially DeepSeek/MATH, where TriAttention performs better?

  5. The results depend on keeping the full prompt cache. How much total memory is actually saved when this preserved cache is included, particularly for long prompts?

Questions for authors
  1. Check the weaknesses.
  2. How does random retention compare with keeping only the prompt and recent window? This would help show how much the older generated states actually contribute.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.