- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
ValueDiff proposes a training-free KV-cache eviction rule that retains tokens whose value vectors lie farthest from the cache mean. Motivated by an observed association between weaker attention sinks and greater relative value dispersion, the method avoids relying on observed attention weights. Experiments under continuous cache eviction show strong performance on several sink-suppressed architectures across long-context and mathematical reasoning tasks.
Critical strengths
- The method is simple, query-independent, and requires neither training nor auxiliary memory modules. Its low implementation complexity makes the empirical improvements practically appealing.
- The evaluation enforces bounded cache capacity during both prefill and generation. This provides a meaningful test of irreversible eviction decisions when later context and future queries remain unavailable.
- The paper evaluates diverse architectural families and includes matched key/value geometry comparisons, direction-versus-magnitude ablations, and an attention-weighted counterpart. These comparisons help identify which scoring components contribute to performance.
Critical weaknesses
-
The architectural explanation remains observational. Models differ simultaneously in training, attention structure, head sharing, gating, and local-context mechanisms. Consequently, weaker sinks and stronger ValueDiff performance may share other causes rather than reflect the proposed relationship directly.
-
The maximum-entropy derivation relies on assumptions that are not established by sink suppression. Removing persistent sink tokens does not imply that future attention is uniform or exchangeable; attention may remain highly selective over content. The derivation therefore motivates the score without showing that its assumptions describe the evaluated models.
-
Single-token eviction optimality does not justify retaining the highest-scoring set during bulk compression. Individually distinctive values may be mutually redundant, while removing many near-mean values can collectively discard important information. Repeated selection also changes the centroid on which subsequent decisions depend.
Questions for authors
Can controlled comparisons within a model family isolate sink suppression from other architectural differences and establish whether it predicts the advantage of value-based eviction?
How closely do actual future attention distributions satisfy these assumptions, and how accurately does ValueDiff predict reconstruction error as attention departs from the symmetric setting?
How large is the gap between independent ranking and joint subset selection, and does sequentially updating scores improve performance on clustered or multimodal value distributions?
3 comments
Response to Q2
Our primary objective is to preserve downstream task accuracy under a fixed cache budget. Reconstruction error is an analytical surrogate; a change in the attention output does not necessarily change the generated answer. We do not claim that centroid distance exactly predicts reconstruction error under general attention.
We directly compared ValueDiff and actual-output CAOTE using the same dense Q/K/V on 15 RULER-8K prompts per Qwen checkpoint: five each from retrieval, variable tracking and QA. Both policies use a 2K cache, sink1/recent128 protection and 128-token prefill chunks. Evictions are irreversible and applied after each chunk.
For an attention row , let indicate whether method retains token . The dense and masked outputs are
We normalize the pooled squared error by the same dense-output energy for both methods:
where contains eviction-active query/head/layer rows across the measured prompts. Lower reconstruction error is better. The separate downstream scores average all 13 RULER tasks at a 2K cache budget.
| Model | VD reconstruction | CAOTE reconstruction | VD RULER | CAOTE RULER |
|---|---|---|---|---|
| Qwen3-4B-Instruct | 5.95% | 4.81% | 91.20 | 72.62 |
| Qwen3.5-4B | 2.55% | 1.57% | 92.96 | 56.71 |
CAOTE has smaller local reconstruction error on both Qwen checkpoints, while ValueDiff achieves higher downstream accuracy. Lower attention-output reconstruction error does not necessarily translate into better downstream task performance. Query-adaptive sparse attention can select different KVs for each query from a retained cache. Token eviction, however, irreversibly removes KVs under a restricted cache budget and must preserve information that may be useful during future generation. This distinction helps explain why minimizing local reconstruction error need not yield the best eviction policy.
Response to W3 / Q3
We agree that single-token optimality does not establish optimality of bulk selection. Our batch analysis instead shows that fixed-centroid ranking minimizes an upper bound on disturbance under uniform attention.
For cached values, let be a removed set of size and the original full-cache centroid. Under uniform attention, the original and renormalized survivor outputs are
Their exact squared difference is therefore
Writing and makes the interactions explicit:
Thus, many individually small deviations can reinforce one another, while larger deviations can cancel. Individual ranking does not optimize these cross-token terms. Applying the triangle inequality and Cauchy–Schwarz gives
For fixed , removing the smallest full-centroid scores minimizes this upper bound, rather than the exact joint error. The implementation uses the eligible-token centroid , excluding protected sink/recent tokens. The corresponding bound is
The centroid-offset term is constant for a fixed cache and removal count, so selecting the smallest implemented scores minimizes this bound too.
Joint subset selection
To assess the selection gap without assuming clustered geometry, we compared three policies:
- Ordinary VD: ranks tokens once by distance from a fixed centroid and removes the lowest-scoring tokens.
- Sequential VD: removes the token closest to the current centroid, then recomputes the centroid before the next removal.
- Exact joint enumeration: evaluates every subset of the required removal size and selects the one minimizing the joint reconstruction error under uniform attention.
We compared these policies on 16-token miniature caches sampled from saved text-only WikiText values. Each policy removes four tokens without protection. The table reports mean reconstruction error normalized by the original centered value variance:
| Model | Miniature caches | VD | Sequential VD | Exact joint |
|---|---|---|---|---|
| Gemma2-9B | 36 | 0.021004 | 0.018291 | 0.013441 |
| Gemma3-4B | 36 | 0.017484 | 0.016500 | 0.013047 |
| GPT-OSS-20B | 72 | 0.017344 | 0.015993 | 0.009548 |
| Qwen3-4B-Instruct | 72 | 0.017933 | 0.017100 | 0.010912 |
Sequential updates lower mean reconstruction error in all four sampled model groups, but leave a gap to exact joint selection.
Exact enumeration requires evaluating subsets when removing of tokens, making it computationally intractable at benchmark cache sizes. We therefore evaluated sequential VD on all 13 RULER-8K tasks with 50 examples per task, using Qwen3-4B-Instruct and a 2K cache.
| Model | Plain VD | Sequential VD | Sequential−VD (pp) |
|---|---|---|---|
| Qwen3-4B-Instruct | 91.20 | 91.51 | +0.31 |
The observed overall difference is small: sequential updates improve two tasks, regress on two and tie on nine. RULER measures task accuracy with a 2K cache, while the miniature-cache experiment measures reconstruction error under uniform attention.
Thank you for your careful review and thoughtful questions. Your comments helped us clarify the limits of the theoretical and architectural interpretation and motivated the additional reconstruction-error, joint-selection, and sequential ValueDiff evaluations.
Response to W1 / Q1
We agree that the architectural explanation is observational. Our architectural groups organize the empirical comparison; they do not isolate the effects of sink suppression from training, gating, head sharing or other model differences.
We further examined checkpoints within the Qwen3.5 and Gemma2 families. We measured dense-prefill attention on the same 150 RULER-8K prompts per model, averaging five corresponding input positions, query heads and softmax-attention layers. Initial-token mass measures sink strength at token zero; normalized entropy measures attention flatness, with one corresponding to uniform attention. Performance gaps average all 13 RULER tasks at a 2K cache budget; positive values favor ValueDiff.
These within-family comparisons provide additional empirical context but do not isolate sink suppression. We will revise the manuscript to present the architectural explanation as an interpretation of the observed results and clarify that a causal test requires controlled interventions.
Response to W2
We agree that weaker attention sinks do not establish uniform attention or exchangeability of future token usage. Our study starts from empirical eviction results; the analysis helps interpret the algorithm's behavior, as reflected in the Section 3.3 title, “A Max-Entropy Interpretation of ValueDiff.”
We measured dense-prefill attention on 150 RULER-8K prompts per model. Initial-token mass measures the strength of the attention sink at token zero; normalized entropy measures how evenly attention is distributed over available tokens, with one corresponding to uniform attention.
The evaluated Qwen checkpoints have weaker initial-token sinks and flatter overall attention than Llama3.1, while attention remains nonuniform. These observations provide empirical context for the interpretation without establishing uniformity or exchangeability.
We will clarify in the manuscript that uniform attention and the symmetric Dirichlet prior are analytical reference settings, rather than consequences of sink suppression.