acceptodds
Withdrawn from ICLR 2027

KVRecon: KV Cache Merging via Reconstruction Error Minimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) increasingly rely on long contexts, making the memory footprint of key-value (KV) caches a major bottleneck for efficient inference and motivating the need for effective KV cache compression. Prior KV cache merging methods rely on predefined criteria for selecting entries to merge, or heuristic merging operations to combine them, which can introduce substantial discrepancies in model predictions after compression. In this paper, we introduce KVRecon, a KV cache merging method that minimizes the reconstruction error in the output of the attention module. It groups KV entries into multiple sets, and derives merged key and value representations for each set to preserve the output after merging. We further identify KV entries whose attention logits or value representations differ substantially from those of their corresponding merged entries to prevent inaccurate merging. Extensive experiments across multiple LLMs and downstream tasks demonstrate that KVRecon consistently outperforms existing KV cache compression methods.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, KV cache compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1dQ9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KVRecon, a merging-based KV cache compression method that addresses the memory bottleneck of the KV cache in long-context inference. The key idea is to formulate KV cache merging as a reconstruction problem of the attention output. Specifically, KVRecon groups KV entries into sets and merges each set into a single entry, where both the set assignment and the merged representations account for reconstruction errors between attention quantities computed with the original and merged caches at different stages of the attention computation: QK⊤QK^\top, exp⁡(QK⊤)\exp(QK^\top), and softmax(QK⊤)V\mathrm{softmax}(QK^\top)V. The merged key and its additive bias are derived using a first-order Taylor approximation, while the merged value is obtained by solving a least-squares problem. To prevent inaccurate merging, KVRecon further excludes outlier KV entries with large reconstruction errors from merging. The method is evaluated on LongBench and a shortened version of SCBench using Llama3.1-8B, Qwen2.5-7B, and Mistral-7B, and KVRecon achieves the highest average performance among the compared compression methods.

Critical strengths

1. Consistent gains in the multi-query setting. KVRecon substantially improves performance in multi-query scenarios, where existing eviction- and merging-based KV cache compression methods struggle. Although a gap to the full-cache performance remains, the improvements are meaningful in practical settings where a compressed cache is reused across multiple queries.

2. Simple and interpretable formulation. The additive bias, the merged key/value representations, and the set assignment are derived consistently from a single perspective of attention output reconstruction.

Critical weaknesses

1. Insufficient analysis of compression overhead and missing reproducibility details. I am concerned that the iterative clustering in Algorithm 1 may be computationally expensive, yet the paper does not analyze the time required for compression, leaving its practical applicability unsupported. In addition, although the Reproducibility Statement claims that the details required for reproduction are provided in the main text and appendix, the paper does not specify the number of merging rounds RR, nor how many queries (NN) are sampled from the repeat-prompt sequence and how they are selected. The final values of τ\tau and θ\theta for each model are also not reported, and no code is provided to fill these gaps.

2. Missing comparisons with the most closely related work. KeepKV directly addresses the output perturbation caused by KV cache merging, ReST-KV uses output reconstruction to decide which entries to evict, and CentroidKV, like KVRecon, merges KV entries through set-level clustering. All three methods are discussed in Section 2 but are not included in the experimental comparisons. In particular, KeepKV shares the core objective of preserving the attention output during merging, so without this comparison it is difficult to assess the contribution of reconstruction-based merging.

Questions for authors

Q1. What is the computational complexity of compressing SS KV entries into TT sets using NN queries? What are the actual peak memory usage and compression latency (including the repeat-prompt pass)? In particular, how many decoding steps are required for the speedup from the compressed cache to offset KVRecon’s compression overhead (i.e., the break-even point)?

Q2. KeepKV, ReST-KV, and CentroidKV are discussed in Section 2 but not included as baselines. Why were they excluded? I understand that KeepKV is designed to preserve the output for recent queries, so it may not directly fit the query-agnostic setting. Could you still compare KVRecon with KeepKV under the same cache budget, e.g., by computing its merging weights from the same repeat-prompt queries, or by adding its merging operation to Table 3?

Q3. What fraction of the final cache consists of merged entries, and what is the average set size? In addition, what values of NN, RR, τ\tau, and θ\theta were used in the experiments, and how were the positions of the NN queries selected?

Reviewer_rvQ3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes KVRecon, a query-agnostic KV cache compression method combining merging and eviction. It assigns cached entries to groups using reference-query logit differences, represents each group with a mean key, a log-cardinality bias, and a least-squares value estimate, and excludes entries that violate logit/value discrepancy thresholds. A final attention-based eviction step enforces the cache budget. Reference queries are obtained through KVzip-style context repetition, without observing downstream questions. Experiments on LongBench and shortened/full-length SCBench report improvements over the evaluated eviction and merging baselines on three 7–8B models.

Critical strengths
  1. Substantial quality improvements in a useful reuse setting. Compressing a context before seeing downstream questions and reusing it across questions is practically relevant. The reported retrieval gains are substantial: on shortened SCBench at a 10% budget, Llama-3.1-8B achieves 69.7 on Retr.KV versus 28.7 for KVzip (Table 2). The full-length SCBench results in Appendix C provide additional evidence beyond short contexts.

  2. A useful formulation and promising component-level evidence. Separately preserving attention mass and the weighted value contribution provides a clear framework for reasoning about merging. The logit-based grouping and merge eligibility conditions are motivated by this framework, and the ablations help explain their roles. For example, Table 5 reports 69.7 versus 48.7 on Llama-3.1-8B Retr.KV for logit-based versus key-cosine grouping under the proposed pipeline. These results suggest that the grouping/eligibility design merits further investigation.

Critical weaknesses
  1. The incremental contribution and the source of the gains are not sufficiently established. A closely related prior work is missing: Zweiger et al., Fast KV Compaction via Attention Matching (arXiv:2602.16284, February 2026) [1] also preserves attention mass and attention outputs using reference queries, per-entry biases, and least-squares value fitting, including a repeat-prefill query variant. KVRecon's grouping and eligibility conditions are different, but the paper needs to establish their advantage relative to this framework. KeepKV, explicitly discussed as an output-preserving merging method in Section 2, is also absent from the experiments.

The existing ablations do not fully resolve this attribution issue. Table 4 shows that removing both eligibility conditions reduces Retr.KV to 0.0 for all three models. This supports the importance of the conditions, rather than invalidating the complete method. However, because rejected merges are retained as original entries and the final budget is enforced by eviction, the gain over a simpler merging/retention/eviction hybrid remains unclear. Table 3(b) further shows only small gains from Eq. 12 over AsymKV's value rule on this task. A matched hybrid control and an analysis of the final cache composition would materially strengthen the contribution claim.

  1. The practical quality–cost tradeoff is unmeasured, with important reproducibility details missing. The paper reports cache-entry budgets and task scores, but no compression time, peak GPU memory, or decoding performance. KVRecon requires reference-query collection, repeated assignment/filtering, and offline threshold calibration. These costs may be worthwhile when a context is reused many times, but the paper does not establish that tradeoff. Repeat prompting is also used by KVzip, so its cost should be accounted for consistently across methods rather than treated as a uniquely KVRecon overhead.

Algorithm 1 does not specify the experimental values of N and R. Appendix B does not give the calibration-set size, resulting thresholds, or whether calibration is repeated across budgets. The attention implementation supporting the per-entry bias is also unspecified. Without these details and measurements, it is difficult to reproduce the results or judge the benefit under a realistic memory and latency budget.

Questions for authors
  1. Related methods and baseline configuration. How does KVRecon compare with Attention Matching [1] under the same downstream protocol, reference-query source, and total cache budget? Please also compare with KeepKV where applicable, explaining any adaptation to the query-agnostic setting. Was KVzip run with its intended head-budget allocation? What explains the near-zero D2O results on Qwen2.5-7B in Table 1?

  2. Attribution within the hybrid pipeline. What fractions of the final cache are merged groups and retained original entries, and what attention score is used for the final eviction? Under the same grouping, eligibility rules, and eviction scorer, how does simple averaging of values with the same key/bias rule compare with Eq. 12? Please also report downstream accuracy at several cache budgets; Figure 4 varies reconstruction error rather than task accuracy.

  3. Scope of the reconstruction claim. Proposition 2 gives the exact least-squares solution to the surrogate in Eq. 26, after Eq. 25 replaces the merged coefficient by the original group's attention mass. It is not generally the exact minimizer of Eq. 24. Can the claim be qualified accordingly and the error from this approximation quantified? Equation 14 bounds an empirical mean absolute logit discrepancy, not the discrepancy of every reference or future query. Is Figure 4 evaluated on fitting queries or held-out downstream queries, and which layer/head is shown? Does the reconstruction-error ordering persist across layers/heads on held-out questions?

  4. Efficiency and reproducibility. What are N, R, the query sampling/GQA aggregation procedure, and the calibration-set size and resulting thresholds? For representative short and full-length SCBench contexts, please report query-collection time, compression time, peak memory, and decoding latency/throughput against full cache and KVzip, including the attention backend used for the logit biases. How does total time change with the number of questions sharing a context, and must thresholds be recalibrated for different budgets?

  5. Numerical consistency. Could the authors explain or correct the Table 1 averages? Taking the arithmetic mean of the 16 displayed task scores gives approximately 45.004 for Llama KVRecon, 32.965 for Qwen AsymKV, and 40.173 for Qwen KVRecon, versus the reported 44.88, 34.24, and 40.40. Please state any different aggregation protocol and update the associated improvement margins. In Appendix C, Qwen KVRecon appears to have five outright task wins and one tie among compressed methods, rather than seven wins. These discrepancies do not overturn its average-score lead, but the quantitative claims should be made consistent.

[1] Adam Zweiger, Xinghong Fu, Han Guo, and Yoon Kim. Fast KV Compaction via Attention Matching. 2026. https://arxiv.org/abs/2602.16284

Reviewer_bDTgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The work merges KV entries to minimize the reconstruction error of the output of attention on reference queries. It builds those queries the way KVzip does ie by repeating the prompt. KV entries are clustered by query projected key distance. each cluster gets a mean key plus a logit bias (a first-order Taylor argument as shown in Prop. 1) and then a closed form LS value (see Prop. 2). Two thresholds block dissimilar entries from merging and whatever fails either falls back to attention score eviction. Evaluation is done on LongBench and SCBench at a 10% budget on Llama-3.1-8B/ Qwen2.5-7B / Mistral-7B and the baselines they compare against are StreamingLLM, KVzip, D2O,. and AsymKV.

Positives

  • Query-agnostic protocol: compress once then reuse. That is i suppose the right setting as eval questions don't leak into the compression method.
  • Ablations are reasonably clean and informative.

Negatives

!!!Most serious:!!! the core idea is published and not cited "Fast KV Compaction via Attention Matching" (Zweiger et al ICML 2026) already has:

  1. the same objective: match attention mass with a bias, fit values by least squares to the original outputs
  2. the same KVzip repeat-prompt reference queries
  3. the same query-agnostic protocol..

The presented method instead uses subset selection instead for cluster means and uses a fixed bias term instead of a fitted one and fits values per set instead of jointly. On paper all three are actually worse choices.

Baselines are ver thin and sometimes also (very problematic!) broken.

  • D2O on Qwen2.5-7B averages 1.63 and is basically 0 on 14 of 16 tasks while the same method scores around ~32 on other two models. That seems very sus...
  • Missing (among others): SnapKV, PyramidKV, AdaKV, Expected Attention, KeepKV, CentroidKV, Attention Matching (long list) etc.
  • Results are computed at a single budget.

Also No cost reported anywhere compression time/ memory/ how queries are pooled under GQA etc

Critical strengths

None

Critical weaknesses

Attention Matching (ICML 2026) basically covers the presented method: same objective, protocol, and arguably stronger solver. neither cited nor compared. That alone decides it for me

Questions for authors

i guess there’s only one real question: how does it do apples to apples against Attention Matching?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.