- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes KV-Surgeon, which builds on the Attention Matching (AM) framework to improve KV cache compaction. While AM-OMP, an AM variant based on orthogonal matching pursuit, selects keys primarily to preserve attention mass and then refits the values to reconstruct attention outputs, KV-Surgeon considers both objectives when deciding which entries to remove. It scores each deletion by the increase in a quadratic surrogate loss after optimally adjusting the remaining entries. It then removes the least costly entry and updates the survivors, using closed-form scores and efficient updates. Finally, it refits the retained weights and values to produce the compressed cache.
Critical strengths
- Well-motivated contribution. Independent eviction scores overlook redundancy and interactions among retained entries. Incorporating compensation directly into selection addresses this limitation naturally. The contribution is clearly distinguishable from post-selection refitting.
Critical weaknesses
- The relationship between the selection surrogate and the deployed objective remains incompletely characterized. Equation (4) follows from freezing the normalization, dropping the cross term, and replacing query-dependent output norms with a common scale. Appendix A.1 establishes a second-order Taylor interpretation for the intermediate objective after freezing the normalization, rather than for the final decoupled surrogate.
Questions for authors
- How robust is the method when the calibration query distribution differs substantially from the query distribution encountered at inference time, for example under domain shift?
2 comments
Thanks for the authors' response; I will keep my positive score.
We thank the reviewer for their thoughtful assessment and constructive feedback.
W1. Relationship between the surrogate and actual output error
For a retained set of cache entries S, let Pij denote the original attention weight of query i on cache entry j, and let oi denote the original attention-head output. The compressed cache uses retained values vj and positive attention multipliers wj, producing oi=∑j∈SPijwj∑j∈SPijwjvj. The error of interest is L=∑i∥oi−oi∥22: the sum of squared Euclidean distances between full-cache and compressed-cache outputs on the same queries.
Write uj=wjvj and μi=∑j∈SPijwj. Freezing the normalization factor μi−2 at one gives
Q=∑i∑j∈SPijuj−μioi22.
Dropping the cross term between ∑j∈SPijuj−oi and (1−μi)oi gives
Qsep=∑i∑j∈SPijuj−oi22+∑i(1−μi)2∥oi∥22.
Replacing ∥oi∥22 by its mean γ2 gives L, with a shared regression matrix for both terms. Selection also relaxes positivity of w and adds ridge regularization.
The second-order Taylor approximation in Appendix A.1 applies to the intermediate quadratic Q, not the final selection problem. Its purpose is to provide intuition for freezing the softmax normalization. Thank you for highlighting this distinction. We will make it more explicit in Appendix A.1.
We also measured how much each approximation changes the numerical objective value for the same compressed cache. We evaluate L, Q, Qsep, and L on compacted KV-caches obtained from KV-Surgeon.
We use Qwen3-4B, five QuALITY articles, and six attention heads spanning early, middle, and late layers per article, at 20×, 50×, and 100× compression. For each compression ratio, we compute the median paired objective-value ratio over the 30 head–article pairs. The table reports the range of these medians across compression ratios. A ratio of one means unchanged numerical value.
Dropping the cross term roughly doubles the objective value. The other approximations have smaller effects on the reported medians. Overall, median paired L/L is 1.84–2.26.
Q1. Robustness to calibration-query distribution shift
We evaluated robustness to query distribution shifts by changing the queries used to calibrate compaction while keeping the downstream evaluation questions and retention budgets fixed. That is, we tested whether calibration focused on broad themes or high-level summaries transfers to the detailed multiple-choice questions in QuALITY and LongHealth.
We compare three calibration conditions. Repeat only collects queries while we prefill the context concatenated with an instruction to repeat it verbatim and the repeated context. Self-study + repeat, our reference condition, additionally collects queries from model-generated responses to generic summarization and restructuring prompts. Mismatched intent calibration uses eight prompts emphasizing broad thematic discussion on QuALITY or high-level patient summaries on LongHealth, rather than detailed factual information.
We evaluate at 50× compaction with three random seeds, using the same 50 QuALITY articles and four LongHealth contexts containing 20 patient records as in the main evaluation. The table reports reference accuracy and seed-paired accuracy changes relative to that reference, in percentage points.
KV-Surgeon retains the highest mean accuracy in all six shifted settings. Removing self-study gives the most favorable accuracy change for KV-Surgeon in each model–dataset setting. Under mismatched intent, its accuracy degradation is not uniformly smaller than that of the baselines.
Substantial mismatch remains a limitation: summary-oriented calibration reduces accuracy by approximately 9–11 points for all methods on LongHealth. These results show sensitivity to calibration coverage despite KV-Surgeon’s retained accuracy advantage.
We will include both results in the appendix.