acceptodds
Under review as a conference paper at ICLR 2027

The KV-Surgeon: Compensation-Aware Offline KV-Cache Compaction

Abstract

Offline compaction of the key-value cache (KV-cache) reduces the cost of repeatedly querying a fixed context by constructing a smaller cache once and reusing it subsequently. The central challenge of compaction is deciding which cache entries to delete and which to retain. Existing methods score each candidate for deletion on its own, with the rest of the cache being held fixed. However, softmax renormalization of the attention operation couples these decisions in a nonlinear manner: removing one entry changes the attention weights of every remaining one. In this work, we introduce KV-Surgeon, which uses calibration data to take interactions into account in a tractable way. We derive a quadratic surrogate that admits closed-form expressions for jointly making the optimal greedy removal decision as well as compensation within the remaining entries. Across benchmark datasets, such as QuALITY, LongHealth, and Long Code Arena, and at every evaluated compaction ratio, KV-Surgeon substantially advances the cost-accuracy Pareto frontier of dense and mixture-of-experts models from 4B to 30B parameters. We observe gains of up to four accuracy points on question-answering at compaction, exceeding the state of the art, attention matching (Zweiger et al., 2026), at comparable cost. Lastly, we introduce an even faster variant, which also exceeds the state-of-the-art while requiring only two fifths of its cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, long-context inference, efficient LLM inference, token eviction, cache compaction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_33xFOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KV-Surgeon, which builds on the Attention Matching (AM) framework to improve KV cache compaction. While AM-OMP, an AM variant based on orthogonal matching pursuit, selects keys primarily to preserve attention mass and then refits the values to reconstruct attention outputs, KV-Surgeon considers both objectives when deciding which entries to remove. It scores each deletion by the increase in a quadratic surrogate loss after optimally adjusting the remaining entries. It then removes the least costly entry and updates the survivors, using closed-form scores and efficient updates. Finally, it refits the retained weights and values to produce the compressed cache.

Critical strengths
  • Well-motivated contribution. Independent eviction scores overlook redundancy and interactions among retained entries. Incorporating compensation directly into selection addresses this limitation naturally. The contribution is clearly distinguishable from post-selection refitting.
Critical weaknesses
  • The relationship between the selection surrogate and the deployed objective remains incompletely characterized. Equation (4) follows from freezing the normalization, dropping the cross term, and replacing query-dependent output norms with a common scale. Appendix A.1 establishes a second-order Taylor interpretation for the intermediate objective after freezing the normalization, rather than for the final decoupled surrogate.
Questions for authors
  • How robust is the method when the calibration query distribution differs substantially from the query distribution encountered at inference time, for example under domain shift?
2 comments

We thank the reviewer for their thoughtful assessment and constructive feedback.

W1. Relationship between the surrogate and actual output error

For a retained set of cache entries SS, let PijP_{ij} denote the original attention weight of query ii on cache entry jj, and let oio_i denote the original attention-head output. The compressed cache uses retained values v^j\widehat v_j and positive attention multipliers wjw_j, producing o^i=∑j∈SPijwjv^j∑j∈SPijwj.\widehat o_i=\frac{\sum_{j\in S}P_{ij}w_j\widehat v_j}{\sum_{j\in S}P_{ij}w_j}. The error of interest is L=∑i∥o^i−oi∥22L=\sum_i\|\widehat o_i-o_i\|_2^2: the sum of squared Euclidean distances between full-cache and compressed-cache outputs on the same queries.

Write uj=wjv^ju_j=w_j\widehat v_j and μi=∑j∈SPijwj\mu_i=\sum_{j\in S}P_{ij}w_j. Freezing the normalization factor μi−2\mu_i^{-2} at one gives

Q=∑i∥∑j∈SPijuj−μioi∥22.Q=\sum_i\left\|\sum_{j\in S}P_{ij}u_j-\mu_i o_i\right\|_2^2.

Dropping the cross term between ∑j∈SPijuj−oi\sum_{j\in S}P_{ij}u_j-o_i and (1−μi)oi(1-\mu_i)o_i gives

Qsep=∑i∥∑j∈SPijuj−oi∥22+∑i(1−μi)2∥oi∥22.Q_{\mathrm{sep}}=\sum_i\left\|\sum_{j\in S}P_{ij}u_j-o_i\right\|_2^2+\sum_i(1-\mu_i)^2\|o_i\|_2^2.

Replacing ∥oi∥22\|o_i\|_2^2 by its mean γ2\gamma^2 gives L^\widehat L, with a shared regression matrix for both terms. Selection also relaxes positivity of ww and adds ridge regularization.

The second-order Taylor approximation in Appendix A.1 applies to the intermediate quadratic QQ, not the final selection problem. Its purpose is to provide intuition for freezing the softmax normalization. Thank you for highlighting this distinction. We will make it more explicit in Appendix A.1.

We also measured how much each approximation changes the numerical objective value for the same compressed cache. We evaluate LL, QQ, QsepQ_{\mathrm{sep}}, and L^\widehat L on compacted KV-caches obtained from KV-Surgeon.

We use Qwen3-4B, five QuALITY articles, and six attention heads spanning early, middle, and late layers per article, at 20×20\times, 50×50\times, and 100×100\times compression. For each compression ratio, we compute the median paired objective-value ratio over the 30 head–article pairs. The table reports the range of these medians across compression ratios. A ratio of one means unchanged numerical value.

ApproximationCalibration queriesHeld-out queries
Freeze normalization: Q/LQ/L0.92–0.980.88–1.10
Drop cross term: Qsep/QQ_{\mathrm{sep}}/Q2.08–2.542.07–2.23
Use common scale: L^/Qsep\widehat L/Q_{\mathrm{sep}}0.88–0.910.99–1.01

Dropping the cross term roughly doubles the objective value. The other approximations have smaller effects on the reported medians. Overall, median paired L^/L\widehat L/L is 1.84–2.26.

Q1. Robustness to calibration-query distribution shift

We evaluated robustness to query distribution shifts by changing the queries used to calibrate compaction while keeping the downstream evaluation questions and retention budgets fixed. That is, we tested whether calibration focused on broad themes or high-level summaries transfers to the detailed multiple-choice questions in QuALITY and LongHealth.

We compare three calibration conditions. Repeat only collects queries while we prefill the context concatenated with an instruction to repeat it verbatim and the repeated context. Self-study + repeat, our reference condition, additionally collects queries from model-generated responses to generic summarization and restructuring prompts. Mismatched intent calibration uses eight prompts emphasizing broad thematic discussion on QuALITY or high-level patient summaries on LongHealth, rather than detailed factual information.

We evaluate at 50×50\times compaction with three random seeds, using the same 50 QuALITY articles and four LongHealth contexts containing 20 patient records as in the main evaluation. The table reports reference accuracy and seed-paired accuracy changes relative to that reference, in percentage points.

Model / datasetMethodSelf-study + repeatRepeat-only changeMismatched-intent change
Qwen3-4B / QuALITYKV-Surgeon65.7−0.8−2.5
LOO63.6−3.7−2.1
AM-HAK61.4−4.0−1.5
Qwen3-4B-Instruct-2507 / LongHealthKV-Surgeon67.0−2.4−10.2
LOO61.8−3.8−10.8
AM-HAK61.3−7.2−9.2
Llama-3.1-8B / QuALITYKV-Surgeon61.0+0.1−3.9
LOO61.7−2.6−6.6
AM-HAK59.3−1.2−5.7

KV-Surgeon retains the highest mean accuracy in all six shifted settings. Removing self-study gives the most favorable accuracy change for KV-Surgeon in each model–dataset setting. Under mismatched intent, its accuracy degradation is not uniformly smaller than that of the baselines.

Substantial mismatch remains a limitation: summary-oriented calibration reduces accuracy by approximately 9–11 points for all methods on LongHealth. These results show sensitivity to calibration coverage despite KV-Surgeon’s retained accuracy advantage.


We will include both results in the appendix.

Thanks for the authors' response; I will keep my positive score.

Reviewer_osdmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes KV-Surgeon, a KV-cache compression method for fixed long contexts that are reused across multiple queries. Rather than selecting tokens based on independent importance scores, KV-Surgeon accounts for how well the remaining tokens can compensate for removed positions. The authors approximate attention-output reconstruction with a ridge regression objective and derive an efficient deletion-and-compensation procedure inspired by Optimal Brain Surgeon, followed by value and attention-bias reconstruction. Experiments on question answering and code tasks demonstrate improved quality-efficiency trade-offs over several existing compression methods, with particularly promising results at high compression ratios.

Critical strengths
  1. The paper introduces a well-motivated approach to KV-cache compression by explicitly considering the compensability of removed tokens. Unlike conventional importance-based selection, the proposed method evaluates deletion costs after accounting for how the remaining positions can reconstruct the lost information. This provides a meaningful conceptual improvement over independent token ranking.

  2. The optimization formulation is elegant and computationally efficient. By relaxing attention-output reconstruction into a ridge regression problem, the authors derive closed-form deletion costs and compensation updates that can be computed using shared matrix operations. The resulting procedure establishes a clear connection between the optimization objective and the practical selection algorithm.

  3. The experimental comparisons are well-designed, which use shared retention budgets and reconstruction procedures to isolate the effect of support selection and the compensation ablation also provides direct evidence for the proposed mechanism. The additional evaluation of held-out reconstruction loss also helps establish that the method is not only optimizing its calibration objective.

Critical weaknesses
  1. The evaluation primarily measures offline compression costs without demonstrating how these costs translate into end-to-end serving efficiency. Since the intended application involves repeatedly querying a compressed context, measurements of decoding latency, peak memory, and the number of reuses needed to amortize compression would strengthen the practical claims.

  2. The relationship between the surrogate optimization objective and actual attention-output reconstruction remains relatively insufficiently characterized. The proposed deletion costs are derived from a ridge regression approximation, while the final reconstructed values and attention biases may differ substantially from those assumed during selection. The reported large gate adjustments and frequent saturation at high compression ratios raise questions about how accurately the surrogate reflects the final reconstruction error. Although the paper provides held-out loss measurements, a more direct analysis of the correspondence between surrogate deletion costs, final reconstruction loss, and downstream accuracy would better justify the approximation.

Questions for authors
  1. What precisely is optimized and enumerated in the exact-refit oracle comparison? How large are the candidate sets, and how is the claimed optimality established?

  2. Which types of queries or contextual information are most vulnerable to compression-induced errors? Can the authors characterize the remaining accuracy gap relative to Full KV and its relationship to reconstruction quality?

3 comments

We thank the reviewer for their thoughtful comments. We address serving efficiency and deletion costs here. The second comment covers the oracle and downstream failures.

W1. End-to-end serving efficiency

In this setup, compaction does not accelerate a single decoding stream. Its benefit is lower memory use, allowing more concurrent requests and higher throughput.

We use Qwen3-30B-A3B on four LongHealth contexts (58.6k tokens on average, five concatenated patient records), one H200, bf16, a 256-token prompt and 128 generated tokens. Recall that, for each attention head, the compacted cache stores the retained keys, refitted values, and one attention multiplier per retained cache entry. These multipliers are implemented as additive logit biases inside a PyTorch FlexAttention kernel.

One-time cost at 50×. Mean seconds per context over four contexts and three seeds:

MethodPrefillCalibration generationCompaction
KV-Surgeon9243161
KV-Surgeon-fast96425

Serving benefit. Medians over the four contexts:

MeasurementFull KVKV-Surgeon, 50×
Batch-1 time to first token1.32 s0.90 s
Batch-1 decode step67.2 ms74.5 ms
Batch-1 request time9.92 s10.39 s
Stored KV per sequence5.86 GB0.12 GB
Batch used for throughput comparison264
Peak memory at that batch96 GB97 GB
Aggregate output throughput21 tokens/s165 tokens/s
Amortized serving time per request7.1 s0.81 s

At batch size 1, faster prompt processing is offset by slower decoding with the compact-cache kernel, so there is no latency break-even. These timings use Hugging Face Transformers’ eager MoE implementation, whose per-expert loop dominates decoding. Full-cache attention accounts for only 3.3 ms of the 67.2 ms step. The compacted cache’s additional ~7 ms comes from the FlexAttention kernel and host-dispatch overhead. Absolute throughput is therefore specific to this implementation and should not be extrapolated to a fused serving engine. At equal peak memory, compaction enables batch size 64 versus 2 and increases throughput 7.9×.

Amortization over reuses. We view calibration and compaction one-time costs per contexts that need to be amortized through reusing the compacted context for multiple requests.For the equal-memory comparison, the break-even request count is ⌈C/(sfull−scompact)⌉.\left\lceil C/(s_{\rm full}-s_{\rm compact})\right\rceil. Here, CC is calibration plus compaction time, and ss is batch execution time divided by batch size. Using the per-article saving of approximately 6.2 s/request gives about 400 requests per context for KV-Surgeon and about 14 requests for KV-Surgeon-fast.

This benefit requires enough concurrent requests to fill the compact-cache batch. We will add these measurements and amortization assumptions to the main text.

W2. Does the surrogate choose useful deletions after the final refit?

We aim to minimize the squared difference between the original and compacted attention outputs. During the greedy elimination of the cache entries, a quadratic surrogate makes it cheap to score deletions while adjusting the remaining entries to compensate. The final refit then fits the values and bounded attention multipliers. We test whether the surrogate’s choices remain good after refitting.

To test this, we pause elimination and compare single-entry deletions from the same current cache. We first rank candidates by their surrogate deletion costs. We then remove the top 64 entries one at a time, independently refit the remaining multipliers and values on the same calibration queries, and measure the actual output error. Ranking candidates by this post-refit error gives a second ordering, computed separately on calibration and held-out question queries. The table compares these rankings and reports the surrogate-chosen deletion’s excess loss over the best tested alternative, relative to the current cache’s loss. Results average over 520 states across five QuALITY articles, six heads per model from early, middle and late layers of Qwen3-4B and Llama-3.1-8B, and 20×/50×/100× compaction.

Mean across statesCalibrationHeld-out
Correlation between surrogate and post-refit rankings0.400.23
Excess loss over best tested deletion (% of current loss)0.9%2.4%

The rankings are not correlated much, but choosing the surrogate’s preferred deletion incurs only a small additional error. When more of the multipliers reach their allowed bounds, individual-deletion rankings get worse. However, in our experiments, selecting through surrogate score always outperforms random selection, no matter how many multipliers saturate. Thus, the surrogate remains useful for selection despite imperfect correspondence with the final refit.

Q1. What is enumerated, and what is optimal?

In Appendix E, we compare KV-Surgeon with an expensive reference that selects cache entries using the actual reconstruction error, to assess how much selection quality is lost by using the cheaper surrogate. The reference tests which next deletion gives the lowest actual attention-output error after refitting the remaining entries, whereas KV-Surgeon scores deletions using its quadratic surrogate.

Both start from the same candidate pool of 4t4t entries, where tt is the budget. At each step, the reference tries every remaining single-entry deletion. For each resulting cache, it fits the bounded attention multipliers and then the values on calibration queries, evaluates the actual output loss of Eq. (2) on those queries, and permanently removes the entry with the lowest loss. This continues until tt entries remain.

The comparison covers 546 Qwen3-4B heads with t≤57t\le57, so there are at most 228 candidate deletions initially, decreasing at each step. Enumeration establishes the best next deletion under this refitting procedure. It does not enumerate all final supports or establish a globally optimal support. Likewise, “exact” refers to evaluating the actual output loss, not global joint optimization of multipliers and values.

For the reported comparison, the reference, KV-Surgeon, LOO and random supports all receive the same final refit on calibration queries and are evaluated on the same held-out question queries. We will rename the oracle the exact-loss greedy reference and specify these distinctions in Appendix E.

Q2. Which questions fail, and how does this relate to reconstruction?

Remaining gap to Full KV. We compare answers question by question on Qwen3-4B / QuALITY (894 questions, five seeds). A “lost correct answer” is a question answered correctly by Full KV but incorrectly after compaction.

MethodCompressionAccuracyLost correct answers
Full KV—72.1%—
KV-Surgeon20×70.2%6.8%
KV-Surgeon50×66.8%10.8%
KV-Surgeon100×63.4%14.7%
LOO50×65.6%12.2%
AM-HAK50×64.1%13.9%

At 50×, KV-Surgeon’s gap to Full KV is 5.3 percentage points, compared with 6.5 for LOO and 8.0 for AM-HAK. The reverse outcome (Full KV wrong, compacted model correct) occurs at similar rates for these methods (5.5–5.9%). KV-Surgeon’s advantage therefore primarily comes from preserving more of Full KV’s correct answers.

What do the failed questions ask for? We manually inspect 50 randomly sampled failures from the 86 in KV-Surgeon’s seed-0 run at 50×:

Information requestedFailures
Specific detail from one passage27
Number or count6
Integration across passages9
Global inference or tone8

Specific-detail and numerical questions account for 33 of 50 inspected failures. This identifies a common failure type, but does not establish disproportionate vulnerability without measuring these categories’ prevalence in the full question set. In 39 cases the model selects a wrong option. In 11 it exhausts the generation budget while re-checking the passage, including 9 specific-detail or numerical questions.

We also test whether larger reconstruction errors incur more lost correct answers. At 100× compaction, we sort questions by attention-output reconstruction error across attention heads. Among the quarter with the largest errors, 18.8% are answered correctly by Full KV but incorrectly by KV-Surgeon, compared with 9.4% in the quarter with the smallest errors. Larger reconstruction errors therefore coincide with more lost correct answers at 100×. At 20× compaction, we observe no clear relationship.

We will add these analyses to the appendix.

Thank you for the detailed response. The addition experiment results address my main questions. I am keeping my score.

Reviewer_x785OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KV-Surgeon, a compensation-aware method for offline KV-cache compaction. The key idea is to jointly consider token removal and the ability of retained cache entries to compensate for the resulting attention-output error. The authors derive a quadratic surrogate and an OBS-inspired greedy elimination algorithm with closed-form compensation updates, followed by a final refitting stage.

Experiments on QuALITY, LongHealth, and Long Code Arena, across several LLM architectures and compaction ratios, demonstrate improved accuracy-efficiency trade-offs over existing methods. Ablations further support the importance of compensation during token selection.

Critical strengths
  1. The paper identifies an important limitation of existing KV-cache compaction methods: token selection is typically performed without considering how the remaining entries can compensate for deletions. The proposed compensation-aware selection provides a principled approach to this problem, with efficient closed-form updates.

  2. The method consistently improves the accuracy-efficiency trade-off, particularly at aggressive compression ratios. The comparisons against LOO and the variant without compensation provide useful evidence that iterative compensation, rather than merely changing the selection score, contributes to the observed improvements.

Critical weaknesses
  1. The core optimization strategy closely follows the classical Optimal Brain Surgeon framework, applied to a quadratic surrogate for KV-cache compaction. Although the adaptation is meaningful, the underlying compensation mechanism and closed-form updates are established techniques. The paper could better distinguish its conceptual contributions from this existing optimization framework.

  2. The evaluation focuses on offline compaction with fixed contexts and repeated queries. It remains unclear how well the approach generalizes to more diverse, dynamically evolving workloads. Furthermore, the reported pipeline time excludes context prefill and subsequent inference, making it difficult to assess the actual end-to-end serving benefits and the number of repeated queries needed to amortize compaction costs.

Questions for authors

Could the authors clarify the main methodological novelty beyond adapting the classical OBS framework to KV-cache compaction? In particular, which aspects are specific to the attention mechanism rather than standard quadratic optimization?

3 comments

We thank the reviewer for the thoughtful assessment. We clarify the contribution beyond OBS and report new workload experiments here. The second comment addresses serving efficiency.

W1 / Q1. Novelty beyond OBS

The reviewer is correct, the quadratic deletion costs and compensation updates are established OBS machinery. Our contribution is the attention-specific formulation that makes output-aware compensation tractable during support selection.

For original attention weights PijP_{ij}, retained cache entries SS, positive multipliers wjw_j and refitted values v^j\widehat v_j, the compacted output is o^i=∑j∈SPijwjv^j∑j∈SPijwj.\widehat o_i=\frac{\sum_{j\in S}P_{ij}w_j\widehat v_j}{\sum_{j\in S}P_{ij}w_j}. Unlike reconstruction in a fixed linear model, deletion changes the normalisation through which every survivor contributes. With U=diag⁡(w)V^U=\operatorname{diag}(w)\widehat V, freezing normalisation, dropping the residual cross term and using a common output scale gives Eq. (4): L^=∥O−PSU∥F2+γ2∥1−PSw∥22,\widehat L=\|O-P_SU\|_F^2+\gamma^2\|\mathbf1-P_Sw\|_2^2, where OO stacks the original outputs and γ\gamma scales the mass penalty. This accounts for both output reconstruction and normalisation mass, while making all dv+1d_v+1 regression targets share one Gram matrix. OBS then provides efficient compensation updates within this surrogate.

The algorithmic contribution is using output-aware compensability to choose the support, rather than only refitting after selection. Compared with mass-based adaptive selection such as AM-OMP, selection also incorporates output reconstruction. Table 14(b) isolates compensation with the same candidate pool, search and final refit: omitting the compensating updates reduces accuracy by 2.9 points. Greedy exact-loss selection without compensation remains 1.1 points below KV-Surgeon. After selection, bounded positive multipliers and values are refitted to obtain the served cache.

We will revise the contributions paragraph and Section 3.3 to distinguish the attention-specific surrogate and selection method from the inherited OBS solutions.

W2. Diverse and evolving workloads

KV-Surgeon targets a long context that is reused across many requests. The KV cache is compacted once and then reused for multiple requests. The aim is to reduce the memory and serving cost of repeated access while preserving answer quality. The reviewer’s concern therefore raises two questions: whether the compacted cache remains useful when subsequent queries or interactions differ from calibration, and whether serving savings amortise the one-time compaction cost. We address both below.

Calibration shift. At 50×50\times compaction, we test how compaction quality responds to a calibration query distribution shift by changing calibration intent. Standard calibration uses self-study plus a repeat pass. Alternatives use only the repeat pass (KV-Surgeon-fast) or mismatched intent (new shifted calibration). The mismatched intent calibration uses thematic discussion for QuALITY and high-level patient summaries for LongHealth to generate calibration queries.

We evaluate 50 QuALITY articles (894 questions) and four five-patient LongHealth contexts (400 questions), with three seeds. Self-study + repeat is accuracy (%) and the other columns are paired changes in percentage points.

Model / datasetMethodSelf-study + repeatRepeat-only changeMismatched-intent change
Qwen3-4B / QuALITYKV-Surgeon65.7−0.8−2.5
LOO63.6−3.7−2.1
AM-HAK61.4−4.0−1.5
Qwen3-4B-Instruct-2507 / LongHealthKV-Surgeon67.0−2.4−10.2
LOO61.8−3.8−10.8
AM-HAK61.3−7.2−9.2
Llama-3.1-8B / QuALITYKV-Surgeon61.0+0.1−3.9
LOO61.7−2.6−6.6
AM-HAK59.3−1.2−5.7

KV-Surgeon has the highest mean accuracy in all six shifted settings and is least affected by removing self-study. Intent mismatch is more consequential: every method loses 9–11 points on LongHealth, showing the importance of calibration coverage.

Growing interaction. We compact an article once at 50×50\times, append an uncompacted discussion, and then let the model answer the context's benchmark questions. Qwen3-4B is evaluated on ten QuALITY articles (181 questions) with three seeds. Each method/seed run independently generates a discussion from the full article using generic prompts and turns containing substrings of the correct answer of at least eight characters are removed to prevent leakage.

Full KV is accuracy (%). The other columns are seed-paired gaps in percentage points.

Appended tokensFull KVAM-HAK gapKV-Surgeon gap
073.3−5.2−4.2
≈46072.9−6.6−4.4
≈355067.8−2.2+0.4

KV-Surgeon changes from 69.1% to 68.2% accuracy, while Full KV falls from 73.3% to 67.8%. Thus its gap does not grow under this interaction workload.

We will add both experiments to the appendix.

W2 (cont.). End-to-end serving efficiency and amortization

In this setup, compaction does not accelerate a single decoding stream. Its benefit is lower memory use, allowing more concurrent requests and higher throughput.

We use Qwen3-30B-A3B on four LongHealth contexts (58.6k tokens on average, five concatenated patient records), one H200, bf16, a 256-token prompt and 128 decode steps with identical forced tokens in both arms. Recall that each compacted attention head stores retained keys, refitted values and one attention multiplier per retained entry. We implement these multipliers as additive logit biases inside PyTorch FlexAttention and Full KV uses stock SDPA.

One-time cost at 50×50\times. Mean seconds per context over four contexts and three seeds:

MethodPrefillCalibration generationCompaction
KV-Surgeon9243161
KV-Surgeon-fast96425

Serving benefit. Median results across the four contexts, with three token seeds and five timed repeats per cell:

MeasurementFull KVKV-Surgeon, 50×50\times
Batch-1 time to first token1.32 s0.90 s
Batch-1 decode step67.2 ms74.5 ms
Batch-1 request time9.92 s10.39 s
Stored KV per sequence5.86 GB0.12 GB
Batch used for throughput comparison264
Peak memory at that batch96 GB97 GB
Aggregate output throughput21 tokens/s165 tokens/s
Amortised serving time per request7.1 s0.81 s

At batch size 1, faster prompt processing is offset by slower decoding with the compact-cache kernel, so there is no latency break-even. These timings use Hugging Face Transformers’ eager MoE implementation, whose per-expert loop dominates decoding. Full-cache attention accounts for only 3.3 ms of the 67.2 ms step. The compacted path adds approximately 7 ms through FlexAttention and host-dispatch overhead. Absolute throughput is therefore specific to this implementation.

At approximately equal peak memory (96–97 GB), compaction nevertheless enables batch size 64 compared with batch size 2 for Full KV, raising measured throughput from 21 to 165 tokens/s.

Amortization over reuses. Calibration and compaction are one-time costs per context, amortised by reusing the compacted cache across requests. Full KV also prefills once, so prefill is excluded from this additional cost. For the equal-memory batched comparison, the break-even reuse count is N∗=⌈C/(sfull−scompact)⌉,N^*=\left\lceil C/(s_{\rm full}-s_{\rm compact})\right\rceil, where CC is calibration plus compaction time, and ss is batch execution time divided by batch size. The measured saving of approximately 6.3 seconds per request gives about 400 requests per context for KV-Surgeon and about 14 for KV-Surgeon-fast.

This throughput benefit requires enough concurrent requests to fill the compact-cache batch, hence, it does not apply to isolated sequential requests. We will add these measurements and amortization assumptions to the main text.

Thank you for your detailed rebuttal and the additional experiments.These additional results help address my concerns and strengthen the practical significance of the proposed approach. Although the methodological novelty remains somewhat limited, I find the overall contribution meaningful. Therefore, I will maintain my original positive rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.