acceptodds
Under review as a conference paper at ICLR 2027

AbsorbKV: Co-designing Selection and Writing for KV Cache Merging

Abstract

KV cache compression reduces the memory cost of long-context inference, but eviction irreversibly discards states that future queries may need. KV cache merging instead transfers information from evicted states into retained ones. However, selection constrains the effectiveness of merging by determining which retained states are available to receive information from evicted ones. Existing methods largely overlook this dependency, so a state may be evicted even when none of the retained states can faithfully preserve its information. We introduce *absorbability* to characterize, relative to a given set of retained states, whether the information of an evicted state can remain readable through a retained key without compromising the information originally stored in that retained state. Guided by this insight, we propose AbsorbKV, which co-designs a Hybrid Selector and Adaptive Absorption. The Hybrid Selector jointly considers importance and absorbability to retain states that are important or difficult to absorb. Adaptive Absorption then merges each evicted state only into a compatible retained state and learns how to adapt its information to the retained key through which future queries will retrieve it, while keeping retained keys unchanged. Extensive experiments demonstrate that AbsorbKV outperforms representative KV cache compression methods across diverse long-context tasks and models. On Qwen3-8B with a 10% KV cache budget, AbsorbKV achieves 42.10 on LongBench and 39.01 on RULER-16K, surpassing representative state-of-the-art KV merging methods such as AsymKV, which obtains 35.44 and 25.59, respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV Cache Compression, Long-Context Inference, Large Language Models, KV Cache Merging, Attention

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JzZPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AbsorbKV addresses KV-cache compression by jointly designing state selection and information merging. Its Hybrid Selector retains states that are important or difficult to absorb, while Adaptive Absorption routes evicted states to compatible retained carriers and learns carrier-conditioned value updates. Retained keys remain unchanged, allowing subsequent decoding to use standard attention. Experiments show improved average performance across multiple models and long-context tasks, particularly under aggressive compression.

Critical strengths
  1. The paper identifies a meaningful dependency between selection and merging: retained states determine both what information survives directly and which retrieval addresses remain available for evicted information. This provides a clear motivation for designing the two operations together.
  2. Carrier-conditioned writing explicitly accounts for the change in retrieval address after merging. Considering both key and value compatibility is well motivated, and preserving retained keys allows the method to work with standard attention kernels without auxiliary KV slots or additional attention biases.
  3. The empirical evaluation covers multiple model families, diverse task categories, and extended context lengths. Component ablations support an interaction between selection and writing, while transfer from benchmark-derived training data to synthetic tasks provides evidence that the learned writer offers benefits beyond its training examples.
Critical weaknesses
  1. Keeping carrier keys unchanged imposes limitations that the learned writer cannot generally overcome. A query-independent value update may struggle when future queries need to distinguish merged sources from the carrier’s original content. The reconstruction analysis motivates support selection but does not establish that the proposed writer preserves both.
  2. The selection ablations do not fully separate improved carrier compatibility from changes in capacity allocation. Global ranking can redistribute retained states across layers and heads, so gains attributed to absorbability-aware selection may partly arise from allocating cache capacity differently.
  3. The comparisons do not fully distinguish the proposed co-design from the benefits of supervised writer training. The writer receives task-aligned reconstruction supervision, whereas the principal merging baselines lack comparable learned adaptation. Synthetic-task transfer provides useful evidence, but broader transfer to unfamiliar natural-language workloads remains uncertain.
Questions for authors

Under what conditions can the writer preserve source information without corrupting carrier information, and how does it behave when future queries distinguish otherwise similar source–carrier pairs?

Does absorbability-aware selection remain beneficial when per-layer and per-head capacities are fixed, allowing only the identities of retained states to change?

Does AbsorbKV retain its advantage when competing selection and merging strategies receive an equivalently trained writer, and when evaluation uses task formats unrelated to writer training?

4 comments

We thank the reviewer for the very positive comments!

Q [When Can Writing Preserve Both Source and Carrier Information?]

A: Thank you for raising this question. AbsorbKV aims to recover the source information without disrupting the information already accessible through the carrier. Appendix B analyzes this through two complementary aspects: recoverability under fixed retained keys and the conditions under which writing reduces reconstruction error.

Recoverability. A carrier key determines how its value is accessed by future queries. Since writing changes only values, it cannot create new attention patterns. Proposition 1 formalizes this constraint: only the component of the missing attention output lying in the column space of the retained attention matrix can be recovered. Exact preservation is possible when the Full KV outputs can be represented by the retained attention patterns. Conversely, if different queries require different outputs but induce indistinguishable retained attention patterns, no value update can fully preserve both.

Beneficial Writing. Recoverability alone does not guarantee that a particular write improves reconstruction. Let EE denote the residual after direct eviction and U=ASΔVU=A_S\Delta V the output change induced by writing. Appendix B further shows that writing reduces reconstruction error only when

2⟨E,U⟩F>∣∣U∣∣F2.2\langle E,U\rangle_F>||U||_F^2.

Thus, the update must align sufficiently with the missing output, with an appropriate magnitude. Our carrier-conditioned transformation and absorption weights are designed to address these two aspects. Together with absorbability-aware selection and compatibility-based routing, these mechanisms mitigate the limitation, but cannot guarantee exact preservation for arbitrary future queries.

A Case Study.

To illustrate the recoverability and limitations of carrier-conditioned writing, we analyze a real question-answering example from our Qwen3-8B evaluation on LongBench. The relevant context contains two passages reporting different numerical improvements:

  • WikiLarge: "... improvement over our baseline NMF by 2.11 BLEU, 1.7 FKGL ..."
  • WikiSmall: "... substantial improvements (6.37 BLEU) than NMT ..."

At the last layer, only "7" in "1.7 FKGL" is retained, while all others are evicted. "7" serves as a carrier for 13 source tokens, including all tokens representing "6.37" and "2.11". We examine three queries: Q1 asks about WikiSmall, while Q2 and Q3 ask about different metrics (BLEU and FKGL) of the same WikiLarge comparison.

QueryGround TruthDirect EvictAbsorbKV
Q1: WikiSmall BLEU6.376.34 ✗6.37 ✓
Q2: WikiLarge BLEU2.111.7 ✗1.7 ✗
Q3: WikiLarge FKGL1.71.7 ✓1.7 ✓

AbsorbKV recovers the correct answer to Q1 while maintaining the correct carrier-associated answer to Q3. However, Q2 still returns the FKGL value (1.7) instead of the BLEU value (2.11). This illustrates both the benefit and limitation of carrier-based writing: some source information can be recovered without disrupting an existing answer, but distinct facts sharing a fixed retrieval address may remain difficult to distinguish. Furthermore, although AsymKV modifies keys, AbsorbKV still outperforms it. For instance, on Qwen3-8B with a 10% KV cache budget, AbsorbKV achieves 42.10 vs. 35.44 on LongBench and 39.01 vs. 25.59 on RULER-16K.

Q [Absorbability-Aware Selection Remains Effective under Fixed Capacity Allocation?]

A: Thank you for this insightful question. To isolate the effect of absorbability-aware selection from capacity allocation, we conduct an eviction-only experiment on Qwen3-8B using the same six LongBench tasks as in our ablation study. We fix an identical 10% budget for every layer and KV head, allowing only the identities of retained states to change.

SelectionCapacity AllocationScore
Importance-onlyUniform42.98
HybridUniform43.99
HybridGlobal Ranking44.32

Under fixed capacities, Hybrid selection improves performance by 1.01 over importance-only selection. Allowing global ranking yields a modest additional gain of 0.33. Overall, these results demonstrate that absorbability-aware selection remains beneficial even when capacity allocation is fixed.

Q [Matched writer training]

A: Thank you for this thoughtful question To control for the benefit of writer supervision, we equip CaM, AsymKV, and D2O with learned value residuals added to their native merging outputs. All writers use the same initial checkpoint, training examples and order, and 250 training steps, matching the training protocol used for AbsorbKV. We evaluate Qwen3-8B at a 10% KV cache budget on the same six LongBench tasks used in our ablation study:

MethodOriginal+ Learned WriterΔ
CaM26.3826.05-0.33
D2O30.6033.77+3.17
AsymKV39.1141.26+2.15
AbsorbKV–47.70–

Learned writers improve D2O and AsymKV by 3.17 and 2.15, respectively, but slightly degrade CaM. Nevertheless, AbsorbKV outperforms the strongest adapted baseline by 6.44 points. These results show that supervised writer training alone does not explain AbsorbKV's advantage, highlighting the importance of coordinating selection and writing.

Q [Transfer to Unseen Tasks]

A: The RULER benchmark provides evidence of writer generalization, since the writer is trained exclusively on examples constructed from the original source datasets of LongBench and is applied to RULER without any additional training or adaptation. Moreover, RULER differs substantially from LongBench in both task construction and query patterns, particularly through its synthetic retrieval, multi-hop tracing, and aggregation tasks.

To further evaluate generalization to more distinct tasks, we extend our evaluation to mathematical reasoning on AIME25 and MATH500. We consider KV cache budgets of 512, 1024, 1536, and 2048, reusing the same trained writer without adaptation. Compression is performed every 128 generated tokens, with a maximum generation length of 8K. The table reports accuracy, averaged across the four budgets for the mathematical tasks.

MethodAIME25MATH500
Full KV26.6786.20
CaM4.1763.15
D2O14.1772.25
AsymKV8.3367.95
CUR16.6779.80
AbsorbKV21.6780.95

AbsorbKV achieves the highest score on both mathematical benchmarks. These results provide additional evidence that the learned writer transfers beyond its training tasks, including to unfamiliar natural-language comprehension and mathematical reasoning formats. We have included the complete accuracy results across all four KV cache budgets for AIME25 and MATH500 in Section 4.3 of the revised manuscript, providing further evidence of the writer's generalization to unseen tasks.

Dear Reviewer JzZP,

We sincerely appreciate your constructive comments. We would like to kindly ask whether our responses and revisions have sufficiently resolved your concerns about our work.

If there are any remaining issues, please feel free to let us know. We would be glad to address them further.

Thank you for your time and consideration.

Reviewer_ksn9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AbsorbKV studies KV-cache merging and argues that selection and writing should be designed together. It introduces absorbability, a measure of whether an evicted KV state can be represented through a retained state without destroying the retained state’s original content. The method combines importance with absorbability when selecting retained states, then routes compatible evicted states to retained carriers and uses a small learned writer to update their values.

Critical strengths
  1. The paper identifies a real dependency that existing KV merging methods often overlook: the retained support determines which evicted states can still be represented. The simple example in Figure 1 makes this motivation clear.
  2. The method is evaluated across three models, multiple context lengths, two benchmarks, and several cache budgets. The gains are largest where the problem is hardest. At a 10% budget on Qwen3-8B, AbsorbKV improves over CUR from 38.62 to 42.10 on LongBench and from 25.59 to 39.01 on RULER-16K.
Critical weaknesses
  1. The learned writer introduces a nontrivial training and deployment dependency. The writer is trained on LongBench-derived examples and appears to be model-specific, but the paper does not clearly state whether a separate writer is trained for each backbone, how much data and compute each writer needs, or how well a writer transfers across models and task distributions. RULER generalization is useful, but it does not establish cross-model transfer.

  2. The throughput advantage depends strongly on workload length. At short prompt-and-generation lengths, AbsorbKV is slower than some baselines because selection, routing, and writing are paid during compression. The reported 2.12–3.17× speedup is relative to Full KV, while the paper also acknowledges that other compression methods can have higher throughput. The practical break-even point and the amortization across repeated requests deserve clearer treatment.

Questions for authors
  1. Are separate writers trained for Llama-3.1-8B, Qwen3-8B, and Qwen3-4B? Please report the training cost, number of examples, and whether a writer trained on one backbone can transfer to another.

  2. What is the end-to-end break-even point for AbsorbKV compared with direct eviction and other merging methods, as a function of prompt length, generation length, and the number of compression events?

  3. Could you report quality, compression-time latency, peak memory, and final decode throughput together under the same hardware and backend for all main baselines?

5 comments

We thank the reviewer for the very positive comments!

Q [Writer training cost and cross-model transfer]

A: Thank you for this interesting question. Since the writer operates on backbone-specific KV representations, whose feature spaces are not aligned across models, we train a separate writer for each backbone in our experiments by default.

As illustrated in Appendix A.2, the training cost is modest. Each writer uses 1500 examples from LongBench source tasks (100 per task except MF-en), disjoint from the evaluation set. With the backbone frozen, training takes 250 steps at approximately 8.5 seconds per step on a single A100 GPU, totaling ~35 minutes. The writer adds only 0.079% trainable parameters on Qwen3-8B. Each writer is trained once and reused across tasks, with no additional training on RULER.

We evaluate cross-model transfer to examine whether the learned writing operation captures reusable patterns beyond its training backbone. The table reports average scores across six LongBench tasks (as used for ablation study) at a 10% KV cache budget.

Writer / Target backboneQwen3-4BQwen3-8B
Qwen3-4B47.5846.42
Qwen3-8B45.5247.70
Llama-3.1-8B44.2844.02
CUR (no writer)42.3542.97

Within-family transfer incurs a 1.28–2.06 point decrease relative to backbone-specific writers, while still outperforming CUR by 3.17–3.45 points. Cross-family transfer incurs a larger 3.30–3.68 point decrease, yet remains 1.05–1.93 points above CUR. These results indicate that while backbone-specific training achieves the best performance, the writer does capture reusable patterns beyond its training backbone. Combined with the zero-shot transfer from LongBench to RULER, they demonstrate generalization across both model and task distributions. We have included the cross-model transfer experiments in Appendix C.8 of the revised manuscript.

Q [End-to-end break-even and amortization]

A: Thanks for the great question. We distinguish break-even against competing methods from the amortization of compression overhead. At a fixed KV cache budget, compression latency primarily depends on input length, while decode throughput is comparable across methods when batch size=1 due to similar KV cache memory traffic. End-to-end efficiency therefore depends jointly on prompt length (LL), generation length (GG), batch size (bb), and the number of compression events (EE). Batch size affects serving throughput and memory requirements, while repeated compression events accumulate additional latency.

We define a compression overhead threshold as

ηcomp(L,G,E,b)=∑e=1EC(Le,b)T(L,G,E,b),\eta_{\mathrm{comp}}(L,G,E,b) =\frac{\sum_{e=1}^{E}C(L_e,b)} {T(L,G,E,b)},

where CC and TT denote compression and end-to-end latency, respectively. An application-specific requirement such as ηcomp≤10\eta_{\mathrm{comp}}\leq 10% defines a range of acceptable workload configurations (LL, GG, bb, EE), e.g. break-even point.

Under b=1b=1, E=1E=1, and a 10% KV-cache budget, we report the compression overhead at G=512G=512 and the estimated generation lengths required to reduce it below 10%:

ContextMethodOverhead (G=512)G for ≤10%
16KAbsorbKV26.9%~1.9K
AsymKV30.8%~2.2K
32KAbsorbKV29.1%~2.5K
AsymKV50.8%~6.1K

Under a constant decode throughput, AbsorbKV reaches the 10% threshold at shorter generation lengths than AsymKV for both context sizes. Meanwhile, it achieves 13.42 and 19.15 higher RULER scores, respectively. More compression events increase the cost, while reuse of compressed prefixes amortizes it across requests. Thus, AbsorbKV can readily reach the practical amortization threshold with relatively modest workload configurations, while delivering substantially higher compression quality.

A: Thank you for this excellent suggestion. To clarify the accuracy-efficiency trade-off, we conduct experiments on Qwen3-8B using RULER-16K and RULER-32K, with a 10% KV cache budget and 512 output tokens on the same hardware. Prefill latency, compression latency, and peak memory are measured at batch size 1 and with one compress event. Decoding throughput is measured at the maximum supported batch size for each method. Note that all methods except AsymKV use the FlashAttention backend. AsymKV requires additional logit bias for each value item, which are not supported by FlashAttention.

ContextMethodScore ↑Prefill (s) ↓Compress (s) ↓Decode (tok/s) ↑Peak (GB) ↓
16KCUR12.662.430.04318118.27
CaM8.442.420.30317620.52
D2O10.822.430.40318218.27
AsymKV25.593.5014.0314322.00
AbsorbKV39.012.436.61311621.05
32KCUR11.316.990.04193620.95
CaM8.837.000.48191923.44
D2O12.807.000.81193820.95
AsymKV26.579.1238.407924.41
AbsorbKV45.726.999.35189624.18

AbsorbKV achieves the highest quality at both lengths, outperforming the best baseline, AsymKV, by 13.42 and 19.15, respectively. Regarding efficiency, AsymKV requires per-slot attention-logit biases that are unsupported by our FlashAttention implementation. It therefore exhibits substantially lower decode throughput. In contrast, AbsorbKV remains compatible with FlashAttention and achieves maximum decode throughput within 2.1% of CUR at both 16K and 32K context length.

These results make the trade-off explicit: AbsorbKV prioritizes compression quality, rather than universally outperforming simpler eviction methods in speed. Since selection, routing, and writing are performed only during compression, their cost can be amortized when the same compressed prefix is reused across multiple requests. We have also included the complete efficiency comparison across all baselines at 16K–128K context lengths in Appendix C.7 of the revised manuscript.

Reviewer_eMhkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AbsorbKV, a method that jointly designs the selection of states to retain in a KV cache and the integration of information from discarded states into those that are kept. The core insight is that the extent to which information from a discarded state can be reconstructed from the retained states depends on which states are chosen for retention.

To this end, a "Hybrid Selector" considers both importance scores and "absorbability" (the feasibility of incorporating one state into another), prioritizing the retention of KV states that are either highly important or difficult to absorb into other states. Subsequently, a learned "writer" transforms and merges information across compatible states; in this process, the retained keys remain unchanged, while only the corresponding values ​​are updated.

The writer is trained using the outputs of a model—with its backbone frozen—that utilizes the original, uncompressed KV cache. The proposed method is evaluated on the LongBench and RULER benchmarks across three different model configurations. Its effectiveness is validated through comparisons with existing pruning and merging techniques, ablation studies of the selector and writer components, and experiments involving context lengths of up to 128K tokens.

Critical strengths
  1. The paper demonstrates the interaction between selection and merging through ablation experiments. As shown in Table 2, the performance gain for the writer model is 1.28 points with importance-based selection and 2.99 points with the Hybrid Selector. This supports the motivation for co-design—that selecting the right states to retain makes subsequent merging more effective.

  2. Performance improvements are observed across various models and on benchmarks not used during training. With a 10% KV budget, the average score on LongBench for all three models is 2.64–3.53 points higher than that of CUR (Table 1). Furthermore, when applied to RULER without any additional training, Llama-3.1-8B achieves a score of 47.27 points (at a 128K context length and 25% budget), outperforming D²O’s 25.66 points.

Critical weaknesses
  1. The link between the definition of "absorbability" and the actual selection scores has not been sufficiently validated. While Eq. (3) utilizes the similarity to the mean direction of the total K/V, actual absorbability depends on the compatibility with the selected carrier. Since Figure 1(b) examines the similarity of individual source–carrier pairs, it remains unclear how well the scores derived from Eq. (3) predict carrier compatibility and reconstruction error following the selection process.

  2. The accuracy–efficiency comparison remains incomplete for relevant query-agnostic methods and longer contexts. KVzip—which is directly relevant to the evaluation setting of "pre-query compression"—is omitted from the comparisons. Furthermore, the cost analysis in Figure 6 is limited to a maximum input length of 8K, and the throughput of AbsorbKV is lower than that of the compression baselines used for comparison. Consequently, it has not been sufficiently demonstrated whether the accuracy gains observed at 32K–128K sequence lengths remain competitive when accounting for the associated computational costs.

Questions for authors
  1. Regarding Weakness 1: Could you categorize the removed sources based on their scores from Eq. (3) and provide both the maximum similarity to the retained carriers and the output reconstruction error following the merging process?

  2. Regarding Weakness 2: Could you compare your method against KVzip in terms of accuracy and total inference time (including prefill, compression, and decoding) using a standard Qwen3-8B configuration, while maintaining the same KV budget and execution conditions?

6 comments

We thank the reviewer for the very positive comments!

Q [Validating Eq. (3) as an Absorbability Proxy]

Thank you for highlighting this distinction and suggesting a direct validation. We agree that Fig. 1(b) examines similarity between individual source–carrier pairs, which alone does not establish whether the mean direction similarity in Eq. (3) is a useful absorbability proxy.

Following your suggestion, we used Qwen3-8B on MultiFieldQA-en, the same model and dataset as in Fig. 1 for the following analysis. We categorize the sources into 5 quantiles of their Eq. (3) scores, with Q1 denoting the lowest scores and Q5 the highest. For each source, we computed its maximum joint K/V cosine similarity over the retained carriers, then averaged within each quantile. We also measured the reconstruction MSE after merging.

Eq. (3) score quantileQ1Q2Q3Q4Q5
maximum carrier cosine0.7013930.7099870.7138880.7185260.723845
reconstruction error132.81260.86459.22142.52431.939

As shown in the table, higher Eq. (3) scores are associated with higher best carrier similarity and lower reconstruction error. From Q1 to Q5, the maximum cosine increases, while MSE decreases by approximately 76%. The results support Eq. (3) as a useful proxy for absorbability during selection, when the retained carrier set is not yet known. The additional experiments and analyses have been included in Appendix C.6.

Q [Accuracy and End-to-End Efficiency vs. KVzip]

A: Thank you for highlighting KVzip as a relevant baseline for pre-query compression. We have added comparisons with KVzip under 10% budgets and the same execution conditions. We evaluate six LongBench tasks used in our ablation study and RULER task (QA, CWE, FWE, and VT) for longer context. Since the standard Qwen3-8B supports a maximum context length of 32K, we use Qwen3-8B for contexts up to 32K and Llama-3.1-8B-Instruct for longer contexts, ensuring consistent model configurations across methods.

We additionally replace CUR importance score of AbsorbKV with KVZip, since AbsorbKV is independent of the importance estimator.

BenchmarkKVzipAbsorbKVKVZip+AbsorbKV
LongBench45.9847.7047.68
RULER-32K59.2539.3373.92
RULER-64K53.9036.4063.03
RULER-128K39.2321.3340.87

On LongBench, AbsorbKV outperforms KVzip by 1.72, while KVzip+AbsorbKV achieves comparable performance. On RULER, KVzip outperforms the original AbsorbKV. However, incorporating KVzip's importance scores into AbsorbKV improves over KVzip by 14.67, 9.13, and 1.64 at 32K, 64K, and 128K, respectively. These results demonstrate that AbsorbKV can effectively complement different importance estimators to accommodate diverse tasks.

For inference efficiency, we test the input lengths from 16K to 128K, using RULER benchmark with a 10% KV cache budget. Prefill latency, compression latency, and peak memory are measured at batch size 1. Decoding throughput is measured at the maximum supported batch size for each method.

Input lengthMethodPrefill (s)Compress (s)Decode throughput (tok/s)Peak Memory (GB)
16K (Qwen3-8B)KVzip2.425.74298622.01
16K (Qwen3-8B)AbsorbKV2.426.61311621.05
32K (Qwen3-8B)KVzip7.0216.07178628.76
32K (Qwen3-8B)AbsorbKV6.999.35189624.18
64K (Llama-3.1-8B)KVzip20.1245.24108338.97
64K (Llama-3.1-8B)AbsorbKV20.0714.73112829.13
128K (Llama-3.1-8B)KVzip67.57151.7251361.84
128K (Llama-3.1-8B)AbsorbKV67.5832.5358839.10

Under matched input lengths, KV cache budgets, and execution conditions, both methods exhibit comparable prefill costs. During compression, KVzip requires an additional forward pass for context reconstruction, doubling the prompt length (e.g., from 32K to 64K tokens) and enlarge peak memory pressure. This introduces substantial overhead for long contexts. Although we adopt chunked scoring (chunk size=2048) to reduce the complexity from O(n2)O(n^2) to O(mn)O(mn), where nn is the prompt length and mm is the chunk size, KVZip still incurs innegligible overhead.

Unlike KVzip, AbsorbKV operates directly on the cached keys and values without additional forward passes. The importance scores require only blockwise K/V norm computations, followed by similarity routing and lightweight value updates. The measured compression time grows more moderately: from 64K to 128K, it increases from 14.73 to 32.53 seconds, scaling approximately linearly with prompt length.

In addition, KVzip's highly imbalanced token allocation across KV heads leads to slightly lower decoding throughput than AbsorbKV, although their throughput remains comparable. We have included the KVzip quality comparison in Section 4.6 and the corresponding efficiency analysis in Appendix C.7 of the revised manuscript.

Thank you for the detailed response and additional experiments. The analysis of Eq. (3) substantially addresses my first concern.

However, I remain concerned about the accuracy–efficiency trade-off. The current comparison focuses on the original KVzip, whose expensive reconstruction process may not reflect more efficient alternatives such as Fast KVzip. Could the authors compare AbsorbKV with Fast KVzip under the same model, KV budget, and hardware, reporting both accuracy and end-to-end inference time at longer context lengths (e.g., 32K–128K)?

Additionally, since the strongest RULER results are achieved using KVzip importance scores, please report the compression latency of KVzip+AbsorbKV to clarify whether its accuracy improvements are accompanied by comparable efficiency.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviewers feedback and authors response

Dear Chairs and Reviewers,

We sincerely thank all reviewers for their constructive feedback and valuable suggestions, which have helped us substantially strengthen the paper.

Strengths: Reviewers consistently recognized the importance and clear motivation of co-designing selection and writing for KV cache merging, highlighting the overlooked dependency between retained states and the recoverability of evicted information. They also appreciated the carrier-conditioned writing mechanism and its compatibility with standard attention kernels, as well as the strong empirical performance across multiple models, diverse tasks, and aggressive compression budgets.

Response to Main Concerns

We have carefully addressed the main concerns raised by the reviewers through additional theoretical analyses and comprehensive evaluations:

  • Absorbability Proxy Validation (Reviewer eMhk): We validated the absorbability score in Eq. (3) by grouping evicted states into five quantiles. Higher scores are associated with greater carrier compatibility and lower reconstruction error, supporting Eq. (3) as a useful proxy during selection. (Added in Appendix C.6.)
  • Accuracy–Efficiency Trade-offs (Reviewers eMhk, ksn9): We expanded efficiency evaluations to context lengths of 16K–128K, reporting compression latency, peak GPU memory, and decoding throughput alongside accuracy results. We further replaced AbsorbKV's importance estimator with KVzip and Fast KVzip, demonstrating that AbsorbKV can accommodate different importance estimators to balance accuracy and inference efficiency. (Added in Section 4.6, Appendix C.7 and Appendix C.9.)
  • Writer Generalization (Reviewers ksn9, JzZP): We further evaluated cross-model transfer and extended the evaluation to unseen mathematical reasoning tasks (AIME25 and MATH500), demonstrating the writer's generalization beyond its training tasks. (Added in Section 4.3 and Appendix C.8.)
  • Recoverability and Conditions for Beneficial Writing (Reviewer JzZP): We theoretically characterized the reconstruction limits under fixed retained keys and derived the conditions under which writing reduces reconstruction error. We also provided a real case study illustrating both successful information recovery and its limitations. (Added in Appendix B.)
  • Effectiveness of Selection–Writing Co-design (Reviewer JzZP): We conducted experiments with uniform budget allocation and equipped baselines with equally trained writers. The results show that absorbability-aware selection remains beneficial under fixed budgets, while AbsorbKV's advantage over equally trained baselines further supports the effectiveness of our selection–writing co-design. (Additional experiments provided in our response.)

We believe the revised manuscript provides a more rigorous and comprehensive evaluation of selection–writing co-design for KV cache merging.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.