- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AbsorbKV addresses KV-cache compression by jointly designing state selection and information merging. Its Hybrid Selector retains states that are important or difficult to absorb, while Adaptive Absorption routes evicted states to compatible retained carriers and learns carrier-conditioned value updates. Retained keys remain unchanged, allowing subsequent decoding to use standard attention. Experiments show improved average performance across multiple models and long-context tasks, particularly under aggressive compression.
Critical strengths
- The paper identifies a meaningful dependency between selection and merging: retained states determine both what information survives directly and which retrieval addresses remain available for evicted information. This provides a clear motivation for designing the two operations together.
- Carrier-conditioned writing explicitly accounts for the change in retrieval address after merging. Considering both key and value compatibility is well motivated, and preserving retained keys allows the method to work with standard attention kernels without auxiliary KV slots or additional attention biases.
- The empirical evaluation covers multiple model families, diverse task categories, and extended context lengths. Component ablations support an interaction between selection and writing, while transfer from benchmark-derived training data to synthetic tasks provides evidence that the learned writer offers benefits beyond its training examples.
Critical weaknesses
- Keeping carrier keys unchanged imposes limitations that the learned writer cannot generally overcome. A query-independent value update may struggle when future queries need to distinguish merged sources from the carrier’s original content. The reconstruction analysis motivates support selection but does not establish that the proposed writer preserves both.
- The selection ablations do not fully separate improved carrier compatibility from changes in capacity allocation. Global ranking can redistribute retained states across layers and heads, so gains attributed to absorbability-aware selection may partly arise from allocating cache capacity differently.
- The comparisons do not fully distinguish the proposed co-design from the benefits of supervised writer training. The writer receives task-aligned reconstruction supervision, whereas the principal merging baselines lack comparable learned adaptation. Synthetic-task transfer provides useful evidence, but broader transfer to unfamiliar natural-language workloads remains uncertain.
Questions for authors
Under what conditions can the writer preserve source information without corrupting carrier information, and how does it behave when future queries distinguish otherwise similar source–carrier pairs?
Does absorbability-aware selection remain beneficial when per-layer and per-head capacities are fixed, allowing only the identities of retained states to change?
Does AbsorbKV retain its advantage when competing selection and merging strategies receive an equivalently trained writer, and when evaluation uses task formats unrelated to writer training?
4 comments
Q [Absorbability-Aware Selection Remains Effective under Fixed Capacity Allocation?]
A: Thank you for this insightful question. To isolate the effect of absorbability-aware selection from capacity allocation, we conduct an eviction-only experiment on Qwen3-8B using the same six LongBench tasks as in our ablation study. We fix an identical 10% budget for every layer and KV head, allowing only the identities of retained states to change.
| Selection | Capacity Allocation | Score |
|---|---|---|
| Importance-only | Uniform | 42.98 |
| Hybrid | Uniform | 43.99 |
| Hybrid | Global Ranking | 44.32 |
Under fixed capacities, Hybrid selection improves performance by 1.01 over importance-only selection. Allowing global ranking yields a modest additional gain of 0.33. Overall, these results demonstrate that absorbability-aware selection remains beneficial even when capacity allocation is fixed.
Q [Matched writer training]
A: Thank you for this thoughtful question To control for the benefit of writer supervision, we equip CaM, AsymKV, and D2O with learned value residuals added to their native merging outputs. All writers use the same initial checkpoint, training examples and order, and 250 training steps, matching the training protocol used for AbsorbKV. We evaluate Qwen3-8B at a 10% KV cache budget on the same six LongBench tasks used in our ablation study:
| Method | Original | + Learned Writer | Δ |
|---|---|---|---|
| CaM | 26.38 | 26.05 | -0.33 |
| D2O | 30.60 | 33.77 | +3.17 |
| AsymKV | 39.11 | 41.26 | +2.15 |
| AbsorbKV | – | 47.70 | – |
Learned writers improve D2O and AsymKV by 3.17 and 2.15, respectively, but slightly degrade CaM. Nevertheless, AbsorbKV outperforms the strongest adapted baseline by 6.44 points. These results show that supervised writer training alone does not explain AbsorbKV's advantage, highlighting the importance of coordinating selection and writing.
Q [Transfer to Unseen Tasks]
A: The RULER benchmark provides evidence of writer generalization, since the writer is trained exclusively on examples constructed from the original source datasets of LongBench and is applied to RULER without any additional training or adaptation. Moreover, RULER differs substantially from LongBench in both task construction and query patterns, particularly through its synthetic retrieval, multi-hop tracing, and aggregation tasks.
To further evaluate generalization to more distinct tasks, we extend our evaluation to mathematical reasoning on AIME25 and MATH500. We consider KV cache budgets of 512, 1024, 1536, and 2048, reusing the same trained writer without adaptation. Compression is performed every 128 generated tokens, with a maximum generation length of 8K. The table reports accuracy, averaged across the four budgets for the mathematical tasks.
| Method | AIME25 | MATH500 |
|---|---|---|
| Full KV | 26.67 | 86.20 |
| CaM | 4.17 | 63.15 |
| D2O | 14.17 | 72.25 |
| AsymKV | 8.33 | 67.95 |
| CUR | 16.67 | 79.80 |
| AbsorbKV | 21.67 | 80.95 |
AbsorbKV achieves the highest score on both mathematical benchmarks. These results provide additional evidence that the learned writer transfers beyond its training tasks, including to unfamiliar natural-language comprehension and mathematical reasoning formats. We have included the complete accuracy results across all four KV cache budgets for AIME25 and MATH500 in Section 4.3 of the revised manuscript, providing further evidence of the writer's generalization to unseen tasks.
Dear Reviewer JzZP,
We sincerely appreciate your constructive comments. We would like to kindly ask whether our responses and revisions have sufficiently resolved your concerns about our work.
If there are any remaining issues, please feel free to let us know. We would be glad to address them further.
Thank you for your time and consideration.
We thank the reviewer for the very positive comments!
Q [When Can Writing Preserve Both Source and Carrier Information?]
A: Thank you for raising this question. AbsorbKV aims to recover the source information without disrupting the information already accessible through the carrier. Appendix B analyzes this through two complementary aspects: recoverability under fixed retained keys and the conditions under which writing reduces reconstruction error.
Recoverability. A carrier key determines how its value is accessed by future queries. Since writing changes only values, it cannot create new attention patterns. Proposition 1 formalizes this constraint: only the component of the missing attention output lying in the column space of the retained attention matrix can be recovered. Exact preservation is possible when the Full KV outputs can be represented by the retained attention patterns. Conversely, if different queries require different outputs but induce indistinguishable retained attention patterns, no value update can fully preserve both.
Beneficial Writing. Recoverability alone does not guarantee that a particular write improves reconstruction. Let E denote the residual after direct eviction and U=ASΔV the output change induced by writing. Appendix B further shows that writing reduces reconstruction error only when
2⟨E,U⟩F>∣∣U∣∣F2.
Thus, the update must align sufficiently with the missing output, with an appropriate magnitude. Our carrier-conditioned transformation and absorption weights are designed to address these two aspects. Together with absorbability-aware selection and compatibility-based routing, these mechanisms mitigate the limitation, but cannot guarantee exact preservation for arbitrary future queries.
A Case Study.
To illustrate the recoverability and limitations of carrier-conditioned writing, we analyze a real question-answering example from our Qwen3-8B evaluation on LongBench. The relevant context contains two passages reporting different numerical improvements:
At the last layer, only "7" in "1.7 FKGL" is retained, while all others are evicted. "7" serves as a carrier for 13 source tokens, including all tokens representing "6.37" and "2.11". We examine three queries: Q1 asks about WikiSmall, while Q2 and Q3 ask about different metrics (BLEU and FKGL) of the same WikiLarge comparison.
AbsorbKV recovers the correct answer to Q1 while maintaining the correct carrier-associated answer to Q3. However, Q2 still returns the FKGL value (1.7) instead of the BLEU value (2.11). This illustrates both the benefit and limitation of carrier-based writing: some source information can be recovered without disrupting an existing answer, but distinct facts sharing a fixed retrieval address may remain difficult to distinguish. Furthermore, although AsymKV modifies keys, AbsorbKV still outperforms it. For instance, on Qwen3-8B with a 10% KV cache budget, AbsorbKV achieves 42.10 vs. 35.44 on LongBench and 39.01 vs. 25.59 on RULER-16K.