- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper investigates how document-specific KV-cache compression methods affect the general capabilities of LLMs. It finds that learned compression methods such as Cartridges, despite generally achieving better performance on document-related question answering, can significantly interfere with other model capabilities, including instruction following and answering unrelated questions. In contrast, simpler eviction-based methods tend to preserve these capabilities better, although they generally perform worse on document-related tasks. The authors attribute this behavior to the training objective of Cartridges, which optimizes the compressed KV representations exclusively on question-answer pairs related to the document, without explicitly preserving the model's behavior on unrelated queries. To address this issue, they introduce Cartridges++, which proposes two mitigation strategies. The first mixes document-related and unrelated question-answer pairs during KV optimization, using the original LLM without context as the teacher for unrelated queries. The second introduces a nearest-neighbor-based relevance router that determines whether a given query requires access to the compressed KV cache. Through extensive experiments across multiple LLMs, compression methods, and benchmarks, the authors demonstrate that optimization-based compression methods can substantially degrade general model capabilities despite their superior document-specific performance. They further show that both proposed mitigation strategies significantly reduce this degradation while largely preserving document-related performance.
Critical strengths
-
The proposed mitigation strategies are simple, intuitive, and easy to incorporate into the original Cartridges framework.
-
The paper presents extensive experiments that convincingly demonstrate the existence of the identified problem and the effectiveness of the proposed mitigation strategies.
Critical weaknesses
-
The practical significance of the problem is somewhat unclear. If a query does not require information from the compressed document, a straightforward solution is simply not to use the Cartridge. Therefore, it is unclear to what extent the identified interference constitutes a fundamental limitation of KV-cache compression rather than a deployment issue that can be avoided through appropriate memory selection.
-
The evaluation primarily considers two clearly separated types of queries: those related to the document and those completely unrelated to it. In realistic applications, queries may depend only partially on the document, combine document-specific information with general knowledge, or be closely related to the document's topic without actually requiring its contents. Such ambiguous cases seem both more challenging and more representative of realistic use. Evaluating the proposed methods in these settings would provide stronger evidence of their practical effectiveness, particularly for the relevance-based routing approach.
-
The methodological contribution is relatively limited. Both proposed methods are minor modifications to the original Cartridges framework, relying on familiar techniques such as mixing general-purpose training data and relevance-based routing. While the empirical findings are interesting, the proposed solutions themselves seem somewhat incremental, raising questions about whether they constitute a sufficiently substantial methodological contribution for a full conference paper.
Questions for authors
-
How well do the proposed methods generalize to queries that are only partially related to the compressed document, combine document-specific and general knowledge, or are semantically similar to document-related queries without requiring the document?
-
Could the authors provide an end-to-end computational efficiency analysis of Cartridges++ compared with standard Cartridges, including the overhead introduced by relevance routing?
6 comments
Answer to reviewer fNX3 [2/N]
In realistic applications, queries may depend only partially on the document, combine document-specific information with general knowledge, or be closely related to the document's topic without actually requiring its contents. Such ambiguous cases seem both more challenging and more representative of realistic use. How well do the proposed methods generalize to these?
As the reviewer points out, our main evaluation tests whether a fixed, reusable KV cache harms performance on questions that do not need its document, as this is a realistic scenario when serving a model. To test the applicability of our mitigations to other mixed settings and questions on related topics, we have now added results on 200 LongHealth questions combined with IFEval instructions, requiring both document information and general skills; as well as health-related questions from the MMLU-pro dataset.
For the first setting, we measure answer correctness, instruction compliance using IFEval’s official strict checker, and joint success, which requires both. In this setting, data mixing improves instruction following and joint success over original Cartridges, as seen below. Routing improves instruction following but leaves joint success unchanged. We believe that future work with more expressive routers could better recognize mixed queries, but an on/off decision alone cannot prevent interference when the cache needs to remain active, as in mixed queries.
| Method | Answer accuracy (%) | Instruction following (%) | Joint success (%) |
|---|---|---|---|
| Full context | 70.6 ± 2.6 | 80.1 ± 2.6 | 57.9 ± 2.9 |
| No context | 36.5 ± 2.8 | 88.1 ± 1.9 | 33.6 ± 2.7 |
| Cartridges | 43.8 ± 2.5 | 77.2 ± 2.5 | 35.7 ± 2.5 |
| Cᴰ++ | 46.4 ± 2.9 | 84.3 ± 2.1 | 40.9 ± 2.8 |
| Cᴿ++ | 42.0 ± 2.6 | 80.2 ± 2.4 | 35.6 ± 2.5 |
In the second experiment, we evaluated 100 MMLU-Pro health questions with LongHealth-10, to study the effect of general questions on a semantically similar topic. Accuracy is 64.0% without context, 47.0% with Cartridges, 66.0% with Cᴰ++, and 56.0% with Cᴿ++. These results show that regularization also helps in this related-topic setting, bringing Cᴰ++ accuracy close to the no-context level. Routing still recovers 9 pp. despite activating the unnecessary memory on 42% of these questions.
Our results suggest that data mixing (Cᴰ++) is better suited to these mixed settings, keeping document information available while helping preserve instruction following and general-knowledge performance on related topics. In contrast, Cᴿ++ is better suited to queries that do not need the document or when the cartridge has already been trained in advance; when correctly rejected, the cache is bypassed entirely, removing its influence rather than merely regularizing it. We have now updated the manuscript to include this discussion in Appendix J.1 (Tables 4–5), with a summary in Section 7.
The methodological contribution is relatively limited. Both proposed methods are minor modifications to the original framework, relying on familiar techniques. While the empirical findings are interesting, the proposed solutions themselves seem somewhat incremental, raising questions about whether they constitute a sufficiently substantial methodological contribution for a full conference paper.
Our central contribution is identifying the breadth and severity of the capability-preservation failure in learned KV compression, and demonstrating how to mitigate it. Strong document fidelity can coexist with substantial losses in general knowledge, instruction following, and response quality, even with frozen model weights. To our knowledge, we are the first to systematically characterize and introduce mitigations to this trade-off across these capabilities.
To highlight the relevance of our contribution, we point the reviewer to recent papers published at major conferences on eviction-based KV compression that similarly establish the importance of uncovering compression failures missed by conventional evaluations and propose practical fixes. For example, [1] address context-native instruction-following failures, while [2] address reasoning degradation when the CoT is compressed. We advance this effort by characterizing and mitigating off-context interference in learned KV compression. This is particularly important as learned KV caches have shown strong document performance even at extreme compression ratios, making it essential to understand and address the potential side effects of these increasingly promising methods. This is very relevant in deployment, where a fixed cache serves requests with different information needs: document fidelity alone does not establish reliable behavior, and accessible mitigations make this finding actionable. The paper therefore contributes both a deeper understanding of the failure and practical ways to mitigate it. We have clarified this in Section 1 and Appendix A.
Answer to reviewer fNX3 [N/N]
Could the authors provide an end-to-end computational efficiency analysis of Cartridges++ compared with standard Cartridges, including the overhead introduced by relevance routing?
Cᴰ++ uses the same number of optimizer updates as standard Cartridges and adds no inference operations, while Cᴿ++ adds a relevance decision before generation. Below, we break down the inference cost for accepted and rejected queries and describe the one-time offline preparation required by each variant.
Let q = prompt length, c = Cartridge length, o = output length, B = bank size, and d = embedding dimension.
For a fixed model:
-
Prompt prefill:
P(q, c) = O(q) + O(q × c + q²)O(q): projections and MLPs.O(q × c + q²): attention.
-
Answer decoding:
D(o, q + c) = O(o) + O(o × (q + c) + o²)
| Method | Order of operations | Total computation |
|---|---|---|
| Cartridges | Prefill with cache → decode | P(q, c) + D(o, q + c) |
| Cᴰ++ | Prefill with cache → decode | Same as Cartridges |
| Cᴿ++, rejected | Clean prefill / embedding → kNN → clean decode | P(q, 0) + O(B × d) + D(o, q) |
| Cᴿ++, accepted | Clean prefill / embedding → kNN → prefill with cache → decode | P(q, 0) + O(B × d) + P(q, c) + D(o, q + c) |
This provides an analytical comparison of computational costs; actual latency overhead will depend on query length, generation length, and hardware. We show that the additional cost for an accepted request is P(q, 0) + O(B × d), and our new results (in Appendix J.4 of the updated manuscript) show that B can be reduced to just 100 embeddings with little performance loss.
We have omitted model-dependent constants in this calculation and assume the Cartridge is already cached, so document prefill is not repeated.
To complement this analysis, we also conducted an end-to-end latency benchmark on a single NVIDIA B200, using Qwen3-4B-Instruct-2507 and LongHealth-10 at 5% cache retention. We evaluated 20 questions with five repetitions each (100 runs per path), using batch size 1, prompts of median length 175 tokens, and 256-token responses with greedy decoding. We report median time to first token and total response time.
| Method | Time to first token (ms) | Total response time (s) |
|---|---|---|
| Cartridges | 36 | 6.01 |
| Cᴰ++ | 36 | 6.02 |
| Cᴿ++ (accepted) | 65 | 6.06 |
These results show that Cᴰ++ has virtually identical inference latency to standard Cartridges, while Cᴿ++ adds less than 1% to total response time in this setting. We report the result for accepted queries, where routing adds 29 ms to time to first token, which corresponds to doing a second pre-fill. Rejected queries instead reuse the clean prefill for decoding, so no second prefill is required. Overall, the measured results show that routing introduces a small end-to-end latency overhead for the evaluated workload. We note that the implementation was not optimized for routing efficiency, especially kNN search, and further latency reductions are possible.
Prior to inference, offline, Cᴰ++ requires scoring the mixed Dolci responses with the no-context teacher, with comparable per-token teacher-scoring cost to standard self-study in vanilla cartridges, and Cᴿ++ reuses the self-study questions and requires computing bank and calibration embeddings. These preparation costs are incurred once per document, amortized across subsequent requests, and are minor when compared to training the cartridge itself. We thank the reviewer for bringing this up, and we have now updated the manuscript in Appendix J.5 to extend the computational efficiency discussion.
Answer to reviewer fNX3 [1/N]
Thank you for recognizing the value of our experiments and mitigations. Your feedback has helped us improve the manuscript. We address your concerns below and would be happy to answer any follow-up questions!
The original Cartridges paper itself identifies generality across diverse future queries as a key requirement for replicating in-context learning, which is the goal of having a compressed KV cache. Our work tests whether compressed KV caches actually meet this requirement, and shows why such safeguards are needed. We show that optimizing only a document’s keys and values can preserve document fidelity while substantially degrading general knowledge, instruction following, and response quality. This is not an obvious consequence of compression when all model weights remain frozen, and the limited off-context evaluation in prior work did not establish the breadth or severity of this failure. As pointed out by the reviewer, memory selection is precisely what our Cᴿ++ router provides.
Additionally, the proposed Cᴰ++ addresses deployments where the document memory stays loaded across requests. For example, an assistant served with a fixed compressed documentation or personal-memory prefix must still handle requests that do not need that information. If the serving system does not support per-query memory selection, simply disabling the Cartridge may not be an available solution. Cᴰ++ instead regularizes the cache during training, helping preserve general capabilities without requiring a router or memory switching at inference. As shown later in this reply, **Cᴰ++ is also particularly useful for improving the performance on mixed queries **, where removing the memory would discard information needed to answer. These results demonstrate why regularizing an active memory matters alongside deciding when to activate it. We have clarified this in Section 4 and Appendix J.1.