acceptodds
Under review as a conference paper at ICLR 2027

Cartridges++: KV Cache Compression without Off-Context Derailment

Abstract

Serving long documents to a Large Language Model (LLM) repeatedly is expensive: computations grow with context length, and the memory footprint of the key-value (KV) cache balloons. *Compressed* KV (CKV) representations aim to mimic the cache of a document and are typically computed once and for all, ahead of inference time. Methods to obtain CKVs range from drop mechanisms that reduce their number of columns, to learned approaches. Among the latter, *Cartridges* have emerged as a leading compression method, learning compact KV representations through distillation on relevant Q/A pairs. While existing evaluations focus primarily on whether Cartridges and other CKVs yield approximately similar responses to document-related, *on-context* queries, we investigate the crucial deployment question of whether they can handle *off-context* queries, something the native KV representation is particularly good at, thanks to the mechanics of attention. We observe a fundamental trade-off: while Cartridges perform better for on-context queries, heuristic-variants preserve better the original LLM's ability to operate off-context. We measure this through their capability to avoid context contamination in their response, retain general knowledge, and follow instructions. We propose *Cartridges++*, simple modifications to cartridges that retain off-context abilities at small or negligible cost. The *router* variant decides at inference time whether the query should use the learned long-context memory, while the *data-mixing* variant allocates a small fraction of training Q/As to queries outside the reference long document. Our study shows that assessing CKVs on document utility alone can mask substantial degradation in broader model capabilities, yet those issues can be fixed with benign changes to CKV inference or training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, Long-context LLMs, Context interference, Capability preservation, Learned memories

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fNX3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates how document-specific KV-cache compression methods affect the general capabilities of LLMs. It finds that learned compression methods such as Cartridges, despite generally achieving better performance on document-related question answering, can significantly interfere with other model capabilities, including instruction following and answering unrelated questions. In contrast, simpler eviction-based methods tend to preserve these capabilities better, although they generally perform worse on document-related tasks. The authors attribute this behavior to the training objective of Cartridges, which optimizes the compressed KV representations exclusively on question-answer pairs related to the document, without explicitly preserving the model's behavior on unrelated queries. To address this issue, they introduce Cartridges++, which proposes two mitigation strategies. The first mixes document-related and unrelated question-answer pairs during KV optimization, using the original LLM without context as the teacher for unrelated queries. The second introduces a nearest-neighbor-based relevance router that determines whether a given query requires access to the compressed KV cache. Through extensive experiments across multiple LLMs, compression methods, and benchmarks, the authors demonstrate that optimization-based compression methods can substantially degrade general model capabilities despite their superior document-specific performance. They further show that both proposed mitigation strategies significantly reduce this degradation while largely preserving document-related performance.

Critical strengths
  1. The proposed mitigation strategies are simple, intuitive, and easy to incorporate into the original Cartridges framework.

  2. The paper presents extensive experiments that convincingly demonstrate the existence of the identified problem and the effectiveness of the proposed mitigation strategies.

Critical weaknesses
  1. The practical significance of the problem is somewhat unclear. If a query does not require information from the compressed document, a straightforward solution is simply not to use the Cartridge. Therefore, it is unclear to what extent the identified interference constitutes a fundamental limitation of KV-cache compression rather than a deployment issue that can be avoided through appropriate memory selection.

  2. The evaluation primarily considers two clearly separated types of queries: those related to the document and those completely unrelated to it. In realistic applications, queries may depend only partially on the document, combine document-specific information with general knowledge, or be closely related to the document's topic without actually requiring its contents. Such ambiguous cases seem both more challenging and more representative of realistic use. Evaluating the proposed methods in these settings would provide stronger evidence of their practical effectiveness, particularly for the relevance-based routing approach.

  3. The methodological contribution is relatively limited. Both proposed methods are minor modifications to the original Cartridges framework, relying on familiar techniques such as mixing general-purpose training data and relevance-based routing. While the empirical findings are interesting, the proposed solutions themselves seem somewhat incremental, raising questions about whether they constitute a sufficiently substantial methodological contribution for a full conference paper.

Questions for authors
  1. How well do the proposed methods generalize to queries that are only partially related to the compressed document, combine document-specific and general knowledge, or are semantically similar to document-related queries without requiring the document?

  2. Could the authors provide an end-to-end computational efficiency analysis of Cartridges++ compared with standard Cartridges, including the overhead introduced by relevance routing?

6 comments
AuthorsOpenReview ↗

Answer to reviewer fNX3 [1/N]

Strengths

Thank you for recognizing the value of our experiments and mitigations. Your feedback has helped us improve the manuscript. We address your concerns below and would be happy to answer any follow-up questions!

The practical significance of the problem is somewhat unclear. If a query does not require information from the compressed document, a straightforward solution is simply not to use the Cartridge. Therefore, it is unclear to what extent the identified interference constitutes a fundamental limitation of KV-cache compression rather than a deployment issue that can be avoided through appropriate memory selection.

The original Cartridges paper itself identifies generality across diverse future queries as a key requirement for replicating in-context learning, which is the goal of having a compressed KV cache. Our work tests whether compressed KV caches actually meet this requirement, and shows why such safeguards are needed. We show that optimizing only a document’s keys and values can preserve document fidelity while substantially degrading general knowledge, instruction following, and response quality. This is not an obvious consequence of compression when all model weights remain frozen, and the limited off-context evaluation in prior work did not establish the breadth or severity of this failure. As pointed out by the reviewer, memory selection is precisely what our Cᴿ++ router provides.

Additionally, the proposed Cᴰ++ addresses deployments where the document memory stays loaded across requests. For example, an assistant served with a fixed compressed documentation or personal-memory prefix must still handle requests that do not need that information. If the serving system does not support per-query memory selection, simply disabling the Cartridge may not be an available solution. Cᴰ++ instead regularizes the cache during training, helping preserve general capabilities without requiring a router or memory switching at inference. As shown later in this reply, **Cᴰ++ is also particularly useful for improving the performance on mixed queries **, where removing the memory would discard information needed to answer. These results demonstrate why regularizing an active memory matters alongside deciding when to activate it. We have clarified this in Section 4 and Appendix J.1.

AuthorsOpenReview ↗

Answer to reviewer fNX3 [2/N]

In realistic applications, queries may depend only partially on the document, combine document-specific information with general knowledge, or be closely related to the document's topic without actually requiring its contents. Such ambiguous cases seem both more challenging and more representative of realistic use. How well do the proposed methods generalize to these?

As the reviewer points out, our main evaluation tests whether a fixed, reusable KV cache harms performance on questions that do not need its document, as this is a realistic scenario when serving a model. To test the applicability of our mitigations to other mixed settings and questions on related topics, we have now added results on 200 LongHealth questions combined with IFEval instructions, requiring both document information and general skills; as well as health-related questions from the MMLU-pro dataset.

For the first setting, we measure answer correctness, instruction compliance using IFEval’s official strict checker, and joint success, which requires both. In this setting, data mixing improves instruction following and joint success over original Cartridges, as seen below. Routing improves instruction following but leaves joint success unchanged. We believe that future work with more expressive routers could better recognize mixed queries, but an on/off decision alone cannot prevent interference when the cache needs to remain active, as in mixed queries.

MethodAnswer accuracy (%)Instruction following (%)Joint success (%)
Full context70.6 ± 2.680.1 ± 2.657.9 ± 2.9
No context36.5 ± 2.888.1 ± 1.933.6 ± 2.7
Cartridges43.8 ± 2.577.2 ± 2.535.7 ± 2.5
Cᴰ++46.4 ± 2.984.3 ± 2.140.9 ± 2.8
Cᴿ++42.0 ± 2.680.2 ± 2.435.6 ± 2.5

In the second experiment, we evaluated 100 MMLU-Pro health questions with LongHealth-10, to study the effect of general questions on a semantically similar topic. Accuracy is 64.0% without context, 47.0% with Cartridges, 66.0% with Cᴰ++, and 56.0% with Cᴿ++. These results show that regularization also helps in this related-topic setting, bringing Cᴰ++ accuracy close to the no-context level. Routing still recovers 9 pp. despite activating the unnecessary memory on 42% of these questions.

Our results suggest that data mixing (Cᴰ++) is better suited to these mixed settings, keeping document information available while helping preserve instruction following and general-knowledge performance on related topics. In contrast, Cᴿ++ is better suited to queries that do not need the document or when the cartridge has already been trained in advance; when correctly rejected, the cache is bypassed entirely, removing its influence rather than merely regularizing it. We have now updated the manuscript to include this discussion in Appendix J.1 (Tables 4–5), with a summary in Section 7.

The methodological contribution is relatively limited. Both proposed methods are minor modifications to the original framework, relying on familiar techniques. While the empirical findings are interesting, the proposed solutions themselves seem somewhat incremental, raising questions about whether they constitute a sufficiently substantial methodological contribution for a full conference paper.

Our central contribution is identifying the breadth and severity of the capability-preservation failure in learned KV compression, and demonstrating how to mitigate it. Strong document fidelity can coexist with substantial losses in general knowledge, instruction following, and response quality, even with frozen model weights. To our knowledge, we are the first to systematically characterize and introduce mitigations to this trade-off across these capabilities.

To highlight the relevance of our contribution, we point the reviewer to recent papers published at major conferences on eviction-based KV compression that similarly establish the importance of uncovering compression failures missed by conventional evaluations and propose practical fixes. For example, [1] address context-native instruction-following failures, while [2] address reasoning degradation when the CoT is compressed. We advance this effort by characterizing and mitigating off-context interference in learned KV compression. This is particularly important as learned KV caches have shown strong document performance even at extreme compression ratios, making it essential to understand and address the potential side effects of these increasingly promising methods. This is very relevant in deployment, where a fixed cache serves requests with different information needs: document fidelity alone does not establish reliable behavior, and accessible mitigations make this finding actionable. The paper therefore contributes both a deeper understanding of the failure and practical ways to mitigate it. We have clarified this in Section 1 and Appendix A.

AuthorsOpenReview ↗

Answer to reviewer fNX3 [N/N]

Could the authors provide an end-to-end computational efficiency analysis of Cartridges++ compared with standard Cartridges, including the overhead introduced by relevance routing?

Cᴰ++ uses the same number of optimizer updates as standard Cartridges and adds no inference operations, while Cᴿ++ adds a relevance decision before generation. Below, we break down the inference cost for accepted and rejected queries and describe the one-time offline preparation required by each variant.

Let q = prompt length, c = Cartridge length, o = output length, B = bank size, and d = embedding dimension.

For a fixed model:

  • Prompt prefill: P(q, c) = O(q) + O(q × c + q²)

    • O(q): projections and MLPs.
    • O(q × c + q²): attention.
  • Answer decoding: D(o, q + c) = O(o) + O(o × (q + c) + o²)

MethodOrder of operationsTotal computation
CartridgesPrefill with cache → decodeP(q, c) + D(o, q + c)
Cᴰ++Prefill with cache → decodeSame as Cartridges
Cᴿ++, rejectedClean prefill / embedding → kNN → clean decodeP(q, 0) + O(B × d) + D(o, q)
Cᴿ++, acceptedClean prefill / embedding → kNN → prefill with cache → decodeP(q, 0) + O(B × d) + P(q, c) + D(o, q + c)

This provides an analytical comparison of computational costs; actual latency overhead will depend on query length, generation length, and hardware. We show that the additional cost for an accepted request is P(q, 0) + O(B × d), and our new results (in Appendix J.4 of the updated manuscript) show that B can be reduced to just 100 embeddings with little performance loss.

We have omitted model-dependent constants in this calculation and assume the Cartridge is already cached, so document prefill is not repeated.

To complement this analysis, we also conducted an end-to-end latency benchmark on a single NVIDIA B200, using Qwen3-4B-Instruct-2507 and LongHealth-10 at 5% cache retention. We evaluated 20 questions with five repetitions each (100 runs per path), using batch size 1, prompts of median length 175 tokens, and 256-token responses with greedy decoding. We report median time to first token and total response time.

MethodTime to first token (ms)Total response time (s)
Cartridges366.01
Cᴰ++366.02
Cᴿ++ (accepted)656.06

These results show that Cᴰ++ has virtually identical inference latency to standard Cartridges, while Cᴿ++ adds less than 1% to total response time in this setting. We report the result for accepted queries, where routing adds 29 ms to time to first token, which corresponds to doing a second pre-fill. Rejected queries instead reuse the clean prefill for decoding, so no second prefill is required. Overall, the measured results show that routing introduces a small end-to-end latency overhead for the evaluated workload. We note that the implementation was not optimized for routing efficiency, especially kNN search, and further latency reductions are possible.

Prior to inference, offline, Cᴰ++ requires scoring the mixed Dolci responses with the no-context teacher, with comparable per-token teacher-scoring cost to standard self-study in vanilla cartridges, and Cᴿ++ reuses the self-study questions and requires computing bank and calibration embeddings. These preparation costs are incurred once per document, amortized across subsequent requests, and are minor when compared to training the cartridge itself. We thank the reviewer for bringing this up, and we have now updated the manuscript in Appendix J.5 to extend the computational efficiency discussion.

Reviewer_jiRzOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds on Cartridges [1]. The original Cartridges paper compresses a document/long context into a smaller (swappable) KV cache (termed a “cartridge”). A cartridge is trained by distribution matching/context distillation between an LLM conditioned on the “cartridge” and the original uncompressed document. Such a distillation procedure requires a task distribution over which to conduct distillation, for which [1] uses synthetic Q&A generated about the document.

This paper argues that when a cartridge is being used, and the user asks an out-of-distribution question (i.e something unrelated to the document the cartridge was derived from, “off-context”), performance degrades, and the performance degradation is more than it would be with the full uncompressed document in context. They propose a modification of [1] with the following goals: matching full-context behaviour on in-distribution queries (about the document) and matching no-context behaviour on out-of-distribution queries.

To do this, they propose two methods. The first (C^D++) mixes a small fraction of general instruction data into the self-study training mix. On these tasks the teacher is a no-context base model. The second (C^R++) leaves the cartridge unchanged and adds a per-document kNN router over the model's hidden states. The router compares the incoming query against a bank of self-study questions and only loads the cartridge if the query appears on-document.

They compare Cartridges against eviction-based compression and a KV budget-matched summary. They find that cartridges have the best context fidelity but the worst off-context degradation.

They validate their modifcations on QASPER, LongHealth, MTOB, and QuALITY, using Qwen3-4B-Instruct and Gemma-4-E4B, at compression ratios from 0.1% to 20%. OOD behaviour is measured with MMLU, IFEval, and an LLM-judged "context interference" ( how often the model uses the compressed document to answer an unrelated question). Cartridges show interference of ~10–25%, versus ~3% for eviction methods. Both fixes mostly recover OOD performance with minimal loss in fidelity. Ablations show that the OOD degradation is not caused by position and is not fixed by longer training. They also show that prompting the model to ignore irrelevant context does not help, and that the router results hold across judges and encoders.

[1] Sabri Eyuboglu, Ryan Ehrlich, Simran Arora, Neel Guha, Dylan Zinsley, Emily Liu, Atri Rudra, James Zou, Azalia Mirhoseini, and Christopher R´ e. Cartridges: Lightweight and general-purpose long context representations via self-study. In International Conference on Learning Representations, 2026

Critical strengths

The problem is clearly identified and measured. The paper clearly demonstrates how cartridges harm OOD (off-context) performance, and goes beyond what [1] measured. The paper looks at general knowledge, instruction following, and context interference. The paper’s experiments cover 2 model families, 4 datasets, various compression ratios, and several eviction baselines. The ablations rule out obvious confounds like position, training length. The data-mixing proposed, though simple, is effective. Since I believe the paper's value rests mostly on its empirical findings, this experimental breadth is its main strength.

Critical weaknesses

Limited novelty in both the problem and proposed solution. The problem being addressed is that a cartridge is trained only on in-distribution queries, so behaviour on OOD tasks is unconstrained and degrades. It is well-known that training on a narrow task distribution can induce forgetting and interference. The two fixes the paper proposes are standard remedies for that: replay/data mixing (using the no-context model as teacher) and gating the adapter with an OOD detector. The paper neither builds the OOD data or the router in a new way. They are the most straightforward solutions to the problem as stated. The problem identification is also not new. The original Cartridges paper [1] evaluates OOD queries and reports that cartridges preserve accuracy OOD as well (54.7 -> 54.3). This work mentions this in passing but does not explain why their findings differ from the original work, nor do they attempt to replicate the original finding. Neither the unsurprising failure mode, nor the simplicity of the proposed solution is a problem in and of itself. However, since the methods deployed are straightforward, the paper's value rests on empirical findings. Without engaging with the prior (conflicting) result from [1], it is unclear how much of that finding is new or how robust it is.

Additionally, the router variant has practical costs that limit its usability. The method the paper proposes requires a per-document router and a bank of in-distribution questions. The paper proposes extracting a pooled (layer 18) hidden state representations from the model’s encoding of the query, and using kNN over the pooled representations of in-distribution questions with a conformally calibrated threshold for making routing decisions. Accepted queries must be re-prefilled, and they do not provide sufficient consideration to how this solution would work in multi-turn conversations, though there is potentially quite a bit to discuss there (a cursory discussion in Appendix G relies on oracle relevance labels, which is unconvincing). Furthermore, the OOD Dolci queries they use are likely far from document questions. Calibration on that data provides little insight into how the method fares in harder cases (such as a general question on a related topic or a mixed query that partly needs the document). In conjunction with the concerns from the previous paragraph, the limited practical applicability of the router limits the value of the paper’s empirical findings.

Questions for authors

Why do these findings differ significantly from prior work

8 comments
AuthorsOpenReview ↗

Answer to reviewer jiRz [1/N]

Strengths

Thank you for recognizing the breadth of our experiments and the effectiveness of our solutions. We appreciate your constructive feedback, which helped us improve the manuscript. We address each of your concerns below and would be happy to clarify anything further or answer any follow-up questions.

Limited novelty in both the problem and proposed solution. The problem being addressed is that a cartridge is trained only on in-distribution queries, so behaviour on OOD tasks is unconstrained and degrades. It is well-known that training on a narrow task distribution can induce forgetting and interference.

We disagree that this degradation follows directly from known forgetting in model fine-tuning. Cartridges leave every base-model weight unchanged and optimize only the keys and values representing the prefix document. It is therefore not obvious that compressing the context should impair capabilities already present in the frozen model for new incoming questions. An unconstrained off-context objective identifies a potential risk; our experiments establish that this risk materializes across different settings and can cause larger degradation than retaining the full document or using eviction.

Our contribution is to systematically characterize this trade-off between document fidelity and capability preservation across general knowledge, instruction following, and response contamination, and show how to mitigate it while retaining document utility. The novelty lies in establishing and addressing this previously uncharacterized failure of learned KV compression; the simplicity of the interventions makes the finding actionable. We have clarified the empirical contribution in Section 1 and revised Section 3.3 to distinguish interference from optimized KV memories on a frozen model from forgetting caused by model fine-tuning.

The two fixes the paper proposes are standard remedies for that: replay/data mixing (using the no-context model as teacher) and gating the adapter with an OOD detector. The paper neither builds the OOD data or the router in a new way. They are the most straightforward solutions to the problem as stated.

We introduce two mitigations specifically for off-context interference in learned KV compression. The contribution is adapting established principles to this setting and demonstrating a solution to preserve general capabilities while retaining document utility.

To highlight the relevance of our contribution, we point to recent work on eviction-based KV compression that similarly identifies failures missed by standard evaluations and proposes practical fixes. For example, [1] address context-native instruction-following failures, while [2] address reasoning degradation when the CoT is compressed. We advance this effort by characterizing and mitigating off-context interference in learned KV compression. Given the strong document performance of learned KV caches even at extreme compression ratios, understanding and addressing their potential side effects is particularly important. This is very relevant in deployment where a fixed cache serves requests with different information needs, accessible mitigations make this finding actionable. We have sharpened the contribution statement in Section 1 and the comparison with existing locality and selective-activation methods in Appendix A.

AuthorsOpenReview ↗

Answer to reviewer jiRz [2/N]

The problem identification is also not new. The original Cartridges paper [1] evaluates OOD queries and reports that cartridges preserve accuracy OOD as well (54.7 -> 54.3). This work mentions this in passing but does not explain why their findings differ from the original work, nor do they attempt to replicate the original finding.

Thanks for pointing this out! We agree that this deserves a deeper discussion and appreciate the opportunity to clarify our work. The mentioned result from the original paper measures stability across Cartridge sizes, not preservation relative to “no context”. Appendix A.3 explicitly acknowledges degradation of up to 4 pp. (Figure 6). The original paper does not specify its full protocol, but its released code points to an example of five-shot, and forced direct-answer evaluation; while we use zero-shot prompting with a rationale before the answer. Additionally, our “MMLUs” evals combine TinyMMLU and the harder MMLU-Pro, whereas the original paper evaluates an MMLU subset. Finally, the model differs, as the original experiments use Llama-3.2-3B-Instruct, while ours use Qwen3-4B-Instruct-2507 and the Gemma family.

Although we cannot confirm that this evaluator produced their published results, we tested both protocols while keeping our Qwen model and QASPER-16 Cartridges fixed. We used their 512 MMLU questions and prompts for direct answering, and our 1,400 MMLU-Pro questions as a testbed:

BenchmarkProtocolNo-context accuracy (%)Full-context change (pp)Cartridge 5% change (pp)Cartridge 10% change (pp)
MMLU, 512 questionsZero-shot, rationale (ours)77.3 ± 1.9−3.5 ± 1.6−7.4 ± 1.9−11.1 ± 2.0
MMLU, 512 questionsFive-shot, direct answer (original)71.7 ± 2.0−1.0 ± 1.1−3.3 ± 1.5−4.9 ± 1.6
MMLU-Pro, 1,400 questionsZero-shot, rationale (ours)64.2 ± 1.3−1.1 ± 1.0−8.1 ± 1.3−7.0 ± 1.2
MMLU-Pro, 1,400 questionsFive-shot, direct answer (original)48.1 ± 1.3+0.4 ± 0.7−3.1 ± 0.9−3.4 ± 1.0

The direct-answer protocol produces smaller drops but also lower baseline accuracy. On MMLU-Pro, even the degraded Cartridges scores with our protocol exceed the direct-answer no-context baseline. Preserving performance under this higher-accuracy protocol is therefore practically relevant to test capability preservation, particularly for MMLU-Pro, which was designed to test more demanding reasoning abilities.

Concurrent, independent work [3] also reports a substantially larger MMLU drop: Cartridges reduce Gemma-3-12B’s MMLU accuracy from 72.6% to 56.9%, 15.7 pp., consistent with our findings. This corroborates that the failure extends independently beyond our work.

Overall, differences in evaluation protocol help explain the discrepancy. We do observe larger drops under our protocol, but degradation persists under both. Our contribution goes further: we study effects on general knowledge, instruction following, and response contamination, and develop mitigations to preserve these capabilities, which are practical requirements when serving models. We have added the controlled evaluation-protocol comparison and expanded discussion of the original results in Appendix J.3 (Table 7), with a pointer from Section 2.

Neither the unsurprising failure mode, nor the simplicity of the proposed solution is a problem in and of itself. However, since the methods deployed are straightforward, the paper's value rests on empirical findings. Without engaging with the prior (conflicting) result from [1], it is unclear how much of that finding is new or how robust it is.

Thanks for the pointer, we believe the expanded comparisons above help explain the differences with the original findings and show that the problem extends beyond MMLU. Learned KV compression can preserve document fidelity while impairing general capabilities, which makes our validated mitigations important for reliable deployment. Appendix J.3 now presents the protocol comparison, explains the differences from prior findings, and discusses independent evidence of degradation beyond our model and evaluation setup.

AuthorsOpenReview ↗

Answer to reviewer jiRz [3/N]

Additionally, the router variant has practical costs that limit its usability. The method the paper proposes requires a per-document router and a bank of in-distribution questions. The paper proposes extracting a pooled (layer 18) hidden state representations from the model’s encoding of the query, and using kNN over the pooled representations of in-distribution questions with a conformally calibrated threshold for making routing decisions.

We would like to point out that the per-document routing matches the per-document construction of Cartridges themselves. Additionally, the kNN router requires no classifier-weight training: it reuses the existing model’s representations and questions already generated during self-study. Bank embeddings are computed once offline, and threshold calibration selects a quantile of held-out scores.

To provide an intuition of the added cost of the router, we have done a study of the memory requirements. Each document stores approximately 1,700 embeddings of dimension 2,560 and one threshold, requiring 17.4 MB in fp32. This is about 0.22% of the model’s 8.0 GB of bf16 weights and 1.0–1.1% of the bf16 Cartridge storage at 10% compression ratio: 1.68 GB for LongHealth-10 and 1.52 GB for QASPER-16. Our evaluated logistic-regression alternative in Appendix C.2 further reduces storage to a weight vector, bias, and threshold, without retaining the embedding bank.

Additionally, the full bank is also not a fixed storage requirement. We have conducted a bank-size ablation on LongHealth-10 with the Qwen model and 5% compression, and show that reducing the bank to 800 embeddings saves 53% of storage, with only 0.3 pp. lower MMLUs accuracy. Even further, using 100 embeddings saves 94%, with approximately 1 pp. lower MMLUs accuracy, while on-context accuracy remains broadly stable (see Table below). The router already has a small memory footprint, and these results show it can be reduced further with little performance loss. Regarding the compute cost, the hidden-state router adds one extra query prefill only for accepted requests; rejected requests continue directly from the initial clean prefill. We have added the storage analysis and bank-size ablation in Appendix J.4 (Table 8), and the analytical inference-cost comparison in Appendix J.5 (Table 9). Section 6 also summarizes the computational implications.

Bank sizeStorage (MB)On-context accuracy (%)MMLUs accuracy (%)
1,712 (full)17.5350.169.5
8008.1950.069.2
1001.0250.268.6

Accepted queries must be re-prefilled, and they do not provide sufficient consideration to how this solution would work in multi-turn conversations, though there is potentially quite a bit to discuss there (a cursory discussion in Appendix G relies on oracle relevance labels, which is unconvincing).

Thanks for the suggestion! We have replaced the oracle-routing results in the manuscript’s multi-turn figure with learned-router results. Scores only differ by at most 1.0 percentage point on LongHealth and 1.9 points on QASPER across the alternating relevance schedules. The table below directly compares oracle and learned routing, with Cᴰ++ included for reference.

DatasetScheduleTurnCᴰ++Cᴿ++ oracleCᴿ++ learned
LongHealth-10R/O/R152.0 ± 3.553.5 ± 3.553.5 ± 3.5
280.5 ± 2.080.5 ± 2.079.5 ± 2.0
353.0 ± 3.551.2 ± 3.550.8 ± 3.5
O/R/O181.8 ± 1.981.5 ± 1.980.5 ± 2.0
245.8 ± 3.552.0 ± 3.552.0 ± 3.5
380.8 ± 2.080.2 ± 2.081.2 ± 1.9
QASPER-16R/O/R135.9 ± 3.229.3 ± 3.029.3 ± 3.0
282.7 ± 3.082.7 ± 3.080.8 ± 3.1
330.6 ± 3.129.0 ± 2.929.2 ± 2.9
O/R/O184.6 ± 2.982.1 ± 3.081.4 ± 3.1
230.5 ± 3.129.7 ± 3.129.5 ± 3.1
380.8 ± 3.180.8 ± 3.179.5 ± 3.2

Scores (%) ± SE. R: document-relevant turn; O: off-context turn. Relevant turns use LongHealth accuracy or QASPER token F1; off-context turns pool TinyMMLU and IFEval accuracy.

Our multi-turn implementation of the router does re-prefill the conversation when the router decision changes between turns. This processes existing tokens in a forward pass; it does not regenerate previous answers. Re-prefilling therefore has a cost that grows with conversation length; Cᴰ++ avoids this additional cost and is preferable when that overhead is a concern. We would like to highlight that Appendix G is a proof of concept extending beyond the original Cartridges paper’s single-turn evaluations, rather than the main focus of our study. It tests whether Cartridges remain effective across multiple turns and shows that Cᴰ++ and Cᴿ++ retain their benefits. We have expanded Appendix G to explain conversation reconstruction, its limitations, and length-dependent cost.

Reviewer_WKRcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how compressed KV cache reusing hurt the model performance. It frames two requirements: 1) context fidelity on related queries and capability preservation on unrelated queries. The experiments use multiple long-context benchmark and evaluted on different backbones. Carteidges are found best on-context, but hallucition, often citing and inventing document contents. Eviction methods and summaries stay close to no-context behavior and attention matching sits in between. The authors propose CARTRIDGES++ to recover most of the off-context loss at little on-context cost.

Critical strengths
  • [S1] Targeting a real problem, and comprehensive evaluation.

The method design is well-motivated by the experiments, and the experiments are comprehensive across different tasks and different backbones.

  • [S2] Clear presentation.

The paper is clearly written, and the main idea is easy to follow.

Critical weaknesses
  • [W1] Limited novelty.

Both fixes are standard locality tools, and the paper's own related work already applies gating, locality losses or distractor training to learned memories. The degradation follows directly from the one-sided objective the authors formalize, and the attention-based explanation in the introduction is never tested directly. This makes the paper a careful empirical note on one recipe rather than a broadly new contribution.

  • [W2] The off-context evaluation only covers the easy case.

Capability preservation is measured only on prompts far from the document (TinyMMLU, MMLU-Pro, IFEval). Dolci is used for both mixing and router calibration, and it contains Tülu 3 Precise IF data built on IFEval's constraint set. The more realistic cases are not tested with the actual mitigations. Same-domain unanswerable questions are evaluated only for vanilla Cartridges. So the claim that CARTRIDGES++ "cancels" the degradation holds only for the in-distribution data.

Questions for authors
  • [Q1] How do C^D++ and C^R++ perform on the mismatched same-domain questions of Appendix F? What fraction does the router accept, and does mixing restore abstention? If neither helps, the claims in the abstract and Figure 1 should be narrowed.

  • [Q2] The original Cartridges paper reports that MMLU accuracy with the prefix parameterization barely changes as the cache grows and describes out-of-domain performance as robust. Your off-context drops are larger and, in Figure 3, worsen as the cache grows. Is this due to the model family, generative CoT scoring, or recipe details?

  • [Q3] Since Dolci contains IFEval-style constraint data, how much of the IFEval recovery under C^D++ remains if that subset is removed?

  • [Q4] How does CR++ behave in multi-turn conversations with the learned router instead of oracle labels? How does it handle mixed queries that need both the document and general skills?

6 comments
AuthorsOpenReview ↗

Answer to reviewer WKRc [1/N]

[S1] Targeting a real problem, and comprehensive evaluation. [S2] Clear presentation.

Thank you for recognizing the relevance of the problem, our comprehensive evaluation, and the clear presentation. We have answered the questions and weaknesses point-by-point below, and thank you for the constructive feedback that helped improve the manuscript!

[W1] Limited novelty. Both fixes are standard locality tools, and the paper's own related work already applies gating, locality losses or distractor training to learned memories.

We appreciate the reviewer’s perspective, but disagree with the assessment of limited novelty. Our contribution concerns KV compression, rather than the prompt- or parameter-based memories studied in prior work. To our knowledge, we are the first to systematically show that directly optimized document-specific keys and values can preserve document fidelity while amplifying off-context interference, and to develop mitigations targeting this failure.

The cited work addresses different settings: [1] introduces specific factual updates through learned input-prompt vectors; [2] selectively includes textual context for RAG; and [3] gate LoRA parameter memories. We instead study directly optimized key/value tensors inside attention layers used as a fixed document-specific prefix. This cache is built before future questions are known, and we ask whether keeping it loaded harms general capabilities on questions that do not need the document. These related works do not establish this compression–capability trade-off or the effectiveness of our solutions in this setting.

The novelty therefore lies in identifying this KV-compression failure and developing and evaluating ways to mitigate it. We have clarified this in the text in the introduction and Appendix A.

The degradation follows directly from the one-sided objective the authors formalize,

Leaving off-context behavior unconstrained does not directly imply degradation, particularly when all base model weights remain frozen. Our key finding is that optimizing only the KV cache can cause greater degradation than keeping the full KV or eviction, we have clarified this in Section 3.3 in the updated manuscript.

and the attention-based explanation in the introduction is never tested directly.

We understand this comment as referring to the sentence: “The rationale is that the mechanics of attention can ....” Note that here, ‘queries’ means user questions. This sentence is meant to provide motivation, inspired by prior work that shows that question-to-context attention carries a relevance signal; less relevant passages receive lower scores [4,5]. This motivates asking whether learned KV cache compression preserves the model’s ability to handle unrelated questions. However, our focus is behavioral: we measure how learned KV caches affect off-context performance. Thanks for pointing this out, we have revised this introductory passage and added the supporting references to avoid implying that we investigate the underlying mechanism, which is beyond the scope of our study.

[W2] Dolci is used for both mixing and router calibration, and it contains Tülu 3 Precise IF data built on IFEval's constraint set. [Q3] How much of the IFEval recovery under C^D++ remains if that subset is removed?

The Precise IF subset shares instruction-constraint types (not exact samples) with IFEval and makes up 6.36% of the Dolci mix. To check whether Precise IF confounds our results, we removed it from both data mixing and router calibration, and reran LongHealth-10 and QASPER-16 at two compression ratios, using Qwen3-4B-Instruct-2507. IFEval performance is comparable in all results (see Table below), supporting that the mitigation is not driven by this subset. Note that for the main kNN router, Dolci negatives only set the activation threshold; the reference bank consists only of positive, document-specific self-study questions. Each cell reports original Dolci → Dolci without Precise IF. We have updated the manuscript with Appendix J.2, and a clarification in Appendix B.1.

LongHealth-10

MethodMetric5% compression ratio10% compression ratio
Cᴰ++MMLUs accuracy (%) ↑70.4 ± 2.2 → 69.9 ± 2.269.7 ± 2.2 → 71.8 ± 2.0
Cᴰ++IFEval (%) ↑81.1 ± 1.7 → 81.5 ± 1.780.0 ± 1.7 → 81.0 ± 1.7
Cᴿ++MMLUs accuracy (%) ↑69.5 ± 2.4 → 69.5 ± 2.470.7 ± 2.4 → 70.7 ± 2.4
Cᴿ++IFEval (%) ↑83.9 ± 1.6 → 83.9 ± 1.683.9 ± 1.6 → 83.9 ± 1.6

QASPER-16

MethodMetric5% compression ratio10% compression ratio
Cᴰ++MMLUs accuracy (%) ↑70.1 ± 2.2 → 69.9 ± 2.271.1 ± 2.1 → 68.5 ± 2.2
Cᴰ++IFEval (%) ↑78.2 ± 1.8 → 79.3 ± 1.781.1 ± 1.7 → 79.4 ± 1.8
Cᴿ++MMLUs accuracy (%) ↑70.8 ± 2.4 → 71.4 ± 2.369.9 ± 2.4 → 71.0 ± 2.3
Cᴿ++IFEval (%) ↑83.2 ± 1.6 → 83.7 ± 1.683.4 ± 1.6 → 84.1 ± 1.6
AuthorsOpenReview ↗

Answer to reviewer WKRc [2/N]

Capability preservation is measured only on prompts far from the document (TinyMMLU, MMLU-Pro, IFEval). [Q4, mixed query] How does it handle mixed queries that need both the document and general skills?

Our main evaluation tests whether a fixed KV cache harms performance on questions that do not need its document, as this is a realistic scenario when serving a model. We have now added an evaluation of mixed questions requiring both document information and instruction following. We append one instruction taken from IFEval to each of the 200 LongHealth-10 questions. We measure answer correctness, instruction compliance using IFEval’s official checker, and joint success, which requires both in the same response.

Data mixing improves instruction following and joint success. Routing improves instruction following but leaves joint success unchanged. We believe that future work with more expressive routers could better recognize mixed queries, but an on/off decision alone cannot prevent interference when the cache needs to remain active, as in mixed queries. Our results suggest that regularization through data mixing is better suited to this document-plus-instruction setting. We have added the results below and in the manuscript in Appendix J.1, with a summary in Section 7, using Qwen3-4B-Instruct and a 5% compression ratio.

MethodAnswer accuracy (%)Instruction following (%)Joint success (%)
Full context70.6 ± 2.680.1 ± 2.657.9 ± 2.9
No context36.5 ± 2.888.1 ± 1.933.6 ± 2.7
Cartridges43.8 ± 2.577.2 ± 2.535.7 ± 2.5
Cᴰ++46.4 ± 2.984.3 ± 2.140.9 ± 2.8
Cᴿ++42.0 ± 2.680.2 ± 2.435.6 ± 2.5

The more realistic cases are not tested with the actual mitigations. Same-domain unanswerable questions are evaluated only for vanilla Cartridges. [Q1] How do C^D++ and C^R++ perform on the mismatched same-domain questions of Appendix F? What fraction does the router accept, and does mixing restore abstention? If neither helps, the claims in the abstract and Figure 1 should be narrowed.

We have now evaluated both mitigations on the mismatched same-domain questions from Appendix F. This Appendix and the original experiment were designed as a controlled diagnostic of abstention, i.e., the tendency to answer rather than abstain, rather than a model of a “more realistic” setting: we deliberately ask questions about a different, unavailable document and do not expect the loaded memory to answer them. Accuracy is not a measure of document utility here. The original experiment therefore focused on vanilla Cartridges to characterize this additional failure mode of abstention, beyond the setting our mitigations target.

In the new results, mitigations have little effect on abstention. The router accepts 100% of mismatched LongHealth questions and 72% of mismatched QASPER questions, which results in an improvement in QASPER abstention and leaves LongHealth unchanged. The goal of this appendix was to highlight this additional failure mode, rather than suggest that our mitigations solve it. Improving abstention is left for future work and may require targeted training or routing based on evidence sufficiency. We have extended the evaluation to both mitigations (in Figure 29 and the table below). We have also narrowed the abstract and Figure 1 claims to the evaluated capability benchmarks and clarified this scope in the introduction, and conclusion.

Abstention and accuracy on same-domain questions without supporting evidence. LongHealth-10

MethodAbstention (%) ↑Answer accuracy (%)
Full context980
No context900
Cartridges820
Cᴰ++1026
Cᴿ++820

QASPER-16

MethodAbstention (%) ↑Answer accuracy (%)
Full context4612
No context364
Cartridges412
Cᴰ++414
Cᴿ++168
AuthorsOpenReview ↗

Answer to reviewer WKRc [N/N]

[Q2] The original Cartridges paper reports that MMLU accuracy with the prefix parameterization barely changes as the cache grows and describes out-of-domain performance as robust. Your off-context drops are larger and, in Figure 3, worsen as the cache grows. Is this due to the model family, generative CoT scoring, or recipe details?

The original paper describes robustness across Cartridge sizes; this does not directly imply preservation relative to empty context. Appendix A.3 explicitly acknowledges degradation, and Figure 6 shows up to a 4 pp drop with respect to the base model. Moreover, our Figure 3 combines general knowledge and instruction following. The MMLU results shown separately in Figure 18 do not decrease monotonically as the cache grows, consistent with the original paper findings.

Additionally, our “MMLUs” evals combine TinyMMLU and the harder MMLU-Pro, whereas the original paper evaluates an MMLU subset and both the model and evaluation protocol differ: the original experiments use Llama-3.2-3B-Instruct, while ours use Qwen3-4B-Instruct-2507. The original paper does not specify its full protocol, but its released code points to an example of five-shot, direct-answer evaluation; we use zero-shot prompting with a rationale before the answer.

Although we cannot confirm that this evaluator produced their published results, we tested both protocols while keeping our Qwen model and QASPER-16 Cartridges fixed. We used their 512 MMLU questions and prompts for direct answering, and our 1,400 MMLU-Pro questions as a testbed. The results show smaller, but persistent, drops under the released direct-answer protocol:

BenchmarkProtocolNo-context accuracy (%)Full-context change (pp)Cartridge 5% change (pp)Cartridge 10% change (pp)
MMLU, 512 questionsZero-shot, rationale (ours)77.3 ± 1.9−3.5 ± 1.6−7.4 ± 1.9−11.1 ± 2.0
MMLU, 512 questionsFive-shot, direct answer (released-code protocol)71.7 ± 2.0−1.0 ± 1.1−3.3 ± 1.5−4.9 ± 1.6
MMLU-Pro, 1,400 questionsZero-shot, rationale (ours)64.2 ± 1.3−1.1 ± 1.0−8.1 ± 1.3−7.0 ± 1.2
MMLU-Pro, 1,400 questionsFive-shot, direct answer (released-code-style protocol)48.1 ± 1.3+0.4 ± 0.7−3.1 ± 0.9−3.4 ± 1.0

The direct-answer protocol produces smaller drops but also lower baseline accuracy. On MMLU-Pro, even the degraded Cartridges scores with our protocol exceed the direct-answer no-context baseline. Preserving performance under this higher-accuracy protocol is therefore practically relevant to test capability preservation, particularly for MMLU-Pro, which was designed to test more demanding reasoning abilities.

Concurrent, independent work [6] also reports a substantially larger MMLU drop: Cartridges reduce Gemma-3-12B’s MMLU accuracy from 72.6% to 56.9%, 15.7 pp , consistent with our findings. This corroborates that the failure extends independently beyond our work.

Overall, differences in evaluation protocol help explain the discrepancy. We do observe larger drops under our protocol, but degradation persists under both. Our rationale-based protocol achieves higher absolute accuracy, making preservation in this stronger-performing setting relevant for deployment. Our contribution goes further: we study effects on general knowledge, instruction following, and response contamination. We have added a controlled protocol comparison and discussion of the original findings in Appendix J.3 (Table 7).

[Q4] How does CR++ behave in multi-turn conversations with the learned router instead of oracle labels?

Thanks for the suggestion! We have replaced the oracle-routing results in the manuscript’s multi-turn Figure 31, Appendix G, with learned-router results. Scores differ by at most 1.0 pp on LongHealth and 1.9 pp on QASPER across the alternating relevance schedules. The table below directly compares oracle and learned routing, with Cᴰ++ included for reference.

DatasetScheduleTurnCᴰ++Cᴿ++ oracleCᴿ++ learned
LongHealth-10R/O/R152.0 ± 3.553.5 ± 3.553.5 ± 3.5
280.5 ± 2.080.5 ± 2.079.5 ± 2.0
353.0 ± 3.551.2 ± 3.550.8 ± 3.5
O/R/O181.8 ± 1.981.5 ± 1.980.5 ± 2.0
245.8 ± 3.552.0 ± 3.552.0 ± 3.5
380.8 ± 2.080.2 ± 2.081.2 ± 1.9
QASPER-16R/O/R135.9 ± 3.229.3 ± 3.029.3 ± 3.0
282.7 ± 3.082.7 ± 3.080.8 ± 3.1
330.6 ± 3.129.0 ± 2.929.2 ± 2.9
O/R/O184.6 ± 2.982.1 ± 3.081.4 ± 3.1
230.5 ± 3.129.7 ± 3.129.5 ± 3.1
380.8 ± 3.180.8 ± 3.179.5 ± 3.2

Scores (%) ± SE. R: document-relevant turn; O: off-context turn. Relevant turns use LongHealth accuracy or QASPER token F1; off-context turns pool TinyMMLU and IFEval accuracy.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Reponse to all Reviewers

Dear Reviewers,

Thank you all for your constructive feedback, which has helped us improve our work. We have updated the manuscript, with changes highlighted in blue. Here is a brief summary of the most relevant additions following the feedback:

  • Added experiments comparing MMLU evaluation protocols with prior work (WKRc, jiRz): We add new results and show that differences in evaluation protocols, models, and datasets explain the discrepancy with the original Cartridges results. Substantial degradation persists and is relevant.

  • Extended the evaluation to mixed and semantically related queries (WKRc, jiRz, fNX3): New results show that C++ improves capability preservation in these more challenging settings, while also clarifying the complementary strengths of Cᴰ++ and Cᴿ++ and why both mitigations are useful.

  • Extended multi-turn experiments with learned routing (WKRc, jiRz): We replaced oracle routing with the learned router and show that its performance remains close to the oracle results.

  • Added computational cost analysis and router memory ablations (jiRz, fNX3): We clarify the computational overhead of Cᴿ++ and show that the router has a small memory footprint, which can be reduced further with little performance loss.

  • Clarified the novelty and significance of our contribution (WKRc, jiRz, fNX3): We emphasize that our main contribution is systematically characterizing the previously unexplored off-context failures of learned KV compression, which is key when serving models, and showing successful solutions to mitigate them.

Further details can be found in our point-by-point responses. Thank you again for your feedback! We are very happy to answer any further questions or comments.

Dear reviewers,

A reminder that the discussion period will end on November 18 at 11:59pm AoE. I encourage you to interact with the authors' rebuttal to clarify any remaining issue (and how well the rebuttal addressed your concerns) before the next phase.

Thanks, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.