acceptodds
Under review as a conference paper at ICLR 2027

Which Key-Value Cache Compressions Can Continued Pretraining Repair?

Abstract

Retrofitting pretrained transformers with smaller key–value (KV) caches can reduce memory use and accelerate decoding with limited additional training. Next-token prediction incentivizes the use of context, but whether continued pretraining restores retrieval after cache compression is unclear. We compare head merging, joint low-rank compression of keys and values, and token pooling at matched cache sizes and training budgets on Qwen3 from 0.6B to 8B and on Falcon3-1B, and extend the merge to five model fameters. Each conversion isevaluated against an unconverted control trained with the same recipe. Recovery in language-modeling loss and commonsense performance does not reliably predict multi-key retrieval, whichrget key from many similarentries. At 4K tokens and half the cache, a low-rank latent retains 97.9% of the control's multi-key retrieval at 4B, but head merging only 34.0%, despite similar commonsense performance. Ahe retrieval headsidentified in one forward pass also improves recovery at the training length, at a smaller cache saving, but the benefit is gone at 16K tokens. Continued pretraining also fails on hing. Rotating value headswithout adjusting their output projections permits an exact inverse correction, yet continued pretraining lowers the loss without restoring retrieval. From 0.6B to 8B, a closed-form localst 93.6% of the control'smulti-key retrieval without continued pretraining. These results motivate three principles for KV-cache retrofitting: evaluate multi-key retrieval explicitly, preserve retrieval-relevann misalignments locallybefore continued pretraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
architecture retrofit, continual pretraining, model conversion, attention, KV head merging, initialization, long-context retrieval, pre-registration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Z48fOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
3(+2)
Read review
Paper summary

This paper studies retrofitting of the KV-cache. Retrofitting KV cache consists of changing the KV architecture to a different type (possibly compressing it). For example, merging heads, converting a traditional attention head to a latent attention, or token pooling. After such KV-cache conversions are done, it is necessary to finetune the model to recover from performance degradation. In this context, this is referred to as continued pretraining. The paper argues such KV-cache compression approaches are typically evaluated on language modeling and commonsense reasoning, leaving it open whether retrieval capabilities also recover during continued pretraining (as measured by RULER's multi-key retrieval tasks) . Considering that, is is also not clear which types of KV compression can continued pretraining recover. As stated in the introduction, these are the questions the paper aims to cover: which KV-cache compression methods can be retrofitted with continued pretraining, and what determines whether performance on retrieval (associative recall) tasks also recovers after continued pretraining.

Their approach to verify this is to compare retrofitting by head merging, latent attention and token pooling across several models, including 5 different sizes of the Qwen3 family. They also evaluate the resulting models in downstream tasks and in the RULER multi-key retrieval tasks (primarily multikey 2). This task consists, essentially, in retrieving the correct value from the context which is written almost as a list of key-value associations.

The paper observes a strong discrepancy between the model's performances, after retrofitting, in typically evaluated tasks versus evaluation on multi-key retrieval. This is an interesting result. Overall, however, I've found it hard to obtain an answer for their stated question in the introduction. The presentation of the paper includes several results, but lacks synthesis. Furthermore, I believe it would benefit from a more clear "claim and evidence" structure, as it was hard for me to gather what were the main claims in the paper and what exactly supported them. Finally, from my understanding, the claims were not well supported by the given evidence in the main paper. For those reasons I recommend rejection.

Note: following the rebuttal, I've updated my scores and ratings.

Critical strengths

S1. Discrepancy between performance on retrieval and commonsense tasks: The paper shows some empirical evidence that performance on downstream tasks (such as MMLU) and language modeling does not correlate so well with performance on associative recall tasks, particularly in long contexts. This is an interesting result that could support those concerned with retrofitting their KV caches architecture while retaining performance on tasks that require significant retrieval.

Critical weaknesses

W1. Claims should be better supported I've found it hard to identify what are the main claims of this paper. Despite the general question being clearly given, I've found it difficult to identify what was the answer to this question and in this sense what were the main claims of the paper. Nevertheless, I've identified a few possible claims, which I generally believe are not well supported.

Claim 1 - Commonsense performance recovery hides retrieval performance: They claim that the predictive performance that is recovered through continued pretraining after converting the KV-cache architecture hides a lack of improvement in retrieval, as measured by associated recall on the RULER's multi-key tasks.

I think it is generally supported that performance recover on multi-key tasks (2 and 3) can be significantly worse than on the commonly used downstream tasks. They evaluate this at least in 5 Qwen3 models of varying sizes (Table 1).

However, there is a caveat: the performances in Table 1 are given for their head merging method. In Table 2, the Qwen3 results for latent attention seem to support that latent attention preserves retrieval capability. For instance, when keeping half of the cache size with Qwen3-4B, it preserves ≥97.0%\geq 97.0\% of the performance on multi-key retrieval for 4K, 8K and 16K tokens. Similarly, Section 6.2 mentions that if retrieval heads are retained then Qwen3-4B is able to recover performance.

Hence, it seems possible that the poor performance in retrieval might be actually an issue with their approach for merging heads, instead of a general phenomena. In fact, I've found the choice of head merging method (averaging two adjacent heads) to be not well-motivated. It could be better to use an existing established method from the literature.

(~ l. 111) They also claim a set of principles that should be considered when doing KV-cache retrofitting. I state these through the following claims:

Claim 2 - One should evaluate multi-key retrieval explicitly: The reason is related to the Claim 1. I do find Claim 1 has some level of support on the evidence. However, one could interpret this somewhat differently: MK2 and MK3 are not the most realistic datasets, and they measure a very specific capability. Hence, it is reasonable to measure that if the final application, after KV compression, requires it. However, if it does not require it, then evaluating on the common benchmarks is more appropriate, as they are typically measuring more general capabilities. In this sense, unless not preserving multikey retrieval performance has an implication I didn't catch from the paper, I see no reason for it to become an essential measure when doing general KV compression. Nevertheless, I do find this discrepancy interesting, and I've added it to the strenghts.

Claim 3 - Preserve Retrieval Relevant Structure Also related do Claim 2, I don't see this is a general relevant guideline when doing KV-cache retrofitting, unless the level of retrieval capability preservation measured by the more challenging datasets is required by the practicioner.
On the other hand, assuming the user is interested in that, then I find that this claim is supported. However, as cited by the authors, a mechanistic analysis work already provided evidence for the existence of retrieval heads that supported recall in long-contexts. In this sense, this guideline is not novel.

Claim 4 - Correct known misalignments locally before continued pretrianing This claim relates to Section 7 of the paper, where they consider their value-rotation KV-cache architectural modification. The method consists in applying a lossless transformation to the value projection matrix, specifically a rotation. As such, continued pretraining should in principle be able to recover full performance, since no information is actually removed. The training should learn the inverse rotations to the one applied.

However, their results showed that continued pretraining was not able to recover performance even for the value-rotation technique. As far as I understand it, it was possible to recover performance when doing a form of ridge fitting on the attention outputs. Results are shown in Table 5 for the value-rotation approach, and it seems this is the evidence to support this Claim 4. Hence, this analysis was apparently not performed on the actual KV compression methods (head merging, latent attention and token pooling). Having it done solely for the value-rotation, a scenario where there is a clear inverse, is not enough to support the general stated guideline.

W2. Presentation could be improved I believe this work could better state its claims and more clearly present the evidence that supports them. In its current form, the work seems to present several results, without properly synthesizing them and showing how they support their claims. I collected a few examples of presentation issues that could help improving the work, and also to clarify what concerns me with respect to the work's presentation.

(~ l. 52) "Head merging spans five model families from 0.6B to 14B parameters, and the comparisons across conversions use Qwen3 from 0.6B to 8B and Falcon3-1B.". It is not clear what is being compared. Across the paper, it seems that different models were used for each case, which makes it hard to interpret results.

(~ l. 80) "Cache sizes are matched as well, except that the 0.6B latent keeps a quarter of the cache, since at half it would be exact." This sentence is not clear.

Section 6.2 seems to be just listing results, without making a claim and then supporting it. I believe the goal of this section is to show that specifically preserving retrieval heads when doing head merging enables retrofitting. This could be clearly stated in the beginning of the section. Instead, the section starts by describing what was done.

Section 6.3 is also not clearly motivated, and I have a hard time understanding what is being measured.

In Section 5.4, the paper studies replacing the SiLU with ReLU in the MLP layers. This seems very much outside of the scope of this work.

Section 5.5 reports a result where more training did not improve results. However, this question is not motivated, and is also not clearly stated.

Questions for authors

Q1. Is my selection and description of the main claims appropriate? Could you clarify what are the main claims and the evidence that supports them?

Q2. Do you believe performance on MK2 and MK3 has some larger implication that is not more realistically measured by language modeling, downstream tasks and MK1, and that would justify the importance of measuring MK2 and MK3 when doing KV-cache retrofitting in general?

Q3. Is my interpretation of the evidence supporting the claim to correct local disagreements (Claim 4) correct?

6 comments
AuthorsOpenReview ↗

Part 1/4: the main claims and what the paper studies (W1, Q1)

Thank you for the detailed review. We first answer Q1 with our main claims, then clarify the study's motivation and comparison framework. Parts 2 to 4 address your other points one by one.

Reply to W1 and Q1: the main claims. To Q1: your selection is close to ours, with one claim missing (C2 below). Our main contribution is a controlled comparison of three KV-cache conversions at matched cache sizes and training budgets. The central finding is that recovering language-modeling loss and commonsense performance does not reliably predict recovery of multi-key retrieval. On Qwen3 at half the cache and our training budget, the low-rank latent ends close to its control on multi-key retrieval (the control is the unconverted model trained with the same recipe). Head merging and token pooling recover only part of the control's retrieval. Whether retrieval recovers depends on what each conversion preserves (C2, C3), and continued pretraining may not find a repair even when one exists (C4). The claims stated in §1 (C1-C4 are our labels here), together with the sections supporting them, are:

  • C1 (your Claim 1): "Recovery on conventional benchmarks hides large differences in retrieval" (§5.1-5.3). For example, on Qwen3-4B, the head merge and the latent both stay close to the control's commonsense average, but the head merge keeps only 34.0% of its multi-key retrieval and the latent 97.9% (Figure 1a).
  • C2 (not in your list): Retrieval differences reflect what each conversion preserves. Head merging and token pooling average heads or tokens and discard their differences. At the same cache size, a joint latent can represent any head merge and starts at least as close to the original key and value weights as the merge does (Proposition 1). In our experiments it can keep far more multi-key retrieval (§6.1, Table 3).
  • C3 (your Claim 3): Preserving retrieval heads substantially improves retention. Leaving the retrieval heads unmerged "keeps nearly all multi-key retrieval on Qwen3-4B at the training length, against about half for as many randomly chosen heads" (§6.2).
  • C4 (your Claim 4): Continued pretraining does not find the exact repair of the value rotation and leaves multi-key retrieval below the control, while a closed-form local solve, without continued pretraining, recovers at least 93.6% of the control's retrieval at every size from 0.6B to 8B (§7).

Your Claim 2, to evaluate multi-key retrieval explicitly, is the recommendation we draw from C1, summarized at the end of the paper (§8) as practical advice.

What the paper studies. Retrofitting studies typically examine a single type of conversion and assess recovery through language-modeling loss and commonsense benchmarks. Model developers, in contrast, evaluate retrieval explicitly: DeepSeek-V4 reports OpenAI MRCR, and MiniMax Sparse Attention reports RULER. This gap motivates the central question in §1: "Which KV-cache compressions can continued pretraining repair, and what determines whether retrieval recovers?"

We investigate this question through a controlled comparison of three KV-cache conversions. A model's KV cache has four axes, [layers, heads, tokens, features], and each conversion reduces a different axis: head merging (as in GQA uptraining) reduces heads; token pooling (similar to that used in DeepSeek-V4's CSA attention) reduces tokens; and joint low-rank latent compression (as in MLA) reduces features.

Two other ways to shrink the cache are outside our comparison. The first is cross-layer cache sharing, which shrinks the layer axis, as in DeepSeek-V4.1's CSA2 attention; we have recently begun investigating this approach. The second is quantization, which shrinks none of the four axes and only lowers the bit precision of each cached number; it is usually studied on its own.

Recent work also emphasizes the importance of retrieval evaluation. For example, after our submission, ByteDance Seed released a paper (arXiv:2609.36322) studying chunked KV-cache compression, corresponding to the token-axis compression studied in our paper. They pretrain models from scratch and we retrofit pretrained ones. Both settings reveal retrieval degradation that conventional metrics such as language-modeling loss fail to capture.

AuthorsOpenReview ↗

Part 2/4: your Claims 1 and 2 (Q2)

Reply to Claim 1: commonsense recovery hides retrieval.

On your concern that the retrieval loss may come from our head merge: you are right that not every conversion loses retrieval, and the two examples you raise (using latent attention and retaining the retrieval heads) are cases where it is kept. Claim 1 is about what the usual measures show: the language-modeling loss and the commonsense benchmarks cannot tell whether retrieval is kept or lost. Regarding your second example, on Qwen3-4B the merge that retains the retrieval heads and a merge of the same number of random pairs, both at 64% of the cache (§6.2), keep about the same commonsense average (94.2% and 94.6% of the control). Yet the first keeps 96.8% of the control's multi-key retrieval and the second 44.5% (Table 8, 400 prompts). The same gap shows in checkpoints released by another research group: MHA2MLA's SmolLM-1.7B checkpoints keep at least 95.1% of the commonsense average of their full-cache counterparts but less than half of their multi-key retrieval under either scoring, while its Llama-2-7B releases keep both (Table 8).

Regarding the head merge, we average the two heads of a pair (mean pooling). This is the standard method, used in the original GQA paper (Ainslie et al., 2023). Its ablation study found that mean pooling is the best of the initialization methods it compared. Using a better initialization method proposed by a later paper helps little. Rotating the two heads of a pair into a common basis before merging, as proposed in Align-heads (Jin et al., 2025), adds only 1.1-5.4 points on the three models we tried it on, and multi-key retrieval still stays low, at 8.5-39.4% of the control (App. C.7, 100 prompts).

Reply to Claim 2 and Q2: why multi-key retrieval matters.

To Q2: we believe the answer is yes. Evaluating retrieval is useful for any application that has to pick one entry among similar ones. We agree that MK2 and MK3 are synthetic, and we do not argue that multi-key retrieval should replace the standard benchmarks. As noted in Part 1, model developers report in-context retrieval for their releases, while the head-merging and latent studies we surveyed do not report multi-key retrieval at a controlled context length (App. D). Besides, multi-key retrieval checks a model's ability to select information among distractors, an ability that matters in realistic retrieval workloads. We show that the losses it reveals also appear on such realistic retrieval tasks, for example:

  • On HELMET's RAG tasks, where the answer passage sits among retrieved distractors, the Qwen3-4B head-merge model retains 66.5% of the control at 8K tokens and 49.0% at 16K, compared with 99.5% and 99.9% for the latent (Table 24).
  • On LongBench, the Qwen3-8B head-merge model retains 39.9% of its control on retrieval tasks but 75.8% on summarization (§5.2).

Single needles and MK1 show much less of the loss, which is why we ask for MK2. MK1 has only three distractor needles in ordinary text, while in MK2 the whole haystack is distractor needles. The Qwen3-4B merge keeps 92.8% of the control on single needles and 76.9% on MK1, but 40.9% on MK2 (Table 2, harness scoring).

Thus, we propose multi-key retrieval as a cheap check beside an application's own tasks, not in place of them.

AuthorsOpenReview ↗

Part 3/4: your Claims 3 and 4 (Q3)

Reply to Claim 3: what is new beyond retrieval heads.

You are right: Wu et al. (2025) identified the retrieval heads, and we cited this paper in our original submission. We use their retrieval score, with the setting described in §6.2. What differs is how the heads are used. Wu et al. found these heads in the original model and showed that disabling them breaks retrieval. Inference-time cache methods built on the same observation, such as DuoAttention, RazorAttention and HeadKV, keep every KV head and use head importance to decide how many tokens each head keeps. We study a different case: a conversion that permanently merges KV heads and then continues pretraining. The question there is whether it matters which heads are merged, since training might rebuild retrieval on the heads that remain. Our results show that it does matter. With the same number of pairs merged, sparing the retrieval heads keeps 96.8% of the control's multi-key retrieval on Qwen3-4B, and merging random pairs only keeps 44.5% (Table 8, 400 prompts). This result suggests that the heads to spare have to be chosen at the conversion.

As you note, the guideline matters when an application needs retrieval. Part 2 shows that such applications are common, for example question answering over retrieved passages with distractors, and not only synthetic tests.

Reply to Claim 4 and Q3: correct misalignments locally.

To Q3: yes. In our original manuscript, the local repair was shown only on the value-rotation probe, as our Limitations section said.

The reason to conduct the value-rotation probe is to study whether continued pretraining alone is enough to recover the retrieval lost in retrofitting. The three conversions we compare (head merge, latent, and token pooling) cannot answer this, because they delete information: when retrieval does not come back, we cannot identify whether it is because the information is gone or because training failed to find it. The value rotation has a useful property here: it does not delete any information, and it has a closed-form fix, which gives us a ground truth. So if retrieval does not come back, training is the reason, and we can measure whether the gradients point toward the ground truth. Retrieval did not come back at the budgets we tested. Thus, our results on the probe suggest fixing a known local mismatch before next-token prediction training.

We agree with you that this guideline needs more evidence on practical conversions, and we ran the following experiments (§7, Table 19). The local fix is applied before continued pretraining to compensate for the local mismatch: each attention module is first trained to match the original model's module output, a local alignment loss as in LoLCATs (Zhang et al., 2025), and then the standard next-token training is conducted. On the Qwen3-4B head merge, it raises the final multi-key retrieval from 31.5% to 39.2% of the control. This gain of 7.7 points is modest. The Qwen3-4B latent has little to repair, since it keeps nearly all of the control's retrieval at half the cache (Figure 1a), so we tested the latent on a smaller model and cache. On our 0.6B latent at an eighth of the cache, the plain latent loses most of its retrieval, although its rank still leaves room for what retrieval needs; the same local fix raises multi-key retrieval from 27.3% to 52.5% of the control (all numbers here use 400 prompts). On the same Qwen3-0.6B, it raises the head merge from 9.8% to 23.4%, about half of the latent's gain. Thus, on practical conversions the local fix helps this latent clearly and the head merge less. Unfortunately, on token pooling it does not help: the model ends at 33.6% of the control, against 55.2% without the fix. The abstract and §1 are now rewritten to present the guideline as a step to try where a misalignment is known, and §8 concludes with this scope. §8 also recommends measuring retrieval again after continued pretraining, as continued pretraining may lose what the fix restored.

Reviewer_kud4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies which KV-cache compressions continued pretraining (CPT) can repair, judged by head merging, a joint low-rank latent, and token pooling each halve the cache and get the same CPT budget, and each is compared with a control trained the same way. The key contributions of the paper are:

  1. Evaluation gap: loss, commonsense benchmarks, and single-needle tests hide large drops in multi-key retrieval.
  2. Low-rank latent beats merging and pooling: on Qwen3-4B it keeps 97.9% of retrieval, while head merging keeps 34.0%. Merging gets worse at longer contexts and smaller sizes.
  3. Takeaways: test multi-key retrieval, keep the structure retrieval depends on, and fix known misalignments locally before CPT.
  4. Impactful as a guide for choosing the type of KV cache compression to use

Some limitations are:

  1. Small training budget, with one recipe that weakens some models' controls.
  2. The latent and pooling comparisons cover only a few models.
  3. Only base models; no reasoning, math, or code evaluation.
  4. The latent that keeps retrieval decodes slowly (0.27x), and the fast MLA form keeps little retrieval at this budget.
Critical strengths
  1. A controlled demonstration that standard recovery metrics miss retrieval loss. Every conversion is compared with an unconverted control trained with the same recipe, and conversions are compared at matched cache sizes and budgets. This shows that loss, commonsense benchmarks, and single-needle tests can stay near the control while multi-key retrieval collapses (e.g., Qwen3-4B at half the cache: 97.9% of the control's multi-key retrieval for the latent vs. 34.0% for head merging, with similar commonsense scores). The finding holds across model sizes, families, context lengths, and two evaluation harnesses, and re-scoring released MHA2MLA checkpoints shows the same blind spot in published work. This shows that the current way of evaluating KV-cache retrofits misses a real failure, and it gives a concrete, cheap test to fix that.

  2. The value-rotation probe cleanly separates information loss from optimization failure. The probe deletes nothing and has an exact closed-form repair (Prop. 2), yet continued pretraining recovers the loss while leaving multi-key retrieval at <=8.5% of the control from 0.6B to 8B, even with 10x the tokens. This turns an observation into an explanation of when continued pretraining can and cannot repair a conversion, and it supports the practical advice to correct known misalignments locally before training.

Critical weaknesses
  1. The practical conclusions reflect the small training budget and a single recipe rather than the conversions themselves. All conversions are trained on ~150M tokens (at most ~1.5B in the 10x runs), at most a quarter of TransMLA's budget, using one recipe tuned on Qwen3. That recipe weakens the controls on other families (e.g., Mistral-7B multi-key retrieval falls from 85% to 1% at lr 1e-4; Llama-3.1-8B from 99% to 58%), so retention reported as a percentage of these controls is hard to interpret. The paper's own evidence points the same way: TransMLA's released MLA-form models, trained on 30-60B tokens, keep full retrieval, while the authors' MLA-form conversion kept 1.3% at their budget. The setup also cannot disentangle budget from conversion type. The only conversion that keeps retrieval (the paper's latent) decodes at 0.27x the full cache's speed, and the fast MLA form fails at this budget. The paper says compressing the feature dimension is the safer choice, but it does not show a conversion that saves memory, decodes faster, and keeps retrieval at a realistic budget. That limits how practical the guidance is and how far the conclusions generalize.

  2. The paper's main mechanistic result, that continued pretraining fails to find an exact repair and a closed-form local refit succeeds, is demonstrated only on the value-rotation probe, which the authors do not propose as a conversion. On the real conversions, the local remedies do little: a closed-form refit of the head merge leaves retrieval at 5.3% of control, and Align-heads adds only 1-5 points. Evaluation is also limited to base models with short-answer retrieval and commonsense tasks; reasoning or instruct models, chain-of-thought math/code, and long generation are not tested, even though long decoding is where KV compression matters most. The principle "correct known misalignments locally before continued pretraining" is supported in a constructed setting but not shown to help any practical retrofit. It is also unclear whether the retrieval failures carry over to the reasoning workloads with longer traces that motivate cache compression.

Questions for authors
  1. Budget and recipe. The MLA-form conversion of Llama-3.1-8B keeps 1.3% of multi-key retrieval at your budget, while TransMLA's releases (30-60B tokens) keep all of it. Was the MLA form trained at the lower learning rate (2e-5) that kept the Llama-3.1-8B control at 97% rather than 58%? Do you have any MLA-form or latent run at a larger budget (e.g., the 10x schedule)? If the gap between conversions narrows with budget or tuning, the main practical conclusion would weaken substantially.
  2. Weak controls. For families where your recipe lowers the control (Llama-2-7B, Llama-3.1-8B, Mistral-7B, Gemma-2-2B, Qwen3.5), do the head-merge vs. latent/pooling orderings still hold when retention is measured against the untrained base, or with a learning rate tuned per family? Could the authors report the latent on at least one non-Qwen model with more than 1B parameters?
  3. Retrieval-head sparing at matched cache. Sparing retrieval heads keeps 64% of the cache, against 50% for the full merge. How does it compare with a latent or pooling conversion at the same 64% cache? Do you have an explanation for why its benefit disappears at 16K tokens?
  4. Could the authors instruct and reasoning models on chain-of-thought math and code with long generation, where KV compression matters most?
  5. Could the authors run continued pretraining at long context to separate conversion effects from the degradation caused by training at 4K?
4 comments
AuthorsOpenReview ↗

Part 1/2: budget, recipe and a practical conversion (W1, Q1, Q2)

Thank you for the careful review. We ran additional experiments to answer your questions.

1. Budget (W1, Q1)

The MLA form was trained with a learning rate of 2e-5. Within the budgets we could run, neither the lower learning rate (Q2 below) nor more tokens close the gap. We have one MLA-form run with a larger training budget: trained four times longer, it keeps only 0.8% of the matched-budget control's multi-key retrieval. It loses its retrieval at the conversion, scoring 0% before any training, and four times the tokens do not bring it back. We did not train the latent longer, as it is already near the control.

Budgets like TransMLA's (30-60B tokens) can recover retrieval, and §8 states this limit. The paper's main finding, that recovery on commonsense benchmarks does not show whether multi-key retrieval recovers, is not specific to our budget. The released MHA2MLA checkpoints of SmolLM-1.7B, trained by their authors, keep 95.1-97.6% of their full-cache control's commonsense average but less than half of its multi-key retrieval under either scoring (Table 8).

2. Recipe and weak controls (W1, Q2)

Yes, where we could test it. Of the five families you list, two have a second conversion to compare the head merge with, and results show the ordering holds on both.

On Llama-3.1-8B we lowered the learning rate to 2e-5. The control then scores 97% on multi-key retrieval, against 99% for the untrained base, and the latent keeps 95.9% of that control where the head merge keeps 9.3% (Table 15). This is also the latent on a non-Qwen model above 1B that you asked for. The head-merge model keeps about the same percentage of the weak control: at 1e-4, where the control falls to 58% accuracy, it keeps 8.6%.

On Llama-2-7B we have only the 1e-4 runs. Pooled tokens keep 32.7% of the control and the head merge 5.8%. Measured against the untrained base instead, both percentages shrink and the order stays the same.

The other three families were only merged, so they have no ordering to test. On Mistral-7B, which your review mentioned, the control reaches 71% accuracy at 2e-5 (base 85%), and the merge still keeps only 7.0% of it (App. C.6).

3. A practical conversion (W1)

We agree: we do not yet have a conversion that saves memory, decodes faster and keeps retrieval at a realistic budget.

Much of the difficulty lies in the rotary embedding, and some in the kernels. Our latent keeps the original rotary embedding and keeps the retrieval, but it must rebuild every cached key before rotating it, at each step. Its kernel is our own fused kernel: per layer it is three times faster than the straightforward implementation, and the whole model still decodes at 0.27x the full cache's speed, and a lower key rank improves the decoding speed but costs retrieval before it gets fast enough (App. C.12). The MLA form, as used in TransMLA, changes the rotary embedding so that this reconstruction folds into the query projection, and it runs on kernels built for it. At 28% of the cache it decodes 1.41x as fast as the full cache at 16K tokens and batch 8 (Table 4), though end to end at batch 1 it is slower than the full cache up to 64K (Table 30). But the converted model starts without its retrieval, and our budget does not bring it back, while TransMLA's own releases, trained on 30-60B tokens, keep theirs.

The MLA form with a local alignment step is the closest method we can find. Matching each attention module to the original's output for 400 steps, as in LoLCATs (Zhang et al., 2025), with no continued pretraining, brings its retrieval back to 95.2% of the control. However, continued pretraining at our budget loses most of that retrieval again (W2 below). And without that training, the aligned form holds at short range only: on the harness's needle test it keeps 82.0% of the control's retrieval at 8K and 36.4% at 16K (§7).

AuthorsOpenReview ↗

Part 2/2: local repair, reasoning workloads and long context (W2, Q3, Q4, Q5)

4. Local repair and reasoning workloads (W2, Q4)

You are right that the submission showed the local repair only on the probe, and that the closed-form refit (App. C.7) and Align-heads (Jin et al., 2025) did little for the merge. In the revision we apply two other local steps before continued pretraining, a fitted initialization and module alignment, to four practical conversions: the latent, the head merge, the MLA form and pooled tokens (§7, Table 19). The result is mixed, and details are given below:

On the 0.6B latent at an eighth of the cache size, where the plain latent loses most of its retrieval, an initialization fitted in closed form to the original's keys and values raises the retrieval from 27.3% to 58.8% of the control (n=400). At a sixteenth of the cache size the rank is the limit, and the same fit gains little.

Unfortunately, the alignment does not help pooled tokens. It restores almost no retrieval (0.8%), and the training ends at 33.6% of the control, against 55.2% without it.

On the 4B head-merge model, module alignment alone reaches 63.7% of the control. Continued pretraining then lowers it to 39.2%, 7.7 points above the plain merge's 31.5% (n=400). The MLA form shows the same pattern more sharply: 95.2% after the alignment, 16.8% after the training. So on these two conversions our continued pretraining lowers the retrieval that the alignment restored. Our hypothesis, not tested, is that web text has few tokens that need a precise lookup (§6.3), so its loss gives the training little reason to keep the retrieval. Lookup data such as the synthetic registry documents helped the plain merge significantly (§6.3), although its gain carried over poorly to other retrieval formats, and we have not tried it after an alignment. We have not tested keeping the alignment loss during that training either, but our intuition is that it might work. §8 now states the principle with this limit, and asks to measure retrieval again after any training that follows an alignment.

On base models (Q4): We ran GSM8K with an eight-shot chain-of-thought prompt, MATH-500 with up to 1,024 generated tokens, MBPP and HumanEval, on the 4B and 8B checkpoints (App. C.10, Table 25). Results show that the head-merge model loses more here than MMLU shows. On GSM8K the 4B merge keeps 40.7% of the control and the latent 95.5%, where on MMLU the same merge keeps 80.0%. MATH-500 and MBPP give the same order at both sizes.

However, our recipe also lowers the control itself on these tasks. HumanEval is the extreme case: the 4B control falls to 9.8% accuracy, so we do not read its ratios at 4B, and at 8B the merge and the latent score about the same.

We did not continue pretraining a reasoning model, as web text would likely overwrite the post-training and we cannot reproduce the model developers' recipe for a fair control. So we converted the post-trained Qwen3-8B in thinking mode and repaired it with the module alignment only. On GSM8K with up to 3,072 generated tokens, the unaligned latent keeps 46.2% of the original's accuracy and the unaligned merge fails to solve any. After the alignment the latent keeps 101.3% and the merge 75.3% (Table 26). This is a single model and benchmark, with one sampling seed. We did not test the same model with thinking off, as a plain instruct model, or code on a post-trained model. We agree these cases matter.

5. Sparing at a matched cache, and long context (Q3, Q5)

On the matched cache (Q3), we did not run the comparison at 64%: we trained no latent at that size, and fixed pooling cannot reach it because its block size is an integer. What we have is at half the cache, on Qwen3-4B at 4K: the latent keeps 97.1% of the control and pooled tokens 55.2% (n=400, Table 35), against the spared merge's 96.8% at 64% (§6.2). So the latent keeps as much as the spared merge with a smaller cache.

On training at long context (Q5): We ran experiments with 16,384-token sequences and the same number of training tokens on Qwen3-4B. Results show that the drop at 16K came from our 4K training recipe. Trained at 16K, the control's multi-key accuracy at 16K goes from 70% to 99%. However, the plain head merge recovers only part of the way. Trained at 16K, it keeps 30.3% of the control at 16K, which is still far below the control. It also scores much lower than the latent model trained only at 4K, which reaches 73% accuracy at 16K (Table 22) against the merge's 30%.

On the reason why the benefit of the spared merge disappears at 16K (Q3): We applied the same experimental setting to the spared merge model and saw a huge improvement: it reaches 90.9% of its control (one seed each, n=100; Table 23). Thus, training at the target sequence length is a simple fix that recovers the retrieval of the spared merge model. We thank you for suggesting the long-context training.

I thank the authors for the clarifications and for responding to my questions. Overall, I think that the paper would benefit a lot from focusing on larger scale models (where KV cache compression becomes increasingly important), reasoning or instruction following models and on generative tasks (GSM8K Humaneval(+), MATH-500) where one can see the effect of compression more clearly. I keep my positive assessment of the paper.

Thank you for your follow-up and for keeping your positive assessment. We agree on the value of scaling to larger models and of generative tasks. Our evaluation already includes some generative tasks and a post-trained model: multi-step math (GSM8K) andcode (Table 25), and Qwen3-8B in its thinking mode (Table 26, with only a local fix and no continued pretraining). In the next version of the paper, we will discuss these generative and post-trained results more fully in the main text and mention scaling to larger models and retrofitting reasoning models beyond the local fix as next steps.

Reviewer_ZTaVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper has tackled an important issues in the efficient LLM inference, where various KV cache compression methods are used to improve the throughput and reduces the inference cost for GPU severing. Authors have compared three widely adopted KV cache approaches -- head merging, joint compression of keys and values, and token pooling -- at matched cache sizes and training budgets for retrofitting. Evaluations are conducted on Qwen3 and Falcon3 model families across different model scale.

Critical strengths

Authors have selected three popular cache compression techniques, and then showed their efficacy for multi-key retrieval, ranging from 4k to 16k sequence length, which are often the under-analyzed. I believe that, this study shed light beyond what is being evaluated for the selection of KV cache methods and retrofititg, and the upcoming approached would take the Multi-key retrieval performance as an important optimization criteria.

Critical weaknesses

Only weakness that I think of, is the evaluation of task-specific acquisition gain and forgetting during the continued pre-training. Evaluating the plasticity loss and forgetting can also help us understand the pros and cons of various KV cache compression methods.

Questions for authors

See the weakness.

1 comment
AuthorsOpenReview ↗

Response to ZTaV

Thank you for the positive assessment and for recognizing the value of broader evaluation in retrofitting. This motivated us to conduct this study in the first place, and we believe fuller evaluation gives a more faithful signal and can guide better, more efficient retrofitting.

1. Task-specific acquisition gain and forgetting. We report each task separately and add new chain-of-thought math and code experiments (GSM8K, MATH-500, MBPP) to test a broader range of skills, on Qwen3-4B at half the cache and the matched training budget. Values are % of the control (the unconverted model trained with the same recipe).

#taskhead mergepooled tokenslatent
1MMLU80.086.399.0
2ARC-E93.898.0100.3
3ARC-C88.189.898.8
4PIQA98.097.8100.4
5HellaSwag92.595.499.9
6OpenBookQA96.295.299.5
7WinoGrande90.693.0100.7
8BoolQ88.493.897.7
9multi-key retrieval at 4K31.555.297.1
10GSM8K-CoT40.771.595.5
11MATH-50037.848.894.5
12MBPP42.736.294.3

As the table shows, the head merge reaches 80.0-98.0% of the control on the eight standard tasks, but only 31.5-42.7% on multi-key retrieval, math and code (rows 9-12). Token pooling shows a similar imbalance, while the latent retains at least 94.3% on all twelve tasks. Thus, how much performance is retained after retrofitting depends strongly on both the task and the conversion.

2. Task-specific training and plasticity. Plasticity describes how readily a model can acquire or recover capabilities after retrofitting. Reaching a given performance level with fewer training tokens, or a higher level with the same tokens, indicates greater plasticity.

Section 6.3 examines recovery at a fixed training budget. Adding synthetic registry documents to continued pretraining brings the head-merged 4B model to 92.0% of its matched control on multi-key retrieval, versus 34.0% when trained on web text alone (row 9 gives 31.5% because it uses 400 prompts instead of 100).

The gain transfers poorly: with only the format of the keys and values changed (using random strings), the same model reaches just 20.0% accuracy, while the unconverted model trained on the same data reaches 95.5% (Table 17). The head-merged model therefore remains capable of task-specific recovery, but this does not establish recovery of broader retrieval ability.

Table 35 and the math/code evaluations in Table 25 are new in the revised manuscript. The task-specific recovery and transfer experiments were already in Section 6.3, recorded in Tables 12 and 17.

Comments on the paper

1 comment

We thank the reviewers for their careful reviews. We have uploaded a revised PDF with the changes marked in blue. In short:

  • New experiments: the latent on Llama-3.1-8B (§6.1); local repair steps on four practical conversions, beyond the value-rotation probe (§7, Table 19); chain-of-thought math and code on base models, and one reasoning model (App. C.10); continued pretraining at 16K tokens (App. C.10); TransMLA's MLA form at four times the budget (§7); a task-by-task table for all three conversions (App. F.1).
  • Presentation: we rewrote the passages Reviewer Z48f points to. For example, §6.2 and §6.3 now open with their claims, and the MLP control moved from §5.4 to Appendix C.1.

The details are in our reply to each reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.