- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 3(+2)
Read review
Paper summary
This paper studies retrofitting of the KV-cache. Retrofitting KV cache consists of changing the KV architecture to a different type (possibly compressing it). For example, merging heads, converting a traditional attention head to a latent attention, or token pooling. After such KV-cache conversions are done, it is necessary to finetune the model to recover from performance degradation. In this context, this is referred to as continued pretraining. The paper argues such KV-cache compression approaches are typically evaluated on language modeling and commonsense reasoning, leaving it open whether retrieval capabilities also recover during continued pretraining (as measured by RULER's multi-key retrieval tasks) . Considering that, is is also not clear which types of KV compression can continued pretraining recover. As stated in the introduction, these are the questions the paper aims to cover: which KV-cache compression methods can be retrofitted with continued pretraining, and what determines whether performance on retrieval (associative recall) tasks also recovers after continued pretraining.
Their approach to verify this is to compare retrofitting by head merging, latent attention and token pooling across several models, including 5 different sizes of the Qwen3 family. They also evaluate the resulting models in downstream tasks and in the RULER multi-key retrieval tasks (primarily multikey 2). This task consists, essentially, in retrieving the correct value from the context which is written almost as a list of key-value associations.
The paper observes a strong discrepancy between the model's performances, after retrofitting, in typically evaluated tasks versus evaluation on multi-key retrieval. This is an interesting result. Overall, however, I've found it hard to obtain an answer for their stated question in the introduction. The presentation of the paper includes several results, but lacks synthesis. Furthermore, I believe it would benefit from a more clear "claim and evidence" structure, as it was hard for me to gather what were the main claims in the paper and what exactly supported them. Finally, from my understanding, the claims were not well supported by the given evidence in the main paper. For those reasons I recommend rejection.
Note: following the rebuttal, I've updated my scores and ratings.
Critical strengths
S1. Discrepancy between performance on retrieval and commonsense tasks: The paper shows some empirical evidence that performance on downstream tasks (such as MMLU) and language modeling does not correlate so well with performance on associative recall tasks, particularly in long contexts. This is an interesting result that could support those concerned with retrofitting their KV caches architecture while retaining performance on tasks that require significant retrieval.
Critical weaknesses
W1. Claims should be better supported I've found it hard to identify what are the main claims of this paper. Despite the general question being clearly given, I've found it difficult to identify what was the answer to this question and in this sense what were the main claims of the paper. Nevertheless, I've identified a few possible claims, which I generally believe are not well supported.
Claim 1 - Commonsense performance recovery hides retrieval performance: They claim that the predictive performance that is recovered through continued pretraining after converting the KV-cache architecture hides a lack of improvement in retrieval, as measured by associated recall on the RULER's multi-key tasks.
I think it is generally supported that performance recover on multi-key tasks (2 and 3) can be significantly worse than on the commonly used downstream tasks. They evaluate this at least in 5 Qwen3 models of varying sizes (Table 1).
However, there is a caveat: the performances in Table 1 are given for their head merging method. In Table 2, the Qwen3 results for latent attention seem to support that latent attention preserves retrieval capability. For instance, when keeping half of the cache size with Qwen3-4B, it preserves of the performance on multi-key retrieval for 4K, 8K and 16K tokens. Similarly, Section 6.2 mentions that if retrieval heads are retained then Qwen3-4B is able to recover performance.
Hence, it seems possible that the poor performance in retrieval might be actually an issue with their approach for merging heads, instead of a general phenomena. In fact, I've found the choice of head merging method (averaging two adjacent heads) to be not well-motivated. It could be better to use an existing established method from the literature.
(~ l. 111) They also claim a set of principles that should be considered when doing KV-cache retrofitting. I state these through the following claims:
Claim 2 - One should evaluate multi-key retrieval explicitly: The reason is related to the Claim 1. I do find Claim 1 has some level of support on the evidence. However, one could interpret this somewhat differently: MK2 and MK3 are not the most realistic datasets, and they measure a very specific capability. Hence, it is reasonable to measure that if the final application, after KV compression, requires it. However, if it does not require it, then evaluating on the common benchmarks is more appropriate, as they are typically measuring more general capabilities. In this sense, unless not preserving multikey retrieval performance has an implication I didn't catch from the paper, I see no reason for it to become an essential measure when doing general KV compression. Nevertheless, I do find this discrepancy interesting, and I've added it to the strenghts.
Claim 3 - Preserve Retrieval Relevant Structure
Also related do Claim 2, I don't see this is a general relevant guideline when doing KV-cache retrofitting, unless the level of retrieval capability preservation measured by the more challenging datasets is required by the practicioner.
On the other hand, assuming the user is interested in that, then I find that this claim is supported.
However, as cited by the authors, a mechanistic analysis work already provided evidence for the existence of retrieval heads that supported recall in long-contexts. In this sense, this guideline is not novel.
Claim 4 - Correct known misalignments locally before continued pretrianing This claim relates to Section 7 of the paper, where they consider their value-rotation KV-cache architectural modification. The method consists in applying a lossless transformation to the value projection matrix, specifically a rotation. As such, continued pretraining should in principle be able to recover full performance, since no information is actually removed. The training should learn the inverse rotations to the one applied.
However, their results showed that continued pretraining was not able to recover performance even for the value-rotation technique. As far as I understand it, it was possible to recover performance when doing a form of ridge fitting on the attention outputs. Results are shown in Table 5 for the value-rotation approach, and it seems this is the evidence to support this Claim 4. Hence, this analysis was apparently not performed on the actual KV compression methods (head merging, latent attention and token pooling). Having it done solely for the value-rotation, a scenario where there is a clear inverse, is not enough to support the general stated guideline.
W2. Presentation could be improved I believe this work could better state its claims and more clearly present the evidence that supports them. In its current form, the work seems to present several results, without properly synthesizing them and showing how they support their claims. I collected a few examples of presentation issues that could help improving the work, and also to clarify what concerns me with respect to the work's presentation.
(~ l. 52) "Head merging spans five model families from 0.6B to 14B parameters, and the comparisons across conversions use Qwen3 from 0.6B to 8B and Falcon3-1B.". It is not clear what is being compared. Across the paper, it seems that different models were used for each case, which makes it hard to interpret results.
(~ l. 80) "Cache sizes are matched as well, except that the 0.6B latent keeps a quarter of the cache, since at half it would be exact." This sentence is not clear.
Section 6.2 seems to be just listing results, without making a claim and then supporting it. I believe the goal of this section is to show that specifically preserving retrieval heads when doing head merging enables retrofitting. This could be clearly stated in the beginning of the section. Instead, the section starts by describing what was done.
Section 6.3 is also not clearly motivated, and I have a hard time understanding what is being measured.
In Section 5.4, the paper studies replacing the SiLU with ReLU in the MLP layers. This seems very much outside of the scope of this work.
Section 5.5 reports a result where more training did not improve results. However, this question is not motivated, and is also not clearly stated.
Questions for authors
Q1. Is my selection and description of the main claims appropriate? Could you clarify what are the main claims and the evidence that supports them?
Q2. Do you believe performance on MK2 and MK3 has some larger implication that is not more realistically measured by language modeling, downstream tasks and MK1, and that would justify the importance of measuring MK2 and MK3 when doing KV-cache retrofitting in general?
Q3. Is my interpretation of the evidence supporting the claim to correct local disagreements (Claim 4) correct?
6 comments
Part 2/4: your Claims 1 and 2 (Q2)
Reply to Claim 1: commonsense recovery hides retrieval.
On your concern that the retrieval loss may come from our head merge: you are right that not every conversion loses retrieval, and the two examples you raise (using latent attention and retaining the retrieval heads) are cases where it is kept. Claim 1 is about what the usual measures show: the language-modeling loss and the commonsense benchmarks cannot tell whether retrieval is kept or lost. Regarding your second example, on Qwen3-4B the merge that retains the retrieval heads and a merge of the same number of random pairs, both at 64% of the cache (§6.2), keep about the same commonsense average (94.2% and 94.6% of the control). Yet the first keeps 96.8% of the control's multi-key retrieval and the second 44.5% (Table 8, 400 prompts). The same gap shows in checkpoints released by another research group: MHA2MLA's SmolLM-1.7B checkpoints keep at least 95.1% of the commonsense average of their full-cache counterparts but less than half of their multi-key retrieval under either scoring, while its Llama-2-7B releases keep both (Table 8).
Regarding the head merge, we average the two heads of a pair (mean pooling). This is the standard method, used in the original GQA paper (Ainslie et al., 2023). Its ablation study found that mean pooling is the best of the initialization methods it compared. Using a better initialization method proposed by a later paper helps little. Rotating the two heads of a pair into a common basis before merging, as proposed in Align-heads (Jin et al., 2025), adds only 1.1-5.4 points on the three models we tried it on, and multi-key retrieval still stays low, at 8.5-39.4% of the control (App. C.7, 100 prompts).
Reply to Claim 2 and Q2: why multi-key retrieval matters.
To Q2: we believe the answer is yes. Evaluating retrieval is useful for any application that has to pick one entry among similar ones. We agree that MK2 and MK3 are synthetic, and we do not argue that multi-key retrieval should replace the standard benchmarks. As noted in Part 1, model developers report in-context retrieval for their releases, while the head-merging and latent studies we surveyed do not report multi-key retrieval at a controlled context length (App. D). Besides, multi-key retrieval checks a model's ability to select information among distractors, an ability that matters in realistic retrieval workloads. We show that the losses it reveals also appear on such realistic retrieval tasks, for example:
- On HELMET's RAG tasks, where the answer passage sits among retrieved distractors, the Qwen3-4B head-merge model retains 66.5% of the control at 8K tokens and 49.0% at 16K, compared with 99.5% and 99.9% for the latent (Table 24).
- On LongBench, the Qwen3-8B head-merge model retains 39.9% of its control on retrieval tasks but 75.8% on summarization (§5.2).
Single needles and MK1 show much less of the loss, which is why we ask for MK2. MK1 has only three distractor needles in ordinary text, while in MK2 the whole haystack is distractor needles. The Qwen3-4B merge keeps 92.8% of the control on single needles and 76.9% on MK1, but 40.9% on MK2 (Table 2, harness scoring).
Thus, we propose multi-key retrieval as a cheap check beside an application's own tasks, not in place of them.
Part 3/4: your Claims 3 and 4 (Q3)
Reply to Claim 3: what is new beyond retrieval heads.
You are right: Wu et al. (2025) identified the retrieval heads, and we cited this paper in our original submission. We use their retrieval score, with the setting described in §6.2. What differs is how the heads are used. Wu et al. found these heads in the original model and showed that disabling them breaks retrieval. Inference-time cache methods built on the same observation, such as DuoAttention, RazorAttention and HeadKV, keep every KV head and use head importance to decide how many tokens each head keeps. We study a different case: a conversion that permanently merges KV heads and then continues pretraining. The question there is whether it matters which heads are merged, since training might rebuild retrieval on the heads that remain. Our results show that it does matter. With the same number of pairs merged, sparing the retrieval heads keeps 96.8% of the control's multi-key retrieval on Qwen3-4B, and merging random pairs only keeps 44.5% (Table 8, 400 prompts). This result suggests that the heads to spare have to be chosen at the conversion.
As you note, the guideline matters when an application needs retrieval. Part 2 shows that such applications are common, for example question answering over retrieved passages with distractors, and not only synthetic tests.
Reply to Claim 4 and Q3: correct misalignments locally.
To Q3: yes. In our original manuscript, the local repair was shown only on the value-rotation probe, as our Limitations section said.
The reason to conduct the value-rotation probe is to study whether continued pretraining alone is enough to recover the retrieval lost in retrofitting. The three conversions we compare (head merge, latent, and token pooling) cannot answer this, because they delete information: when retrieval does not come back, we cannot identify whether it is because the information is gone or because training failed to find it. The value rotation has a useful property here: it does not delete any information, and it has a closed-form fix, which gives us a ground truth. So if retrieval does not come back, training is the reason, and we can measure whether the gradients point toward the ground truth. Retrieval did not come back at the budgets we tested. Thus, our results on the probe suggest fixing a known local mismatch before next-token prediction training.
We agree with you that this guideline needs more evidence on practical conversions, and we ran the following experiments (§7, Table 19). The local fix is applied before continued pretraining to compensate for the local mismatch: each attention module is first trained to match the original model's module output, a local alignment loss as in LoLCATs (Zhang et al., 2025), and then the standard next-token training is conducted. On the Qwen3-4B head merge, it raises the final multi-key retrieval from 31.5% to 39.2% of the control. This gain of 7.7 points is modest. The Qwen3-4B latent has little to repair, since it keeps nearly all of the control's retrieval at half the cache (Figure 1a), so we tested the latent on a smaller model and cache. On our 0.6B latent at an eighth of the cache, the plain latent loses most of its retrieval, although its rank still leaves room for what retrieval needs; the same local fix raises multi-key retrieval from 27.3% to 52.5% of the control (all numbers here use 400 prompts). On the same Qwen3-0.6B, it raises the head merge from 9.8% to 23.4%, about half of the latent's gain. Thus, on practical conversions the local fix helps this latent clearly and the head merge less. Unfortunately, on token pooling it does not help: the model ends at 33.6% of the control, against 55.2% without the fix. The abstract and §1 are now rewritten to present the guideline as a step to try where a misalignment is known, and §8 concludes with this scope. §8 also recommends measuring retrieval again after continued pretraining, as continued pretraining may lose what the fix restored.
Part 1/4: the main claims and what the paper studies (W1, Q1)
Thank you for the detailed review. We first answer Q1 with our main claims, then clarify the study's motivation and comparison framework. Parts 2 to 4 address your other points one by one.
Reply to W1 and Q1: the main claims. To Q1: your selection is close to ours, with one claim missing (C2 below). Our main contribution is a controlled comparison of three KV-cache conversions at matched cache sizes and training budgets. The central finding is that recovering language-modeling loss and commonsense performance does not reliably predict recovery of multi-key retrieval. On Qwen3 at half the cache and our training budget, the low-rank latent ends close to its control on multi-key retrieval (the control is the unconverted model trained with the same recipe). Head merging and token pooling recover only part of the control's retrieval. Whether retrieval recovers depends on what each conversion preserves (C2, C3), and continued pretraining may not find a repair even when one exists (C4). The claims stated in §1 (C1-C4 are our labels here), together with the sections supporting them, are:
Your Claim 2, to evaluate multi-key retrieval explicitly, is the recommendation we draw from C1, summarized at the end of the paper (§8) as practical advice.
What the paper studies. Retrofitting studies typically examine a single type of conversion and assess recovery through language-modeling loss and commonsense benchmarks. Model developers, in contrast, evaluate retrieval explicitly: DeepSeek-V4 reports OpenAI MRCR, and MiniMax Sparse Attention reports RULER. This gap motivates the central question in §1: "Which KV-cache compressions can continued pretraining repair, and what determines whether retrieval recovers?"
We investigate this question through a controlled comparison of three KV-cache conversions. A model's KV cache has four axes, [layers, heads, tokens, features], and each conversion reduces a different axis: head merging (as in GQA uptraining) reduces heads; token pooling (similar to that used in DeepSeek-V4's CSA attention) reduces tokens; and joint low-rank latent compression (as in MLA) reduces features.
Two other ways to shrink the cache are outside our comparison. The first is cross-layer cache sharing, which shrinks the layer axis, as in DeepSeek-V4.1's CSA2 attention; we have recently begun investigating this approach. The second is quantization, which shrinks none of the four axes and only lowers the bit precision of each cached number; it is usually studied on its own.
Recent work also emphasizes the importance of retrieval evaluation. For example, after our submission, ByteDance Seed released a paper (arXiv:2609.36322) studying chunked KV-cache compression, corresponding to the token-axis compression studied in our paper. They pretrain models from scratch and we retrofit pretrained ones. Both settings reveal retrieval degradation that conventional metrics such as language-modeling loss fail to capture.