acceptodds
Under review as a conference paper at ICLR 2027

KVRelay: Reusing KV Caches Across Language Models without Re-Prefilling

17 views

Abstract

Deployed language models reuse context far more often than they compute it: the same document, prompt or session is encoded once and read back many times, which turns its KV cache into stored infrastructure. Model iteration invalidates that infrastructure. The day a deployment upgrades, every cached context still lives in the previous model’s attention space, and the new model has to prefill all of it again. We ask whether a target model can inherit a KV cache produced by another model and continue inference without re-prefilling the context. The obstacle is that a cache is an interface rather than a portable encoding of text: even between models that share a tokenizer, depth and KV geometry, injecting the source cache directly leaves the target below its own no-context accuracy on four of five pairs. We introduce KVRelay, which treats cross-model reuse as interface adaptation. It supervises a translated cache by the computation it induces inside frozen target attention rather than by element-wise reconstruction of the target’s native KV tensors; it starts from a parameter-free, target-calibrated base map and learns a zero-initialized cross-head correction; and it gives the Value branch a short causal mixer over source history. Across five Qwen/Llama model pairs and five context-dependent benchmarks, KVRelay recovers 90.3 ± 1.0% of the tar- get’s native context gain over five seeds with a translator worth 0.3–5.1% of target parameters; replacing functional supervision with direct KV MSE reduces five- pair mean accuracy by 4.88 percentage points. Complete translation costs 13.6% of target prefill on average and reduces end-to-end time to first token by 3.41×. KVRelay turns a model upgrade from a full cache rebuild into a cache handoff.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, cross-model transfer, large language models, efficient inference, model serving, cache translation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_r4BxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KVRelay, a lightweight translator that allows a target model to reuse another model’s KV cache without re-prefilling the context. Both models remain frozen, and the translator is trained by matching the target’s attention outputs rather than its KV tensors. Experiments across five model pairs show improved accuracy over direct cache reuse and Ridge mapping, with reduced time to first token.

Critical strengths

The paper provides useful evidence that a lightweight translator can make existing KV caches usable by another frozen model. Attention-output supervision consistently outperforms direct KV reconstruction across the tested settings. The five-seed results and measured translation costs support the findings within the evaluated scope.

Critical weaknesses

1.The comparison with existing learnable cache translation methods is limited. Previous work has also explored using transferred caches without requiring the receiver to re-encode the source context.¹ ² Although these methods differ in architecture and training objectives, a clearer discussion and controlled comparisons would help establish what the proposed design adds.

2.The length evaluation is limited. Main experiments cap contexts at 512 tokens, while the tests up to 4096 tokens repeat short passages. These settings do not fully reflect the complexity of long documents, ongoing conversations, or agent handoffs discussed in the paper. This scope is understandable for an exploratory study, but the evidence for these applications remains preliminary.

3.The supervision ablation lacks a clean comparison with language-modeling cross-entropy or output KL distillation. Earlier work already uses language-modeling or generation losses to train cache translators.¹ ² The mixed objective in this paper combines attention alignment, KL, and context contrast, so its lower performance does not establish an advantage over either alternative alone. Comparing these objectives with the same translator and training budget would better isolate the contribution of attention-output supervision.

References: [1] Lucio M. Dery et al. Latent Space Communication via K-V Cache Alignment. arXiv:2601.06123, 2026. https://arxiv.org/abs/2601.06123v1 [2] Siyi Chen et al. See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents. arXiv:2606.13594, 2026. https://arxiv.org/abs/2606.13594v1

Questions for authors
  1. Could you clarify the main advantages over the prior cache translation methods cited above and provide a comparison under the same model pairs and evaluation protocol, including translation cost?

  2. How does attention-output supervision compare with standalone language-modeling cross-entropy and output KL distillation when the translator architecture and training budget are fixed?

Reviewer_A7saOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a target LLM can reuse a KV cache produced by a different source model without prefilling the context again. Directly plugging in the source cache makes the target worse than having no context on 44 of 55 pairs. KVRelay removes the source RoPE, maps the cache with a mean/variance matching step plus a small learned linear correction (and a short causal mixer on Values), then applies the target RoPE. It is trained by matching the frozen target's attention outputs rather than the KV tensors. On five Qwen/Llama pairs and five multiple-choice benchmarks, it recovers about 9090% of the target's context gain, beats a Ridge baseline by about 55 points, and gives a 3.41×3.41\times mean TTFT speedup.

Critical strengths

S1: The main finding is well supported. With everything else kept the same, the attention-output loss beats KV MSE in all 2525 pair/task cells (4.884.88 points on average), and the main method is stable over five seeds.

S2: The setting is strict and practical. The target never prefills, recomputes or replays the context, and the timing includes the full conversion cost, so the speedup claims look fair.

Critical weaknesses

W1: The model pairs do not match the motivation. The Limitations section itself describes the pairs as "differently post-trained models within a pretrained family" plus one cross-scale Qwen3 pair, and the Llama pair is described as a base model and its own fine-tune. Yet Section 5.1 and Table 2 call four of the pairs "independently trained", which seems inconsistent with this. The Qwen3 pair, the only cross-scale pair, is also the weakest (0.820.82 recovery, about 2121 points behind the native cache at 40964096 tokens). Independently pretrained families and changes in depth, KV geometry or tokenizer are not tested.

W2: The evaluation is far from where cache reuse matters most. All tasks are multiple-choice, contexts are capped at 512512 tokens, and four of five benchmarks are disjoint splits of training datasets. There is no generation and no natural long-document task. In the repeated-context test, four pairs hold up well, but the Qwen3 pair degrades sharply at 40964096 tokens.

Questions for authors

Q1: Can you add at least one pair from independently pretrained model families? Please also clarify what "independently trained" means in Section 5.1, given the Limitations section.

Q2: How does the method perform on generation tasks and on natural long documents? Appendix E notes that the repeated-context test is not a substitute for natural long-document evaluation.

Q3: The Value mixer's gain (0.940.94 points) comes from one seed on one pair, where the main method's seed std is 0.660.66. Does it hold across seeds and pairs? The same applies to the other ablations in Table 4.

Q4: Identity and moment-matched bases differ by at most 0.580.58 points, and Appendix A says the base form does not drive accuracy. Should the abstract credit a source-conditioned start rather than "calibration"?

Q5: What is the per-pair training cost, and after how many reuses does translation pay off?

Reviewer_CpHROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes KVRelay, a method for reusing KV caches across different language models without requiring the target model to re-prefill the original context. The proposed approach combines functional alignment of attention outputs, a moment-matched base mapping, zero-initialized cross-head linear corrections, and causal mixing of Value representations.

The method is evaluated on five source–target model pairs from the Qwen and Llama families across five context-dependent benchmarks. The authors report recovering approximately 90.3% of the target model's native context gain while achieving an average 3.41× improvement in time to first token.

Critical strengths

Cross-model KV-cache reuse is an important problem for efficient LLM inference, particularly in scenarios involving model upgrades, dynamic routing, and multi-agent systems. Avoiding redundant context prefill has clear practical motivation.

The reported improvements over naive cache reuse and the Ridge mapping baseline are encouraging. The proposed method achieves performance close to native target-model caching while substantially reducing inference latency.

It is a good idea to optimize the attention outputs induced by translated caches instead of reconstructing target KV tensors.

Critical weaknesses

The main ideas overlap substantially with existing cross-model KV-cache transfer methods, particularly Heo et al. (2026), which already explores linear cache mappings and identifies attention-output similarity as an important indicator of transfer quality. KVRelay's main novelty appears to be directly optimizing this similarity, while other components largely build on established techniques.

All evaluated model pairs share compatible KV geometries, equal layer counts, and tokenizers. The evaluation does not cover independently pretrained model families or heterogeneous architectures, limiting the generality of the conclusions.

Experiments focus on multiple-choice benchmarks rather than open-ended generation, long-context reasoning, or multi-turn agent workflows. These settings are important for demonstrating the proposed method's practical value.

Although the Ridge baseline and ablation studies are useful, direct comparisons with stronger learned KV-cache transfer approaches would better establish the method's advantages over existing work.

Questions for authors

How does the proposed functional attention-alignment objective differ conceptually from the attention-output similarity analysis in Heo et al. beyond using the similarity measure as a training loss?

How would KVRelay extend to model pairs with different layer counts, KV-head dimensions, or independently pretrained architectures?

Reviewer_ECMzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KVRelay, a method that lets a target model reuse the KV cache computed by source model, so that the target does not need to prefill the context again. Each layer combines a parameter-free base map that matches the target cache's per-channel mean and variance with a zero-initialized linear correction that mixes all heads. The value branch also gets a short causal convolution over source tokens. The translator is trained so that the frozen target's attention outputs on the query and answer tokens match those it produces with its own native cache, instead of matching the KV tensors directly. On five same-family model pairs and five multiple-choice benchmarks, KVRelay recovers about 90% of the target's accuracy gain from context and reduces TTFT by 3.41× on average.

Critical strengths
  • The paper carefully discusses why the problem setting matters. Reusing a KV cache across models is an important problem and seems to have various applications, e.g., model upgrades, routing, and agent handoffs in Figure 1.
  • The negative finding that motivates the work is clear. Even when tokenizer, depth, and KV geometry all match, injecting the raw source cache is worse than giving no context at all on four of five pairs (Table 2).
  • The translator is simple and small (one linear map per layer and branch). The paper also shows that adding depth or nonlinearity does not help (Table 4).
Critical weaknesses
  1. The experimental setting is too limited to support the method's validity in the scenarios of Figure 1.

    • The experiments use only question-answering datasets such as BoolQ. These datasets have relatively short contexts, and the continuation is just yes/no or one of a few candidates. Since these tasks are quite simple, I am not convinced that the method works well with natural context lengths and with more complex and diverse LLM tasks. Specifically, the paper does not test whether translation errors accumulate over long generations.
    • For the cross-scale setting, the paper only studies transfer from a smaller to a larger model. The opposite direction is just as relevant to Figure 1: dynamic routing typically sends easy queries to a smaller, cheaper model, which needs large -> small translation. Moreover, the only cross-scale pair (Qwen3-0.6B -> 1.7B) has the same depth, KV-head count, and head dimension in both models, so it does not test a geometry mismatch.
    • All pairs share the tokenizer, layer count, KV-head count, and head dimension, and all come from the same model family. The main text calls four of them "independently trained pairs" (Table 2 caption), but this is inaccurate for the three Qwen2.5 pairs: Qwen2.5-Coder-7B and Qwen2.5-Math-7B are both continued-pretrained from the Qwen2.5-7B base model.
  2. There is no evaluation on natural long contexts, where skipping prefill matters most. The only experiment beyond 512 tokens is the repeated-context test in Appendix E. Since the context is repeated, the model can find the answer within the last 512 tokens even in the 4k-context setting, so I do not think this experiment shows that the method can handle long contexts. The authors themselves describe it as "not a substitute for natural long-document evaluation" in Appendix E. This gap is central to the paper's claims. The time saved by skipping prefill grows with context length, and the motivating contexts in Figure 1 are often long. However, the cost in Table 5 is measured only on contexts of at most 512 tokens. Even in the easy repeated-context setting, the gap of the cross-scale pair grows from 3.78 pp at 512 tokens to 21.33 pp at 4096 tokens (Table 13). Evaluation on natural long-context benchmarks (e.g., long-document or multi-document QA), together with the translation cost at those lengths, is needed to support the claims.

  3. The efficiency claim (3.41× mean TTFT speedup) is not convincingly supported. The speedup is measured in a setting that may favor KVRelay, and it is unclear whether it holds in practice.

    • The paper describes the baseline cost as "target prefill plus the post-context forward" (Sec. 5.4), which suggests that the baseline is timed as two calls: a prefill of the context, then a forward pass on the query. This matches neither of the natural deployment cases. If the context and the query arrive together (e.g., an agent handoff), a serving system would process them in a single pass; only KVRelay needs two steps. If the context arrives before the query (e.g., stored documents after a model upgrade), the target can prefill the context in advance, so the prefill is not part of the TTFT at all. In that case, the benefit of KVRelay is the lower total compute (translation instead of re-prefilling), not a lower TTFT.
    • The timing uses a static batch of eight requests that are all handed off at the same moment. This matches neither an idle server (batch size 1) nor a loaded server with continuous batching. A larger batch makes the prefill longer, while the query forward and the translator grow little, so the batch size alone can inflate the speedup.
    • Contexts are left-padded (App. F), so the baseline prefill may also process padding tokens. The paper also gives no absolute times or hardware.

    Without a single-pass baseline at batch size 1 (or a realistic serving benchmark) over a range of context lengths, the 3.41× figure in the abstract may overstate the practical gain. Without such a comparison, the motivation to translate caches instead of re-prefilling is weak, since translation requires training a translator for each model pair and loses about 10% of the context gain. Together with Weakness 2, the paper does not yet show a regime where both claims hold. For short contexts, accuracy is shown, but the latency gain is likely small. For long contexts, the latency gain would be large, but accuracy is not shown. If the intended benefit is throughput on a loaded server rather than latency, short contexts may still benefit, but this needs a different evaluation.

  4. Most comparisons use a single seed, and many differences are within the seed noise. Only the main method is run with five seeds (Table 12). The comparisons in Table 2 and all ablations in Tables 3–4 use a single seed. On the cross-scale pair, the seed standard deviation of the main method is 0.66 pp (Table 12).

    • All single-variable ablations in Table 4 use this one pair and one seed. Most differences there are smaller than 1 pp, including the 0.94 pp gain from Value mixing, which is the main empirical support for Sec. 3.5 and Prop. 2. Conclusions such as "access to a short source history matters more than a particular receptive-field length or direction" (Sec. 5.5) therefore need multiple seeds.
    • Please report the mean and standard deviation over several seeds for the ablations, or at least for the differences that the conclusions rely on.
Questions for authors
  • Why does the paper adopt the "base-and-correct" parameterization? I do not see why this design is necessary. Isn't the learnable residual correction alone enough to learn the mapping? The correction is a full affine map over all heads, so the variant without the base can in principle represent the base map itself.
  • How is the native baseline in Table 5 implemented, and which deployment scenario does it model? Is the baseline timed as two calls (prefilling C, then a forward pass on the query with the cache) or as one forward pass over [C; Q]? Please report (i) the speedup against a single-pass baseline without padding, at batch size 1 and for several context lengths and (ii) the absolute times (ms) of the prefill, the translation, and the query forward, with the hardware and the inference framework. If the intended benefit is throughput on a loaded server, please also report the throughput and TTFT under continuous batching.

Additional minor points:

  • In Eq. 1, symbols such as B, L_s, H_s, and D_s are used without definition.
  • There are no citations for the models and datasets used in the paper.
  • L.283: "Eq. equation 3" → "Eq. 3".
  • The "legacy" objective in Table 3 is defined only in the caption, as "attention+KL+context contrast". The paper does not define the context-contrast term. Please explain why it is called "legacy".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.