acceptodds
Under review as a conference paper at ICLR 2027

KV Cache Translation across Heterogeneous Large Language Models

14 views

Abstract

Heterogeneous Large Language Model (LLM) systems increasingly share contexts, retrieved evidence, and multi-agent dialogue histories, yet their internal key-value (KV) caches remain model-specific and cannot be reused across architectures. Consequently, each model must repeatedly prefill or store caches for the same context, limiting the scalability of multi-model reasoning and long-context generation. We propose Mixture-of-Translators (MoT), a cache translation framework that maps context KV caches from a source LLM into the cache space of a target LLM. Unlike prior approaches that depend on a single projection path or global shared latent space, MoT uses multiple translator modules with token-level routing to capture diverse cache translations. We further introduce a Context Correction Loss that aligns the replayed target trajectory with the native target trajectory, reducing residual translation error. Our analysis reveals two competing failure modes: propagation error from early injection and correction-deficit error from late injection. We evaluate heterogeneous translation among Llama, Gemma, and Qwen, spanning substantially different architectures and KV-cache spaces. MoT achieves an average accuracy of 57.6% and F1 of 0.42. These scores recover 95.0% and 120.6% of native target performance and reach 1.5× and 3.5× the strongest-baseline scores, respectively. In practical case studies, MoT enables accurate heterogeneous-agent discussion with scale-invariant KV-memory behavior as the number of agents grows, while retaining near-native generation quality in long-context cache-augmented generation. These results demonstrate scalable KV-cache reuse across heterogeneous LLMs. Code and pretrained translator checkpoints are available at https://anonymous.4open.science/r/MoT-D7E3 and https://anonymous-hf.com/a/tkelnedcist7, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Translation, Heterogeneous Large Language Models, Model Communication, Multi-Agent Systems, Agent Scalability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UA1aOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Since KV-caches cannot be reused across architectures, each model has to prefill or store caches for identical contexts. This work proposes MoT, a mixture-of-translators architecture, which dynamically assembles multiple translation modules at the token level. A context-correction loss is further introduced to enable KV-cache reuse among heterogeneous large language models while preserving downstream task performance as much as possible.

Critical strengths
  • MoT addresses KV‑cache reuse across heterogeneous large language models. Combined with the Free cache‑eviction policy in multi‑agent settings, MoT reduces normalized KV‑cache overhead while retaining decent downstream performance. Furthermore, theoretical analysis of competing errors together with ablation studies on routing selection and context‑correction loss interpret the rationality of design choices and the contribution of individual components.

  • The paper assesses MoT from multiple perspectives via architectural ablations, context‑correction loss ablations, baseline comparisons on two QA datasets, multi‑agent reasoning experiments and long‑context CAG experiments. Ablation studies analyze contributions of different components and their combinations; further experiments examine performance under various task settings. Results support the effectiveness of MoT and reveal the inherent trade‑off between task quality and inference latency.

Critical weaknesses
  • Although the paper aims to reduce redundant computation through KV-cache reuse, the proposed method still requires target-side context replay. The computational overhead of KV translation and context replay is not sufficiently analyzed, making the actual end-to-end efficiency gains unclear.

  • MoT introduces additional translator capacity and supervision compared with existing methods. However, the experiments lack sufficiently controlled comparisons under matched parameter counts, training objectives, and computational budgets, making it difficult to isolate the contributions of the proposed architecture.

  • In the heterogeneous multi-agent experiments, MoT uses the Free cache-management strategy, while competing methods use Retain. Consequently, the reported memory savings may partly result from different cache-management policies rather than the proposed KV-cache translation method itself.

  • Although the paper identifies propagation error and correction-deficit error, the theoretical analysis does not directly justify the effectiveness of the MoT architecture. Moreover, the experimental findings suggest that performance improvements are not primarily attributable to reducing the correction-deficit coefficient, weakening the connection between the theoretical motivation and the proposed approach.

Questions for authors
  • When multiple distinct LLMbased agents coexist within a multiagent system, does the MoT framework require training a separate translator for every sourcetarget pair?
  • For the observed Ushaped validationloss and competingerror phenomenon: does it stably occur across all evaluated QA datasets or only within specific subsets? Are there any discussions or experiments in the manuscript supporting its generality across different model pairs and diverse tasks?
Reviewer_E7neOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MoT maps a source LLM's context KV cache into a target LLM's cache for a window of depth-ratio-matched layers, using top-K routing over several cross-attention translators. It then replays the context through the target with source-guided sparse attention to fill the other layers. Training uses a KV-level Context Correction loss plus a prompt LM loss, on 64-token contexts for 500 steps. On the six cross-family pairs among Llama-3.2-3B, Gemma-3-4B and Qwen-3-4B, MoT reports 57.6% average accuracy and 0.42 average F1, stated as 95.0% and 120.6% of native and 1.5x and 3.5x the strongest baselien.

Critical strengths
  1. The problem identified is real and the evaluation is broad.
Critical weaknesses
  1. I cannot tell whether the translated cache carries the quality. Translation fills at most about 30% of target layers, and the rest are recomputed by native replay over the context tokens. F1 drops as more layers are translated. MoT beats native F1 on 4 of 6 cross-family pairs (e.g., Gemma→Qwen 0.41 vs 0.24), and Llama→Llama beats native (0.53 vs 0.44) at CosineSim 1.00, so the two paths seem to differ in more than the cache. The native reference is also shaky (Gemma-3: 23.5 on four-way MMLU-Redux, below chance). There is no control that keeps the replay and removes the translator, so neither 95.0%/120.6% nor 1.5x/3.5x measures translation.
  2. The efficiency story does not hold on the paper's own data. In CAG, MoT's TTFT is above Native at every budget (Fig. 14b), and Native F1 is slightly higher (0.21 vs 0.20). In Table 9, peak memory is 38.00 GiB vs 23.78 GiB for Interlat, and the KV/1K advantage comes from running MoT in "Free" mode while all baselines use "Retain". The 0.70 vs 0.65 accuracy gap is one question out of 20.
Questions for authors
  1. Can you add a replay-only control to Table 1, with the translated window filled by native replay, zeros, or an untrained translator?
  2. Against a plain target prefill of the same context, what are MoT's end-to-end TTFT, FLOPs and peak memory, with translator weights?
  3. How are source and target tokens aligned when the tokenizers differ, for example Llama→Gemma?
Reviewer_EiSAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV-cache reuse across heterogeneous language models. It proposes Mixture-of-Translators (MoT), which routes each token through several cache translators, and a Context Correction Loss that encourages the replayed target trajectory to follow the native one. Experiments across several model families show clear gains over the compared baselines. The paper also includes case studies intended to demonstrate the practical value of cache translation.

Critical strengths
  • Reusing context states across genuinely different model architectures is considerably harder than ordinary prefix caching, and this setting is not yet well covered by existing systems.
  • The empirical results are strong across several model pairs and QA tasks. The ablations and case studies also help connect cache reconstruction quality to downstream behavior.
  • The paper is generally well organized and easy to follow at a high level. The distinction between propagation error and insufficient correction provides a useful way to think about where cache translation fails.
Critical weaknesses
  • The architectural novelty of MoT is moderate. Its backbone follows the recurrent cross-attention design of prior work, and the main new component is token-level Top-K routing. This is a reasonable design, but the current ablations do not clearly establish the value of the mixture in the main cross-family setting. Table 8 shows only small and inconsistent changes when more translators are added, while the routing comparison in Table 2 is limited to one Qwen2.5 model pair.
  • The experiments do not cleanly identify where the large improvement comes from. Figure 12 suggests that CC+PL already helps substantially with a single translator. Meanwhile, MoT uses 2.19B translator parameters, far more than Interlat or C2C-Project, and there is no parameter-matched single-translator control using the same loss and replay procedure.
  • The cross-family formulation leaves token alignment undefined. The method is written as if the source and target caches shared the same token positions, but Llama, Gemma, and Qwen can tokenize the same context into sequences of different lengths. It is therefore unclear how the translated sequence is aligned with the native target cache for the correction loss, or how source attention positions are reused during target replay. This is a basic missing piece of the method rather than a minor implementation detail.
Questions for authors

See Weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.