acceptodds
Under review as a conference paper at ICLR 2027

Training-Free Latent Memory Transfer Across Heterogeneous Language Models

26 views

Abstract

Multi-model LLM systems share instructions, evidence, and interaction history across model families, yet each model keeps that context in a private KV cache. Handing it over as text forces the target to prefill the shared context again, repeating computation on every handoff. Direct cache transfer is hard: models differ in tokenization, positions, layer and head layout, and channel bases, and the target’s attention weights each cache error in a query-dependent way. A transfer must preserve the target’s computation, not merely match cache values. We present TACIT (Target-Attention-Guided Closed-Form Inter-Model Transfer), the first training-free method for KV-cache transfer across heterogeneous model architectures. TACIT aligns source and target tokens and positions, uses target attention sensitivity to select source layers, and fits per-head K/V maps in closed form. One frozen mapper is fitted per source–target direction from paired unlabeled prefills, then reused across requests and tasks so the target continues from transferred memory. Across five model families, 13 cross-family directions, and 560M–32B models, TACIT retains 83.33% of target-native accuracy on six multiple-choice tasks and outperforms all evaluated transfer baselines on every task. Cache handoff averages 6.13 ms, 6.77× faster than these baselines. On 1K–32K prefixes, end-to-end time-to-first-token improves by 1.52×–10.48× while retaining 92.4–95.4% of next-token accuracy. Analysis shows why: attention-guided selection preserves the target’s key routing, and sensitivity-weighted fitting preserves its value readout, turning model-private KV state into a reusable latent-memory interface between heterogeneous models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache transfer, Latent memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vh2dOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TACIT converts an existing source model's KV cache into a cache usable by a different target model. It combines causal token alignment and positional conversion with attention-guided layer selection and closed-form affine fitting. A direction-specific mapper is calibrated once on paired unlabeled prefills and then reused. The evaluation covers heterogeneous model pairs, multiple-choice tasks, continuation, and question answering, with substantial reductions in downstream handoff latency.

Critical strengths
  • Transferring between these model families is a useful result in itself. Different token boundaries and positional schemes make this substantially harder than reuse between related checkpoints, and the breadth study includes several genuinely different cache interfaces.

  • Using attention sensitivity to weight reconstruction errors is well motivated. The separate treatment of keys and values leads to a tractable closed-form fit, and the derivation explains what the approximation retains and leaves out.

  • The crossed ablation of layer selection and fitting is the most persuasive part of the analysis. Mapper capacity stays fixed, while the weight-shuffling control helps show why assigning importance to the right cache positions matters.

  • The method offers a useful operating point when an upstream model has already read the context. Handoff is fast, the mapper is reused across tasks, and its construction is cheaper than training the evaluated translators. The full construction measurements are helpful here.

Critical weaknesses
  • I could not tell how much generation quality is lost relative to the target model. Table 4 omits native WikiText perplexity and SQuAD accuracy, and LCF actually has better perplexity than TACIT. These references are needed before concluding that the faster handoff preserves useful generation quality.

  • The benchmark coverage is narrow for the proposed shared-memory interface. C2C evaluates knowledge and reasoning on MMLU-Redux and ARC-Challenge, while the closed-form transfer study includes multi-turn CoQA. TACIT should cover comparable reasoning and multi-turn use, where specific information must survive the handoff; teacher-forced continuation alone does not establish this.

  • The abstract makes the results sound broader than the tables show. Six-task retention is measured on six directions; thirteen directions are tested only on HellaSwag. Several individual pairs also favor a learned baseline, despite TACIT winning every task average. A reader choosing one transfer direction needs this distinction.

  • The contrast with previous objectives is overstated. C2C already trains through next-token prediction, rather than only matching cache coordinates. The contribution is more specifically a closed-form attention surrogate and heterogeneous transfer pipeline; the current framing makes the conceptual departure look larger than the comparison establishes.

Questions for authors
  1. Would replaying a longer tail repair the weaker predictions immediately after handoff? Table C.2 looks less favorable than the longer continuation average. I would like to see how much quality can be recovered for the additional replay cost.

  2. Were the learned baselines given the same batched cache-delivery optimization as TACIT? Separating conversion from delivery time would help interpret how much of the reported handoff speedup comes from the mapper itself.

  3. How should the proposed routing and readout explanation be reconciled with the large gains from value-only Selection? The interventions suggest that Selection and Fit both influence readout, rather than assigning each component one distinct role.

  4. How often does causal alignment reuse one source row for several target tokens, and how does transfer quality vary with this mismatch? I would be interested in whether the weakest transfer directions are also the ones losing the most token-specific information.

Reviewer_7Y4UOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper is about how to use KV-cache across different types of the models when the models do not share common vocabulary or architecture. In the current multi-model LLM systems they need to each process the context separately however in many cases the context is identical among them when working on same tasks/problems. However, direct KV cache transfer is not possible. This paper propose TACIT (Target-Attention-Guided Closed-Form Inter-Model Transfer) to address KV-cache transfer across heterogeneous models.

Critical strengths

1- It targets a very useful problem 2- First training-free KV-cache transfer across heterogeneous architectures 3- Wide range of experiments

Critical weaknesses

1- the paper's own appendix contradicts two of the abstract's three main claims Apparently the method appears to recover roughly two thirds of the information gap at a cost that is only favourable under an amortisation assumption that is asserted rather than measured.
(a) "Retains 83.33% of target-native accuracy" is a ratio with no floor subtracted. All six benchmarks are multiple-choice with chance floors of 20–50% (Table B.1). Every method's Ret. column in Table 3 implies the same target-native mean of 66.42% (55.35/0.8333 = 54.49/0.8204 = 53.31/0.8026 = 48.40/0.7287), so against the mean chance floor of 32.5% TACIT recovers (55.35−32.5)/(66.42−32.5) = 67.4% of the achievable range, not 83.3%.

(b) "Outperforms all evaluated transfer baselines on every task" in the abstract is true only of six-direction averages and is false per direction. Table C.1 shows, for instance, BLOOM-560M to Mistral-7B HellaSwag: TACIT 44.03 vs LCF 50.67, MoT 50.03, C2C 48.26. In SmolLM3-3B to Mistral-7B, TACIT loses or ties on five of six tasks (CICERO 29.41 vs MoT 36.55 / LCF 36.61; SWAG 66.94 vs MoT 67.59; PIQA 73.94 vs MoT 75.14 / LCF 74.97; Story Cloze 59.01 vs LCF 60.73 / MoT 59.58; ToMATO tied with LCF at 47.86).

2- The main results in Table 1 are shown in only one simple multichoice task (Hellaswag). It is difficult to conclude the effectiveness of their solution only based on one simple task for cross family transfer. I highly encourage the authors to report more tasks from lm harness benchmark and also generative tasks such as GSM8K, Math500. Also Table 3 talks about multichoice tasks. Please report generative tasks.

3- The mechanistic story that the paper advertises as its explanatory contribution is contradicted by its own intervention tables. For example in the abstract we have "attention-guided selection preserves the target's key routing, and sensitivity-weighted fitting preserves its value readout." Table C.12 tests exactly this and reports the opposite for the first half: applying TACIT Selection to the K branch across all layers gives +1.17 pp, p = 0.392 (which is not significant) while the V branch gives +27.15 pp and early layers K+V give +27.34 pp. Selection is, by the paper's own measurement, an early-layer value effect.

Questions for authors

1- For each prefix length in Table 2/C.6, what is the target's accuracy and PPL if it natively prefills only the final T tokens of the shared text, where T is chosen so that native prefill time equals TACIT's TTFT (≈21.6 / 35.7 / 91.3 / 168.2 ms)? If TACIT does not beat that, what is the intended deployment regime?

2- Please report the Table 3 numbers with a no-transfer control (target sees only the task suffix), as you already do for continuation in Table C.2, and report floor-normalised retention alongside the ratio. Does the ordering against C2C/MoT/LCF survive that normalisation?

3- Table C.12 reports the K-branch Selection intervention at +1.17 pp, p = 0.392, while V-branch gives +27.15. How do you reconcile this with the claim at l.028–029 and l.482–483 that Selection preserves key routing? Is there a measurement I have misread, or should the claim be restated as a value-branch effect?

4- Given Table C.1, can you restate the abstract's "outperforms all evaluated transfer baselines on every task" as a per-direction win rate across the 36 direction×task cells, with the losing cells named?

5- Table 4 reports 17.48 EM / 31.29 F1 on SQuAD v1.1 with no target-native row. A 3B–8B model reading the gold paragraph natively should be far higher. What is the native EM/F1 under the identical protocol, and what is the no-context floor?

6- Is the method training free? Each mapper contains 67–269M fitted parameters (Table C.5, 256–1024 MiB in FP32) estimated by ridge regression on 12,288 rows from 96 FineWiki documents per direction. Would you consider "backpropagation-free" or "closed-form-fitted" more accurate?

Reviewer_sou4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Method. This paper proposes TACIT, a training-free method for transferring KV caches between heterogeneous language models without repeating the target model's prefill. It aligns tokenizations, handles positional encodings, selects source layers using attention-weighted CKA, and fits per-head affine K/V maps using weighted ridge regression. The weights approximate the sensitivity of target attention outputs to errors in the transferred cache.

Experimental setup. The authors evaluate TACIT across 10 models (560M–32B) from five model families and 13 cross-family transfer directions. They compare against four learned KV-cache translators and native target prefill. The evaluation includes six multiple-choice benchmarks, WikiText-103 language modeling, SQuAD question answering, and continuation quality and latency at context lengths up to 32K.

Results. TACIT achieves 55.35% mean accuracy across the six multiple-choice benchmarks, retaining 83.33% of native target accuracy, compared with 54.49% for the strongest evaluated baseline, LCF. Mean cache-handoff latency decreases from 21.91 ms to 6.13 ms. On Qwen3-8B → Llama-3.1-8B, target-side time-to-first-token improves by 1.52x–10.48x for 1K–32K prefixes, while retaining 92.4%–95.4% of native next-token accuracy.

Critical strengths
  1. Attention-weighted fitting with useful ablations. The paper proposes a simple objective that weights cache reconstruction errors by their effect on target attention outputs while retaining a closed-form affine mapping. With unweighted CKA selection held fixed, attention-weighted fitting improves HellaSwag accuracy by 5.86–20.90 percentage points across four directions (Table 6). The authors also evaluate alternative mapper architectures and sensitivity-weight shuffling, providing useful evidence that the attention-based weighting contributes to the improvements.

  2. Cross-family transfer with measured efficiency benefits. TACIT supports differences in tokenization, positional encodings, and KV layouts while keeping both language models frozen. Table 3 reports 55.35% mean accuracy and 6.13 ms mean handoff latency, compared with 54.49% and 21.91 ms for LCF. The target-side timing results also show increasing savings with prefix length when the source cache is already available. These are useful efficiency improvements, even though the average accuracy gain over the strongest baseline is modest.

Critical weaknesses
  1. Incomplete/unclear baseline comparisons. Two aspects of the baseline evaluation remain unclear.

    • Layer selection. Heo et al. [1] already propose per-head affine regression with source-layer selection and RoPE removal. Their selection criterion uses predictive R² to measure how well source representations reconstruct the target cache. Table 6 shows that, with attention-weighted fitting enabled, weighted CKA selection improves accuracy by less than one percentage point in three of four directions, although it provides a 10-point gain in the fourth. Comparing against predictive R² selection under the same calibration data, source-layer budget, and fitting procedure would help establish the contribution of weighted CKA.

    • Learned baselines. C2C and LCF originally use both source and receiver caches, whereas TACIT avoids computing the receiver cache for the shared prefix. The paper does not explain how these baselines are adapted to this setting, or whether they still require target prefill. This distinction is important for interpreting the accuracy and latency comparisons in Table 3.

  2. Limited evaluation of downstream and long-context capabilities. The existing results are encouraging, but it remains unclear how well TACIT preserves the target model's capabilities relative to native prefill.

    • Standard LM evaluation: The six-task comparison primarily covers commonsense and dialogue tasks, while Table 4 reports WikiText-103 perplexity and SQuAD results. A broader evaluation including common sense language modeling, held-out validation perplexity, and real-world in-context retrieval at 2K would help characterize the quality retained after transfer. Reporting native target performance alongside TACIT is important for interpreting the degradation.

    • Long-context evaluation: The 1K–32K experiment evaluates teacher-forced continuations on six documents per length for one model pair (Table 2). Table C.2 includes useful early-token and no-history controls, but the paper does not evaluate long-document loss versus token position or retrieval across context lengths. These experiments would help establish how well transferred caches preserve the benefits of longer contexts.

Questions for authors
  1. Comparison with predictive source-layer selection.

    Could the authors compare TACIT's attention-weighted CKA selection with the predictive-R² layer selection proposed by Heo et al. [1]? In particular, I would like to see the following comparisons on the transfer directions in Table 6:

    • Compare predictive-R², unweighted CKA, and attention-weighted CKA under the same calibration data, source-layer budget (k=2), token alignment, and replay protocol.
    • Evaluate each selection method with both uniform and attention-weighted affine fitting, keeping the mapper architecture and fitting procedure unchanged.

    This would help establish whether attention-weighted CKA provides additional benefits over selecting layers based on their ability to reconstruct the target cache, independently of the attention-weighted fitting objective.

  2. Implementation of the learned baselines.

    Could the authors clarify how C2C and LCF were adapted to the proposed setting? In particular, both methods originally use the receiver model's cache, whereas TACIT aims to avoid computing the target cache for the shared prefix.

    • Do the baseline implementations receive a native target KV cache for the shared prefix? If so, is the computation required to construct this cache included in the reported latency? If not, what replaces this input, and how are the original architectures modified?
    • Please specify the training objectives, number of optimization steps, and checkpoint-selection criteria used for the learned baselines.

    These details are important for interpreting both the accuracy and latency comparisons in Table 3.

  3. How well does TACIT preserve standard language model capabilities?

    Could the authors extend the evaluation on representative cross-family model pairs to include the following?

    • LM benchmarks: Evaluate the standard LM-7 tasks (LAMBADA, HellaSwag, PIQA, ARC-Easy, ARC-Challenge, WinoGrande, OpenBookQA).
    • Language modeling: Report held-out validation perplexity on representative text domains, including native-target perplexity for the existing WikiText-103 experiment.
    • Real-world retrieval: Evaluate in-context retrieval and information extraction tasks at approximately 2K context length for SWDE, FDA, Drop, NQ, TQ, SQuad.

    For each evaluation, compare TACIT with the same target checkpoint using native prefill, keeping prompts, context, and scoring protocols fixed. Report absolute performance and degradation relative to native prefill.

  4. Long-context language modeling and retrieval performance.

    Table 2 evaluates teacher-forced continuations from six documents per context length on one model pair. How well does TACIT preserve the target model's ability to use information from longer contexts? I would be interested in the following experiments:

    • Language modeling: Evaluate held-out long-document perplexity across context lengths (e.g., 1K, 4K, 8K, 16K, 32K), including loss versus token position (as in [2]). Compare TACIT against native target prefill using the same prefix/continuation splits and scoring the same tokens.
    • Retrieval: Evaluate NIAH/RULER tasks, including single-key, multi-key, multi-value, and multi-query variants, across context lengths and evidence positions. Selected LongBench tasks would provide a complementary downstream evaluation.

    I would particularly like to see performance relative to native prefill as context length increases. The target checkpoint should perform meaningfully on the evaluated tasks, so that any degradation can be attributed to cache transfer rather than limitations of the underlying model.

References

[1] Heo et al. Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse. 2026. https://arxiv.org/abs/2608.03893

[2] Tandon et al. End-to-End Test-Time Training for Long Context. 2025. https://arxiv.org/abs/2512.23675

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.