acceptodds
Under review as a conference paper at ICLR 2027

Prefill-Free Cross-Family KV Cache Transfer for Heterogeneous Multi-Agent LLMs

21 views

Abstract

Recent multi-agent LLM systems increasingly combine heterogeneous models for specialized agent roles. However, text-based communication requires each receiver to prefill shared context already processed by the sender. Reusing the sender's key-value (KV) cache avoids this redundancy, but prefill-free transfer across model families must handle differences in tokenization, model depth, and KV representations. To address these issues, we propose HeteroFold, a prefill-free cross-family KV cache transfer method that keeps both the sender and receiver frozen. HeteroFold aligns model structures, maps the sender cache into the receiver space, and calibrates it to preserve receiver behavior. Across six transfer directions, HeteroFold achieves the best cache-transfer performance on all four long-context benchmarks and most short-context settings. It also matches text-based communication on the multi-agent benchmark. At 32K context length, Llama-3.1-8BMinistral-3-14B transfer is faster than Native Prefill and – faster than the state-of-the-art prefill-free baselines, Dense Latent and KV Ridge. These results show that HeteroFold enables efficient cross-family KV reuse without receiver prefill.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV transfer, Cross-Family, Multi-Agent Systems, Efficient Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_a4jFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prefill-free KV-cache transfer between models from different families, where the sender and receiver differ in tokenizer, depth, and KV structure. The proposed method, HeteroFold, aligns tokens by shared character boundaries (TA), concatenates three sender layers around the proportionally matched depth, maps K/V with a moment-matching affine map (Recolor), and adds a low-rank correction trained to match native receiver attention and outputs. The correction is folded into the affine map, so inference uses one fixed map per layer and role.

Across six directions among Llama-3.1-8B, Qwen3-4B, and Ministral-3-14B, HeteroFold outperforms Dense Latent and KV Ridge (with and without TA) on all long-context benchmarks and most short-context ones. At 32K context, transfer is reported to be 10.7× faster than native receiver prefill, including inter-GPU payload transfer over NVLink.

Critical strengths
  • The cross-family setting seems new and worth studying. Existing prefill-free methods are evaluated only within the same model family, and cross-family methods such as C2C still require receiver prefill. Handling both together fits well with heterogeneous multi-agent systems.

  • The paper clearly shows why tokenizer mismatch matters. Without TA, KV Ridge and Dense Latent drop to almost zero on long-context tasks, and same-index pairing is off by hundreds of characters at 32K. TA is a simple rule, but these results make its necessity convincing.

  • Even when the baselines are given the same TA, HeteroFold is consistently better on long-context QA in all six directions, sometimes by a large margin (e.g., HotpotQA 17.1 → 42.7 for Qwen→Ministral). Similar gains in the same-family setting suggest that the improvement does not come only from TA.

  • The latency evaluation seems fair. Sender and receiver are on separate GPUs, KV transfer over NVLink is included, and contexts up to 32K are tested. Sender prefill is also reported separately, and the method is still faster than native prefill at 32K even when it is added.

Critical weaknesses
  1. Novelty of the components. The setting is new, but the individual parts (character-offset alignment, whitening/coloring, output-aware calibration from KV quantization) are fairly standard. Character-offset alignment is also used in cross-tokenizer distillation, which is not discussed.

  2. Gain over KV Ridge + TA. Most of the gap to the original baselines seems to come from TA. Against KV Ridge + TA, the gain is clear on long-context tasks, but small or mixed on short-context ones.

The ablation is only for one direction, and its TA + Ridge row (e.g., 4.55 on GSM8K) does not match KV Ridge + TA in Table 1 (53.30). It is not clear why. Also, TA + Recolor alone already beats KV Ridge + TA there, so the role of calibration seems smaller than suggested.

  1. Long-context accuracy. Speedup is reported at 32K, but accuracy is not reported by length. The gap to TextMas is also still large in several cases, especially on GSM8K.

  2. Evaluation. No seed variance, and Qasper/HotpotQA use 200-item subsets. HotpotQA is part of the calibration data. HiddenBench has only 65 tasks, so the claim of matching text communication seems a bit strong. Calibration-corpus sensitivity and latency are each tested in a limited setting (one direction, NVLink only, batch 2).

Questions for authors
  1. Why is “TA + Ridge” in Table 3 so different from KV Ridge + TA in Table 1 for the same direction? Could the ablation be shown for at least one more direction?
  2. How does accuracy change at 16K and 32K, where the main speedup is reported?
  3. How is TA different from character-offset alignment used in cross-tokenizer distillation?
  4. Could the authors report seed variance for Qasper, HotpotQA, and HiddenBench?
  5. How would the latency change with PCIe or network transfer instead of NVLink?
6 comments
AuthorsOpenReview ↗

Response to Reviewer a4jF

We thank the reviewer for their constructive feedback. We appreciate their recognition of the novelty of cross-family KV transfer, the importance of token alignment, and our consistent improvements over baselines. We also appreciate their positive assessment of our latency evaluation. We address the concerns below with additional analysis and experimental results.

Response to Question 1: Difference Between TA + Ridge and KV Ridge + TA

Q1. Why is “TA + Ridge” in Table 3 so different from KV Ridge + TA in Table 1 for the same direction?

A1. The two methods use different mapping architectures, despite both employing Ridge Regression. In Table 3, "Ridge" refers to standard ℓ2\ell_2 regularized least squares, used to replace Recolor within our HeteroFold framework. In contrast, "KV Ridge" refers to the complete method proposed by Heo et al. (2026), which uses a different layer selection and mapping strategy.

Specifically, TA + Ridge (Table 3) retains our three sender layer aggregation and cross head mapping, replacing only Recolor with Ridge Regression. The reported row does not include output aware calibration, which is evaluated separately.

In contrast, KV Ridge + TA (Table 1) follows the original KV Ridge method, selecting eight sender layers based on calibration R2R^2 and fitting separate Ridge mappings for each KV head. Only Token Alignment (TA) is added.

Thus, the performance difference arises from their distinct layer selection, mapping architectures, and calibration settings, rather than inconsistent evaluations. We will clarify this distinction in Table 3 to avoid confusion.

Response to Question 2: Accuracy at 16K and 32K

Q2. How does accuracy change at 16K and 32K, where the main speedup is reported?

A2. We additionally evaluate accuracy at 16K and 32K context lengths using 100 QuALITY questions per length for Llama-3.1-8B → Ministral-3-14B transfer, matching the context lengths used in our latency experiments.

Table R1. Accuracy (%) at 16K and 32K context lengths.

Method16K Acc. (%) ↑32K Acc. (%) ↑
Native Prefill81.0080.00
Dense Latent29.0029.00
Dense Latent + TA35.0027.00
KV Ridge34.0017.00
KV Ridge + TA62.0063.00
HeteroFold (Ours)67.0067.00

HeteroFold achieves the highest accuracy among prefill-free methods at both context lengths, maintaining 67% accuracy from 16K to 32K. Notably, incorporating our Token Alignment (TA) improves KV Ridge by 28 and 46 percentage points, respectively, highlighting the importance of token correspondence across different model families. HeteroFold further improves upon KV Ridge + TA, demonstrating the benefits of its KV mapping and receiver-aware calibration.

Response to Question 3: Token Alignment vs. Cross-Tokenizer Distillation

Q3. How is TA different from character-offset alignment used in cross-tokenizer distillation?

A3. We acknowledge that character-offset alignment has been explored in cross-tokenizer distillation and do not claim the alignment principle itself as novel. However, our Token Alignment (TA) addresses a different problem: constructing receiver KV caches without native prefill.

While cross-tokenizer distillation aligns text spans to provide training supervision (Nguyen et al., 2026), our TA establishes causal correspondence between sender KV states and receiver token positions. Specifically, unmatched receiver positions reuse sender KV states from the latest preceding shared character boundary, allowing direct cache construction without receiver prefill.

Our contribution lies in adapting token alignment to prefill-free cross-family KV transfer, together with representation mapping and receiver-aware calibration. We will clarify this distinction and acknowledge the related literature.

References

AuthorsOpenReview ↗

Response to Reviewer a4jF

Response to Question 4: Seed Variance and Calibration Without HotpotQA

Q4. Could the authors report seed variance for Qasper, HotpotQA, and HiddenBench?

A4. We additionally evaluated Qasper and HotpotQA with three decoding seeds in two transfer directions. To address the calibration-data concern, we also replaced HotpotQA with MuSiQue and trained all three methods using 800 Open-R1 and 800 MuSiQue examples. Results across all nine benchmarks are reported below.

Table R2. Long-context results without HotpotQA calibration examples.

Transfer DirectionMethodQasper F1 ↑HotpotQA F1 ↑LoCoMo F1 ↑QuALITY Acc. ↑
Ministral → LlamaDense Latent + TA3.25 ± 0.432.99 ± 0.042.1941.51
KV Ridge + TA16.68 ± 0.6631.27 ± 1.0431.0271.38
HeteroFold32.32 ± 1.6948.19 ± 0.6344.9079.10
Qwen → MinistralDense Latent + TA3.75 ± 0.182.87 ± 0.302.2328.95
KV Ridge + TA9.33 ± 0.3731.10 ± 0.5410.9064.53
HeteroFold19.08 ± 0.4544.35 ± 1.0023.1564.53

Scores use a 0–100 scale. Qasper and HotpotQA each use the same 200 evaluation questions across decoding seeds 1001, 2001, and 3001. We report mean ± sample standard deviation with fixed trained maps.

Table R3. Short-context accuracy (%) without HotpotQA calibration examples.

Transfer DirectionMethodARC-C ↑MMLU ↑WinoGrande ↑HellaSwag ↑GSM8K ↑
Ministral → LlamaDense Latent + TA56.5735.1350.8349.899.40
KV Ridge + TA88.4868.5853.3565.6554.74
HeteroFold90.0271.2363.6169.7664.44
Qwen → MinistralDense Latent + TA30.3833.5850.4350.7821.91
KV Ridge + TA84.4763.6952.3362.1255.72
HeteroFold85.1565.6556.9165.4692.72

HeteroFold retains the highest score across all nine benchmarks in both directions without HotpotQA calibration examples. Its advantage over both baselines holds at every evaluated decoding seed on Qasper and HotpotQA. On HotpotQA, its mean improvements over KV Ridge + TA are 16.92 and 13.25 F1 points, respectively.

Response to Question 5: Latency on PCIe GPUs

Q5. How would the latency change with PCIe or network transfer instead of NVLink?

A5. We additionally evaluate HeteroFold on two H100 GPUs connected via PCIe instead of NVLink, including inter GPU KV cache transfer. Although absolute latency increases, HeteroFold maintains substantial speedups over native receiver prefill.

Table R4. 32K transfer latency on PCIe GPUs (ms ↓, speedup in parentheses ↑).

MethodLlama → MinistralQwen → Ministral
Native Prefill8083.5 (1.00×)8137.9 (1.00×)
Dense Latent + TA1621.8 (4.98×)1703.3 (4.78×)
KV Ridge + TA1467.4 (5.51×)1552.0 (5.24×)
HeteroFold (Ours)1035.5 (7.81×)1119.4 (7.27×)

Measured on two H100 GPUs within the same server using PCIe, including KV data transfer.

HeteroFold achieves the lowest latency in both directions, retaining 7.81× and 7.27× speedups over native receiver prefill. These results demonstrate that HeteroFold's efficiency benefits extend beyond NVLink environments, although cross server network performance depends on available communication bandwidth.

AuthorsOpenReview ↗

Response to Reviewer a4jF

We hope our responses have addressed the reviewer's key concerns through the following clarifications and additional experiments:

  • Methodological clarity: Clarified the differences between Ridge methods and distinguished our token alignment from prior cross-tokenizer distillation.
  • Long-context performance: Provided new accuracy results at 16K and 32K, demonstrating consistent improvements over existing prefill-free baselines.
  • Evaluation robustness: Confirmed performance gains without HotpotQA calibration data and reported seed variance across multiple transfer directions.
  • Practical efficiency: Demonstrated substantial speedups of 7.81× and 7.27× on PCIe GPUs beyond the original NVLink setting.

We sincerely appreciate the reviewer's constructive feedback. Given these additional results and clarifications that directly address the raised concerns, we would be grateful if the reviewer could reconsider their assessment and consider increasing their rating.

Reviewer_3DZpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces HeteroFold, which enables LLM to reuse another LLM’s KV cache without extra prefill on the receiving side, even when the two models come from different model families or scales, and neither model need to be finetuned.

The proposed method avoids the wasted receiver prefill in heterogeneous multi-agent setups. To bridge the arch differences, it aligns tokens at the endpoints of the two tokenizers, and blends the representations from the sender's layers using a Procrustes moment matching step.

While both LLMs remain frozen, HeteroFold is not training-free because it still trains the adapter between them, the adapter learns small low-rank corrections aimed at the receiver’s outputs, which then are added to fixed affine maps.

After evaluating across six transfer directions, HeteroFold keeps the receiver’s attention patterns and task accuracy much better than baselines like Dense Latent and KV Ridge.

Critical strengths

It's the first doable approach to sharing KV caches across different model families. Previous cache-sharing methods only worked when both models used the same tokenizer, which in practice implies the same model family. This paper addresses this by aligning tokens along shared character boundaries and combining sender representations from several layers. It allows models with different arch and vocab can transfer a cache directly, no prefill on the receiver.

HeteroFold adapt its mapping parameters to keep the receiver’s attention patterns and output distributions directly, and then folds low-rank corrections into static affine maps. This keeps accuracy across long contexts without adding any inference modules or slowing down inference.

Critical weaknesses

It seems that the scaling of HeteroFold is pairwise and unidirectional, which becomes expensive increasely as the number of models grows. Each map is learned for a single direction, so an A →\to B map is useless for B →\to A. With NN models that means training, calibrating, and storing N(N−1)N(N-1) separate adapters. At roughly 2 GPU-hours and 200–250M parameters per direction, adding even one new model to the pool is a real cost.

Token alignment works when both models see the exact same raw text, since it matches on character boundaries and forward-filling. This design fails when the two models format context differently, and using model-specific chat templates, different system instructions, or function calls could break it.

Questions for authors

Why wasn't HeteroFold evaluated on any of the standard interactive agent benchmarks, say, SWE-bench, or AgentBench?

Token alignment only works over identical raw-text spans. What happens when two agents need different system prompts, different role headers (Llama's <|start_header_id|> vs. Qwen's <|im_start|>), or different tool-call syntax wrapped around the shared context? How is the cache transferred in that case?

Once a system grows past three models, say N≥5N \ge 5, how do you plan to deal with the O(N2)O(N^2) cost that comes from every mapping being unidirectional? That's roughly 2 GPU-hours of offline calibration per pair plus 201–252M parameters per pair sitting in GPU memory.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 3DZp

Response to Question 1: Evaluation on Interactive Agent Benchmarks

Q. Why wasn't HeteroFold evaluated on any of the standard interactive agent benchmarks, say, SWE-bench, or AgentBench?

A. To address this concern, we additionally evaluate HeteroFold on ALFWorld, an interactive agent benchmark, using all 134 valid_unseen tasks. HeteroFold achieves 74.6% success, approaching native text-based communication (76.9%) while outperforming existing KV transfer baselines.

Experimental Setup. We construct a heterogeneous three-agent system with distinct roles:

  • Tracker (Llama-3.1-8B): Maintains the full interaction history and communicates relevant information.
  • Planner (Ministral-3-14B): Determines the next action based on the available information.
  • Actor (Qwen3-4B): Executes actions in the ALFWorld environment.

Agents communicate at each interaction step. We vary only the communication method, comparing native text prefill with KV-cache transfer. All methods use identical prompts and evaluation settings, with a maximum of 50 steps per task. We reuse the original mappings from Table 1 without any ALFWorld-specific calibration or fine-tuning.

Table R. ALFWorld success rate (%) across all 134 unseen tasks.

Communication MethodSuccess Rate (%) ↑
Native Prefill (TextMas)76.9
Dense Latent29.1
Dense Latent + TA39.1
KV Ridge29.1
KV Ridge + TA65.1
HeteroFold (Ours)74.6

HeteroFold achieves the highest success rate among prefill-free methods, only 2.3 percentage points below native communication and 9.5 points above KV Ridge + TA. These results demonstrate that HeteroFold can effectively support interactive multi-agent collaboration without receiver prefill of transferred messages, even on tasks unseen during calibration.

Response to Question 2: Model-Specific Prompts and Token Alignment

Q. Token alignment only works over identical raw-text spans. What happens when two agents need different system prompts, different role headers, or different tool-call syntax wrapped around the shared context? How is the cache transferred in that case?

A. Token Alignment (TA) applies to the shared raw-text span, not to model-specific chat-template tokens. The receiver processes its own prefix and suffix natively.

The receiver constructs its KV cache in three parts:

  1. Native Prefix: The receiver prefills the system prompt and role headers preceding the shared span, using its native chat template.
  2. Transferred Context: Mapped sender KV states represent the shared span without receiver prefill. In our evaluation, this span includes the question.
  3. Native Suffix: The receiver processes its closing delimiters and assistant header after the transferred span, attending to the assembled cache before generating an answer. The question is not prefilled separately.

HeteroFold maps keys before RoPE. The receiver applies its own RoPE at positions offset by its native prefix length.

Tool-result text could use the same separation, with model-specific syntax handled natively. Our evaluation covers a single shared span; interleaved tool contexts and arbitrary sender prompt prefixes remain future work.

AuthorsOpenReview ↗

Response to Reviewer 3DZp

Response to Question 3: Scalability Beyond Pairwise Mapping

Q. Once a system grows past three models, say N ≥ 5, how do you plan to deal with the O(N²) cost that comes from every mapping being unidirectional?

A. We acknowledge that pairwise KV mapping introduces O(N²) scaling when all directed model pairs are required. However, this limitation is shared by existing KV transfer methods, including KV Ridge and Dense Latent. In practice, fewer mappings are needed when agents reuse the same model checkpoints or communicate only between a coordinator and its workers.

Importantly, HeteroFold achieves higher transfer accuracy with fewer mapping parameters than existing approaches, as shown below.

Table R8. Mapping parameters (millions) across six transfer directions (Table F).

Transfer DirectionDense LatentKV RidgeHeteroFold
Llama → Qwen303604227
Qwen → Llama270537201
Ministral → Qwen303604227
Qwen → Ministral337671252
Ministral → Llama270537201
Llama → Ministral337671252
Average303.3604.0226.7

Table R9. Average accuracy and F1 across six cross-family transfer directions (Table 1).

MethodAvg. Acc. (%) ↑Avg. F1 (%) ↑
Dense Latent30.431.54
Dense Latent + TA34.273.08
KV Ridge30.651.42
KV Ridge + TA62.6923.01
HeteroFold (Ours)67.7935.94

Unweighted averages across six transfer directions and six accuracy benchmarks or three F1 benchmarks.

HeteroFold achieves the highest average accuracy and F1 while requiring approximately 25% fewer mapping parameters than Dense Latent and 62% fewer than KV Ridge. Offline calibration is performed only once per model pair, and the mappings can be reused across subsequent transfers.

The required mappings depend on the communicating model pairs, not the number of agents. In Magentic-One’s default configuration, the Orchestrator, Coder, WebSurfer, and FileSurfer all use the same GPT-4o model version despite performing different roles (Fourney et al., 2024, §5.1). Multiple agents do not necessarily require distinct models. HeteroFold learns a mapping per sender–receiver checkpoint pair, so agents using the same pair under the same transfer configuration can reuse that mapping without additional calibration.

Not every multi-agent system requires KV mappings between all model pairs. For example, with five distinct models, one coordinator exchanging information with four workers requires 8 directed mappings rather than 20. Anthropic’s Research system uses a similar coordinator–worker organization, with an Opus lead and Sonnet workers (Anthropic, 2025).

While fully connected scaling remains a limitation, reducing pairwise mapping complexity through shared representation spaces or factorized mappings is an important direction for future research.

References

AuthorsOpenReview ↗

Response to Reviewer 3DZp

We hope our responses have addressed the reviewer's key concerns through the following clarifications and additional experiments:

  • Interactive agent evaluation: Conducted new experiments on all 134 ALFWorld unseen tasks, achieving 74.6% success compared with 76.9% for native communication and 65.1% for KV Ridge + TA, without task-specific calibration.
  • Model-specific prompts: Clarified how HeteroFold handles different chat templates and system prompts by processing model-specific prefixes and suffixes natively while transferring shared context through KV caches.
  • Mapping efficiency: Demonstrated that HeteroFold requires approximately 25% fewer mapping parameters than Dense Latent and 62% fewer than KV Ridge, while achieving higher average accuracy and F1.
  • Scalability: Clarified that mappings can be reused across agents sharing the same model checkpoints and that coordinator-worker architectures can reduce the number of required mappings from 20 to 8 for five distinct models.

We sincerely appreciate the reviewer's constructive feedback. We hope these additional findings and clarifications have addressed the reviewer's concerns. We would sincerely appreciate it if the reviewer could reconsider their rating based on the additional evidence and experimental results presented.

Reviewer_ophzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HeteroFold transfers a sender's KV cache to a receiver from a different model family, so the receiver can skip prefill. Both models stay frozen. Tokens are aligned on character boundaries. Each receiver layer is built from three sender layers through whitening, Procrustes and recoloring, plus a low-rank output-aware calibration, all folded into one affine map per layer. Across six directions among different models, it beats Dense Latent and KV Ridge (both given the same token alignment) on long-context tasks. At 32K it reports 10.7× lower latency than native prefill.

Critical strengths

The setting fills a real gap, as prior prefill-free methods assume a shared tokenizer. Token alignment matters a lot. The authors also give it to the baselines, so the mapping itself is compared fairly. The final form is a single affine map per layer, and the paper is upfront about sender cost and memory.

Critical weaknesses
  1. Several transferred results beat the receiver's own TextMas. For Ministral -> Llama: ARC-C 90.36 vs 82.76, MMLU 71.76 vs 66.74, QuALITY 79.00 vs 75.07. Agreement with the native receiver's answers stays in the 70% range at most (Fig. 6). So part of the accuracy comes from the stronger sender rather than faithful transfer. This conflicts with the stated goal of preserving receiver behavior.

  2. Quality and speed are never reported at the same operating point. Long-context accuracy is far below TextMas, e.g. Qwen -> Ministral Qasper 20.57 vs 45.59 and LoCoMo 17.75 vs 52.01. The 10.7× figure is measured on 4K–32K contexts with no accuracy numbers, in one of the weaker directions. Without an accuracy–latency comparison against cheaper options (truncated prefill, CacheBlend/DroidSpeak-style partial recompute), it's hard to say when this is worth using. Also, the calibration corpus includes HotpotQA train; with Open-R1 alone, HotpotQA drops to 12.69 (Table E).

Questions for authors
  1. In each direction, what is the sender's own accuracy, and how often does the transferred answer agree with the sender's answer?
  2. What is the accuracy at the 16K/32K settings used for latency?
  3. How does Table 1 look with a calibration corpus that excludes HotpotQA?
  4. For the short-context tasks, which spans are transferred and which are prefilled natively by the receiver?
  5. Dense Latent is near chance even in the same-family setting (Table G). Was the reimplementation checked against the original paper's numbers?
7 comments
AuthorsOpenReview ↗

Response to Reviewer ophz

We thank the reviewer for their constructive feedback. We appreciate their recognition of the practical importance of cross-family KV transfer, fair baseline comparisons, and efficient affine mapping design. We address the concerns below with additional analysis and experimental results.

Response to Question 1: Sender Accuracy and Answer Agreement

Q1. In each direction, what is the sender's own accuracy, and how often does the transferred answer agree with the sender's answer?

A1. The sender only prefills the shared context without generating answers, while the receiver independently answers using transferred KV caches. We report sender native performance and answer agreement below.

Table R1. Native performance of sender models (Table 1).

Sender ModelLong F1 Avg. ↑QuALITY Acc. ↑Short Acc. Avg. ↑
Qwen3-4B46.0170.6168.50
Llama-3.1-8B51.5075.0774.35
Ministral-3-14B54.0583.3278.57

Long F1 averages Qasper, HotpotQA, and LoCoMo. Short Acc. averages five short-context benchmarks. Each sender's native performance is identical across its two transfer directions.

Table R2. Answer agreement (%) between HeteroFold, native sender, and native receiver predictions.

Answer agreement measures how often two models select the same answer, regardless of correctness.

Transfer DirectionHF vs. Sender NativeSender vs. Receiver NativeHF vs. Receiver Native
Qwen → Llama70.470.472.4
Ministral → Qwen76.772.577.4
Llama → Qwen71.470.472.8
Qwen → Ministral74.472.577.8
Llama → Ministral78.177.979.0

Measured on available multiple-choice evaluations using separately refitted HeteroFold mappings.

Across all evaluated directions, HeteroFold exhibits higher agreement with the receiver's native predictions than with the sender's predictions. Furthermore, its agreement with the sender is comparable to the agreement between the two independently evaluated native models.

For example, on HellaSwag, Qwen → Llama achieves 65.60% accuracy despite the sender's 41.54%, while Ministral → Qwen achieves 41.46% despite the sender's 68.33% (Table 1). These results suggest that HeteroFold preserves the receiver's prediction behavior rather than simply reproducing the sender's answers, consistent with our receiver behavior analysis in Fig. 6.

Response to Question 2: Accuracy at 16K/32K Context Lengths

Q2. What is the accuracy at the 16K/32K settings used for latency?

A2. We additionally evaluate accuracy at the same context lengths used in our latency experiments, using 100 QuALITY questions per length for Llama-3.1-8B → Ministral-3-14B transfer.

Table R3. Accuracy (%) at 16K and 32K context lengths.

Method16K Acc. (%) ↑32K Acc. (%) ↑
Native Prefill81.0080.00
Dense Latent29.0029.00
Dense Latent + TA35.0027.00
KV Ridge34.0017.00
KV Ridge + TA62.0063.00
HeteroFold (Ours)67.0067.00

HeteroFold achieves the highest accuracy among prefill-free methods at both context lengths. Notably, incorporating our Token Alignment (TA) improves KV Ridge by 28 and 46 percentage points at 16K and 32K, respectively. These results highlight the importance of token correspondence in cross-family KV transfer, while HeteroFold's further improvements demonstrate the benefits of addressing KV representation mismatches and receiver attention behavior beyond token alignment alone.

AuthorsOpenReview ↗

Response to Reviewer ophz

Response to Question 3: Calibration Without HotpotQA

Q3. How does Table 1 look with a calibration corpus that excludes HotpotQA?

A3. We replaced HotpotQA with MuSiQue and trained all three methods using 800 Open-R1 and 800 MuSiQue examples, keeping the evaluation protocol unchanged. We evaluated Ministral → Llama and Qwen → Ministral across all nine benchmarks.

Table R4. Long-context results without HotpotQA calibration examples.

Transfer DirectionMethodQasper F1 ↑HotpotQA F1 ↑LoCoMo F1 ↑QuALITY Acc. ↑
Ministral → LlamaDense Latent + TA3.25 ± 0.432.99 ± 0.042.1941.51
KV Ridge + TA16.68 ± 0.6631.27 ± 1.0431.0271.38
HeteroFold32.32 ± 1.6948.19 ± 0.6344.9079.10
Qwen → MinistralDense Latent + TA3.75 ± 0.182.87 ± 0.302.2328.95
KV Ridge + TA9.33 ± 0.3731.10 ± 0.5410.9064.53
HeteroFold19.08 ± 0.4544.35 ± 1.0023.1564.53

Scores use a 0–100 scale. Qasper and HotpotQA report mean ± sample standard deviation over three decoding seeds on 200 questions each, using fixed trained maps.

Table R5. Short-context accuracy (%) without HotpotQA calibration examples.

Transfer DirectionMethodARC-C ↑MMLU ↑WinoGrande ↑HellaSwag ↑GSM8K ↑
Ministral → LlamaDense Latent + TA56.5735.1350.8349.899.40
KV Ridge + TA88.4868.5853.3565.6554.74
HeteroFold90.0271.2363.6169.7664.44
Qwen → MinistralDense Latent + TA30.3833.5850.4350.7821.91
KV Ridge + TA84.4763.6952.3362.1255.72
HeteroFold85.1565.6556.9165.4692.72

HeteroFold retains the highest score across all nine benchmarks in both directions without HotpotQA calibration examples. On HotpotQA, it outperforms KV Ridge + TA by 16.92 and 13.25 F1 points, respectively. These results show that its advantage in these directions is maintained when HotpotQA is replaced by another multi-hop QA corpus.

Response to Question 4: Short-Context Transfer Spans

Q4. For the short-context tasks, which spans are transferred and which are prefilled natively by the receiver?

A4. All transferred task content is provided to the receiver LLM exclusively through mapped KV caches, without requiring the receiver to prefill the original task text. The receiver processes only its own chat-template tokens and performs task-specific answer scoring or generation.

Specifically, the transferred spans differ by task:

  • ARC-Challenge / MMLU: Transfer instructions, questions, and answer choices as KV states. The receiver scores candidate answer labels.
  • HellaSwag: Transfer instructions and context as KV states. The receiver scores candidate continuations.
  • WinoGrande: Transfer instructions and the sentence prefix before the blank as KV states. The receiver scores candidate completions and the remaining sentence.
  • GSM8K: Transfer instructions and the complete problem statement as KV states. The receiver generates the reasoning and final answer.

Importantly, the receiver LLM directly uses the mapped KV representations without reconstructing the transferred context through native text prefill.

Response to Question 5: Dense Latent Reproduction

Q5. Dense Latent is near chance even in the same-family setting (Table G). Was the reimplementation checked against the original paper's numbers?

A5. Since the official implementation of Dense Latent is not publicly available, exact reproduction is challenging. Nevertheless, we carefully reproduced the method based on the published methodology and conducted multiple validation checks.

Notably, our implementation achieves 90.30% on GSM8K (Qwen3 4B → 8B), close to the 91.43% reported in the original paper under a comparable context-unaware setting.

The remaining differences may arise from different calibration and evaluation protocols. The original study employs extensive reconstruction training followed by generation supervision using gold-answer-based traces, whereas we use a shared calibration dataset without gold-answer supervision and evaluate additional long-context benchmarks.

Our reproduction follows the published methodology, and all methods are evaluated under consistent experimental settings to ensure fair comparisons.

AuthorsOpenReview ↗

Response to Reviewer ophz

We hope our responses have addressed the reviewer's key concerns through the following clarifications and additional experiments:

  • Receiver behavior preservation: Reported sender accuracy and answer agreement, showing that HeteroFold aligns more closely with the receiver's predictions than the sender's across evaluated directions.
  • Long-context performance: Provided new accuracy results at 16K and 32K, demonstrating consistent improvements over existing prefill-free baselines.
  • Calibration robustness: Confirmed performance gains across nine benchmarks without HotpotQA calibration data.
  • Experimental clarity: Clarified which task spans are transferred through KV caches and which tokens are processed natively by the receiver.
  • Baseline validation: Verified our Dense Latent implementation against the original paper, achieving 90.30% vs. 91.43% on GSM8K under comparable settings.

We sincerely appreciate the reviewer's constructive feedback. Given these additional results and clarifications that directly address the raised concerns, we would be grateful if the reviewer could reconsider their assessment and consider increasing their rating.

Reviewer_etyUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes HeteroFold, a method for transferring KV caches between LLMs from different model families without requiring the receiver to prefill the shared context. The method first aligns tokens and layers across models, then maps the sender's KV representations into the receiver's space using Recolor and output-aware calibration. The resulting mappings are fixed during inference, while both models remain frozen. The authors evaluate HeteroFold on six transfer directions among Llama, Qwen, and Ministral, covering both question-answering and multi-agent tasks. The experiments report better performance than existing cache-transfer baselines on long-context benchmarks and lower latency compared with native receiver prefill.

Critical strengths

Strengths

  1. The paper addresses a practical problem of transferring KV caches between LLMs with different tokenizers and architectures without requiring receiver prefill. This could help reduce redundant computation in heterogeneous multi-agent systems.

  2. The proposed method combines token and layer alignment, Recolor mapping, and output-aware calibration. The learned corrections can be merged into fixed affine mappings, avoiding additional correction modules during inference.

  3. The paper provides extensive experiments across six transfer directions and multiple benchmarks. The authors also include component ablations and latency measurements to examine the contributions of different components.

Critical weaknesses

Weakness

  1. Limited Methodological Novelty: The proposed method mainly combines existing techniques, including token alignment, layer alignment, whitening and Procrustes-based Recolor, and output-aware calibration. In particular, the output-aware calibration is adapted from prior work on KV-cache quantization. The methodological novelty beyond combining these existing techniques for cross-family KV transfer is somewhat limited.

  2. Significant Performance Gap Compared with Native Prefill: Although HeteroFold outperforms existing cache-transfer baselines, there are still substantial performance gaps compared with text-based communication. For example, on Qwen3-4B to Ministral-3-14B transfer, HeteroFold achieves only 20.57 F1 on Qasper compared with 45.59 for TextMas, and 17.75 versus 52.01 on LoCoMo. These results raise concerns about the information loss introduced by cross-family KV mapping and its practical usefulness.

  3. Limited End-to-End Efficiency and Generalization Evaluation: The reported speedups exclude sender prefill, offline calibration, and subsequent decoding. The latency experiments also assume two H100 GPUs connected through NVLink, without evaluating cross-server communication. Furthermore, the experiments only cover three model families, and the multi-agent evaluation contains 65 tasks. More diverse deployment settings would help establish the practical benefits and generalization of the proposed method.

Questions for authors

Please check Critical Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer etyU

We thank the reviewer for their thoughtful evaluation and constructive feedback. We appreciate the recognition of the practical importance of cross-family KV transfer and the efficiency of our fixed affine mappings. We also appreciate the positive assessment of our extensive experiments across multiple transfer directions and benchmarks. We address the three concerns below with additional analysis and experimental results.

Response to Weakness 1: Limited Methodological Novelty

W1. Limited Methodological Novelty. The proposed method mainly combines existing techniques rather than introducing fundamentally new methods.

A1. To our knowledge, HeteroFold is the first method to systematically identify and address the challenges of prefill-free KV-cache transfer across different LLM families, enabling direct reuse of intermediate computations despite incompatible tokenizers, architectures, and KV representations.

Specifically, we identify three key challenges: tokenizer mismatch, architectural differences, and receiver attention distortion. We propose simple yet effective solutions to each challenge, while our analysis demonstrates why conventional KV transfer approaches fail across model families (Figs. 2 and 5) and our ablations validate the necessity of each component (Table 3).

Table R1. Average performance across six cross-family transfer directions (Table 1).

MethodAvg. Acc. (%) ↑Avg. F1 (%) ↑
Dense Latent30.431.54
Dense Latent + TA34.273.08
KV Ridge30.651.42
KV Ridge + TA62.6923.01
HeteroFold (Ours)67.7935.94

Unweighted average across six transfer directions and six accuracy benchmarks or three F1 benchmarks.

As shown above, existing methods exhibit substantial performance degradation when applied to cross-family KV transfer without Token Alignment (TA). Incorporating TA significantly improves their performance, particularly for KV Ridge. Nevertheless, HeteroFold achieves the highest average accuracy and F1, demonstrating the importance of addressing representational mismatches and preserving receiver attention behavior beyond token alignment alone.

Beyond the proposed method, our work provides a systematic understanding of the key challenges in cross-family KV transfer and demonstrates effective ways to address them, offering a foundation for future research on efficient KV reuse across different LLMs.

Response to Weakness 2: Performance Gap Compared with Native Prefill

W2. Significant Performance Gap Compared with Native Prefill. HeteroFold exhibits performance gaps compared with text-based communication, raising concerns about information loss and practical usefulness.

A2. We acknowledge that HeteroFold falls short of native prefill on several benchmarks. At 32K, skipping receiver prefill gives HeteroFold a 10.7× speedup over native prefill in time to first token (Llama → Ministral). It also achieves the smallest average accuracy and F1 gaps among the evaluated prefill-free methods.

Table R2. Average performance gaps relative to TextMas across six cross-family transfer directions (Table 1).

MethodAvg. Acc. Gap ↓Avg. F1 Gap ↓
TextMas (Native Prefill)0.000.00
Dense Latent43.8048.98
Dense Latent + TA39.9647.44
KV Ridge43.5849.11
KV Ridge + TA11.5427.51
HeteroFold (Ours)6.4414.58

Average score differences from TextMas across six transfer directions.

The 32K QuALITY results for Llama → Ministral illustrate the accuracy–latency trade-off. HeteroFold achieved 67% accuracy versus 80% with native prefill. At the same context length and in the same transfer direction, Table 4 shows that reusing the sender cache reduced the time to the receiver’s first token from 5.17 s to 0.48 s.

At this operating point, HeteroFold was both more accurate and faster than the two baselines with the same token alignment: KV Ridge (Heo et al., NeurIPS 2026) + TA achieved 63% accuracy at 0.57 s, and Dense Latent + TA achieved 27% at 0.71 s. Inspired by SpecPrefill (Liu et al., 2025), we would like to investigate whether the receiver can narrow the remaining gap to native prefill by recomputing KV states for selected tokens while reusing the transferred states for the rest.

References

  • Heo, T., et al. (2026). Cross-Model KV Cache Transfer in LLM Families: A Closed-Form Linear Mapping for Prefill Reuse. NeurIPS 2026. arXiv:2608.03893
  • Liu, J., et al. (2025). Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation. ICML 2025. arXiv:2502.02789
AuthorsOpenReview ↗

Response to Reviewer etyU

Response to Weakness 3: Limited End-to-End Efficiency and Generalization Evaluation

W3. Limited End-to-End Efficiency and Generalization Evaluation. The reported speedups exclude sender prefill, offline calibration, and subsequent decoding. The experiments are limited to NVLink GPUs and 65 multi-agent tasks.

A3. Our primary objective is to reduce redundant computation during sender-to-receiver communication, where the sender has already processed the shared context. Nevertheless, we provide additional measurements addressing the reviewer's concerns.

1. Latency Including Sender Prefill. Since sender prefill is required for both methods, its cost is nearly identical (3.276s for TextMas and 3.263s for HeteroFold). Even when including sender prefill, HeteroFold achieves a 2.22× speedup at 32K context length.

Table R3. 32K latency including sender prefill (Llama → Ministral).

MethodTotal Latency ↓Speedup ↑
TextMas8.301 s1.00×
HeteroFold3.741 s2.22×

Measured from sender prefill through the receiver's first output token, excluding subsequent decoding.

2. Efficiency on PCIe GPUs. We additionally evaluate HeteroFold on two H100 GPUs connected via PCIe rather than NVLink. HeteroFold maintains substantial speedups, achieving 7.81× and 7.27× over native receiver prefill at 32K context length.

Table R4. 32K receiver-side transfer speedup on PCIe GPUs.

MethodLlama → MinistralQwen → Ministral
Dense Latent + TA4.98×4.78×
KV Ridge + TA5.51×5.24×
HeteroFold (Ours)7.81×7.27×

Measured on two H100 GPUs within the same server, including GPU data transfer but excluding sender prefill and subsequent decoding.

3. Offline Calibration. Calibration is a one-time process per model pair, rather than a recurring inference cost. For Llama → Ministral, HeteroFold requires approximately 48 minutes on two H100 GPUs, and the resulting mappings can be reused across subsequent transfers.

4. Multi-Agent Evaluation. Our evaluation covers all 65 Hidden Profile tasks in HiddenBench, rather than a selected subset, providing complete coverage of this benchmark.

AuthorsOpenReview ↗

Response to Reviewer etyU

We hope our responses have addressed the reviewer's key concerns through the following clarifications and additional experiments:

  • Methodological contribution: Clarified the unique challenges of prefill-free cross-family KV transfer and demonstrated consistent improvements over existing methods across six transfer directions.
  • Performance gap: Quantified the gaps relative to TextMas, showing that HeteroFold achieves the smallest average accuracy gap (6.44 points) and F1 gap (14.58 points) among prefill-free methods.
  • Latency including sender prefill: Demonstrated a 2.22× speedup at 32K even when including sender prefill, while excluding subsequent decoding.
  • Deployment efficiency: Confirmed substantial speedups of 7.81× and 7.27× on PCIe GPUs and clarified that offline calibration is a one-time cost per model pair.
  • Evaluation coverage: Clarified that our multi-agent evaluation includes all 65 Hidden Profile tasks in HiddenBench. On all 134 unseen ALFWorld tasks, HeteroFold achieved 74.6% success without task-specific calibration, compared with 65.1% for KV Ridge + TA and 76.9% for TextMas.
  • Model generalization: On Phi → Llama, HeteroFold outperformed KV Ridge + TA by an average of 38.80 points across five short-context benchmarks and 29.27 points across four long-context benchmarks.

We sincerely appreciate the reviewer's constructive feedback. We hope these additional findings and clarifications have sufficiently addressed the reviewer's concerns. We would greatly appreciate it if the reviewer would consider increasing their rating in light of these additional results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Reviews

We sincerely thank all reviewers for their thoughtful and constructive feedback. We appreciate their careful evaluation of our work and their valuable suggestions.

We particularly appreciate the reviewers' shared recognition of the significance and novelty of our problem setting. Reviewer ophz emphasizes that our work "fills a real gap," Reviewer 3DZp describes HeteroFold as "the first doable approach to sharing KV caches across different model families," and Reviewer a4jF recognizes that "the cross-family setting seems new and worth studying." Reviewer etyU also highlights the practical importance of reducing redundant computation in heterogeneous multi-agent systems.

Furthermore, the reviewers recognize several important strengths of our methodological design and experimental evaluation. In particular, Token Alignment effectively addresses cross-tokenizer mismatches (Reviewers ophz, 3DZp, a4jF), while receiver-aware calibration and fixed affine mappings enable direct KV cache reuse without additional inference modules (Reviewers etyU, 3DZp). The reviewers also acknowledge our consistent improvements over existing KV transfer baselines across six transfer directions, including comparisons where the baselines are equipped with the same Token Alignment (Reviewers ophz, 3DZp, a4jF). Additionally, Reviewer etyU recognizes our extensive experiments, component ablations, and latency measurements.

We believe these observations highlight the significance of identifying and addressing the fundamental challenges of prefill-free cross-family KV transfer. We provide detailed responses to the reviewers' questions and concerns in our individual rebuttals. We sincerely appreciate the opportunity to clarify our contributions and hope our responses will strengthen the reviewers' confidence in our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.