- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies prefill-free KV-cache transfer between models from different families, where the sender and receiver differ in tokenizer, depth, and KV structure. The proposed method, HeteroFold, aligns tokens by shared character boundaries (TA), concatenates three sender layers around the proportionally matched depth, maps K/V with a moment-matching affine map (Recolor), and adds a low-rank correction trained to match native receiver attention and outputs. The correction is folded into the affine map, so inference uses one fixed map per layer and role.
Across six directions among Llama-3.1-8B, Qwen3-4B, and Ministral-3-14B, HeteroFold outperforms Dense Latent and KV Ridge (with and without TA) on all long-context benchmarks and most short-context ones. At 32K context, transfer is reported to be 10.7× faster than native receiver prefill, including inter-GPU payload transfer over NVLink.
Critical strengths
-
The cross-family setting seems new and worth studying. Existing prefill-free methods are evaluated only within the same model family, and cross-family methods such as C2C still require receiver prefill. Handling both together fits well with heterogeneous multi-agent systems.
-
The paper clearly shows why tokenizer mismatch matters. Without TA, KV Ridge and Dense Latent drop to almost zero on long-context tasks, and same-index pairing is off by hundreds of characters at 32K. TA is a simple rule, but these results make its necessity convincing.
-
Even when the baselines are given the same TA, HeteroFold is consistently better on long-context QA in all six directions, sometimes by a large margin (e.g., HotpotQA 17.1 → 42.7 for Qwen→Ministral). Similar gains in the same-family setting suggest that the improvement does not come only from TA.
-
The latency evaluation seems fair. Sender and receiver are on separate GPUs, KV transfer over NVLink is included, and contexts up to 32K are tested. Sender prefill is also reported separately, and the method is still faster than native prefill at 32K even when it is added.
Critical weaknesses
-
Novelty of the components. The setting is new, but the individual parts (character-offset alignment, whitening/coloring, output-aware calibration from KV quantization) are fairly standard. Character-offset alignment is also used in cross-tokenizer distillation, which is not discussed.
-
Gain over KV Ridge + TA. Most of the gap to the original baselines seems to come from TA. Against KV Ridge + TA, the gain is clear on long-context tasks, but small or mixed on short-context ones.
The ablation is only for one direction, and its TA + Ridge row (e.g., 4.55 on GSM8K) does not match KV Ridge + TA in Table 1 (53.30). It is not clear why. Also, TA + Recolor alone already beats KV Ridge + TA there, so the role of calibration seems smaller than suggested.
-
Long-context accuracy. Speedup is reported at 32K, but accuracy is not reported by length. The gap to TextMas is also still large in several cases, especially on GSM8K.
-
Evaluation. No seed variance, and Qasper/HotpotQA use 200-item subsets. HotpotQA is part of the calibration data. HiddenBench has only 65 tasks, so the claim of matching text communication seems a bit strong. Calibration-corpus sensitivity and latency are each tested in a limited setting (one direction, NVLink only, batch 2).
Questions for authors
- Why is “TA + Ridge” in Table 3 so different from KV Ridge + TA in Table 1 for the same direction? Could the ablation be shown for at least one more direction?
- How does accuracy change at 16K and 32K, where the main speedup is reported?
- How is TA different from character-offset alignment used in cross-tokenizer distillation?
- Could the authors report seed variance for Qasper, HotpotQA, and HiddenBench?
- How would the latency change with PCIe or network transfer instead of NVLink?
6 comments
Response to Reviewer a4jF
Response to Question 4: Seed Variance and Calibration Without HotpotQA
Q4. Could the authors report seed variance for Qasper, HotpotQA, and HiddenBench?
A4. We additionally evaluated Qasper and HotpotQA with three decoding seeds in two transfer directions. To address the calibration-data concern, we also replaced HotpotQA with MuSiQue and trained all three methods using 800 Open-R1 and 800 MuSiQue examples. Results across all nine benchmarks are reported below.
Table R2. Long-context results without HotpotQA calibration examples.
| Transfer Direction | Method | Qasper F1 ↑ | HotpotQA F1 ↑ | LoCoMo F1 ↑ | QuALITY Acc. ↑ |
|---|---|---|---|---|---|
| Ministral → Llama | Dense Latent + TA | 3.25 ± 0.43 | 2.99 ± 0.04 | 2.19 | 41.51 |
| KV Ridge + TA | 16.68 ± 0.66 | 31.27 ± 1.04 | 31.02 | 71.38 | |
| HeteroFold | 32.32 ± 1.69 | 48.19 ± 0.63 | 44.90 | 79.10 | |
| Qwen → Ministral | Dense Latent + TA | 3.75 ± 0.18 | 2.87 ± 0.30 | 2.23 | 28.95 |
| KV Ridge + TA | 9.33 ± 0.37 | 31.10 ± 0.54 | 10.90 | 64.53 | |
| HeteroFold | 19.08 ± 0.45 | 44.35 ± 1.00 | 23.15 | 64.53 |
Scores use a 0–100 scale. Qasper and HotpotQA each use the same 200 evaluation questions across decoding seeds 1001, 2001, and 3001. We report mean ± sample standard deviation with fixed trained maps.
Table R3. Short-context accuracy (%) without HotpotQA calibration examples.
| Transfer Direction | Method | ARC-C ↑ | MMLU ↑ | WinoGrande ↑ | HellaSwag ↑ | GSM8K ↑ |
|---|---|---|---|---|---|---|
| Ministral → Llama | Dense Latent + TA | 56.57 | 35.13 | 50.83 | 49.89 | 9.40 |
| KV Ridge + TA | 88.48 | 68.58 | 53.35 | 65.65 | 54.74 | |
| HeteroFold | 90.02 | 71.23 | 63.61 | 69.76 | 64.44 | |
| Qwen → Ministral | Dense Latent + TA | 30.38 | 33.58 | 50.43 | 50.78 | 21.91 |
| KV Ridge + TA | 84.47 | 63.69 | 52.33 | 62.12 | 55.72 | |
| HeteroFold | 85.15 | 65.65 | 56.91 | 65.46 | 92.72 |
HeteroFold retains the highest score across all nine benchmarks in both directions without HotpotQA calibration examples. Its advantage over both baselines holds at every evaluated decoding seed on Qasper and HotpotQA. On HotpotQA, its mean improvements over KV Ridge + TA are 16.92 and 13.25 F1 points, respectively.
Response to Question 5: Latency on PCIe GPUs
Q5. How would the latency change with PCIe or network transfer instead of NVLink?
A5. We additionally evaluate HeteroFold on two H100 GPUs connected via PCIe instead of NVLink, including inter GPU KV cache transfer. Although absolute latency increases, HeteroFold maintains substantial speedups over native receiver prefill.
Table R4. 32K transfer latency on PCIe GPUs (ms ↓, speedup in parentheses ↑).
| Method | Llama → Ministral | Qwen → Ministral |
|---|---|---|
| Native Prefill | 8083.5 (1.00×) | 8137.9 (1.00×) |
| Dense Latent + TA | 1621.8 (4.98×) | 1703.3 (4.78×) |
| KV Ridge + TA | 1467.4 (5.51×) | 1552.0 (5.24×) |
| HeteroFold (Ours) | 1035.5 (7.81×) | 1119.4 (7.27×) |
Measured on two H100 GPUs within the same server using PCIe, including KV data transfer.
HeteroFold achieves the lowest latency in both directions, retaining 7.81× and 7.27× speedups over native receiver prefill. These results demonstrate that HeteroFold's efficiency benefits extend beyond NVLink environments, although cross server network performance depends on available communication bandwidth.
Response to Reviewer a4jF
We hope our responses have addressed the reviewer's key concerns through the following clarifications and additional experiments:
- Methodological clarity: Clarified the differences between Ridge methods and distinguished our token alignment from prior cross-tokenizer distillation.
- Long-context performance: Provided new accuracy results at 16K and 32K, demonstrating consistent improvements over existing prefill-free baselines.
- Evaluation robustness: Confirmed performance gains without HotpotQA calibration data and reported seed variance across multiple transfer directions.
- Practical efficiency: Demonstrated substantial speedups of 7.81× and 7.27× on PCIe GPUs beyond the original NVLink setting.
We sincerely appreciate the reviewer's constructive feedback. Given these additional results and clarifications that directly address the raised concerns, we would be grateful if the reviewer could reconsider their assessment and consider increasing their rating.
Response to Reviewer a4jF
We thank the reviewer for their constructive feedback. We appreciate their recognition of the novelty of cross-family KV transfer, the importance of token alignment, and our consistent improvements over baselines. We also appreciate their positive assessment of our latency evaluation. We address the concerns below with additional analysis and experimental results.
Response to Question 1: Difference Between TA + Ridge and KV Ridge + TA
Specifically, TA + Ridge (Table 3) retains our three sender layer aggregation and cross head mapping, replacing only Recolor with Ridge Regression. The reported row does not include output aware calibration, which is evaluated separately.
In contrast, KV Ridge + TA (Table 1) follows the original KV Ridge method, selecting eight sender layers based on calibration R2 and fitting separate Ridge mappings for each KV head. Only Token Alignment (TA) is added.
Thus, the performance difference arises from their distinct layer selection, mapping architectures, and calibration settings, rather than inconsistent evaluations. We will clarify this distinction in Table 3 to avoid confusion.
Response to Question 2: Accuracy at 16K and 32K
Table R1. Accuracy (%) at 16K and 32K context lengths.
HeteroFold achieves the highest accuracy among prefill-free methods at both context lengths, maintaining 67% accuracy from 16K to 32K. Notably, incorporating our Token Alignment (TA) improves KV Ridge by 28 and 46 percentage points, respectively, highlighting the importance of token correspondence across different model families. HeteroFold further improves upon KV Ridge + TA, demonstrating the benefits of its KV mapping and receiver-aware calibration.
Response to Question 3: Token Alignment vs. Cross-Tokenizer Distillation
While cross-tokenizer distillation aligns text spans to provide training supervision (Nguyen et al., 2026), our TA establishes causal correspondence between sender KV states and receiver token positions. Specifically, unmatched receiver positions reuse sender KV states from the latest preceding shared character boundary, allowing direct cache construction without receiver prefill.
Our contribution lies in adapting token alignment to prefill-free cross-family KV transfer, together with representation mapping and receiver-aware calibration. We will clarify this distinction and acknowledge the related literature.
References