- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes KV-Lingo, a framework that enables KV cache translation between large language models with fixed weights using linear mappings and self-distillation. Experiments across multiple model families and tasks demonstrate its effectiveness in KV cache transfer. Compared with re-prefilling the context, KV-Lingo achieves up to 29x faster model switching.
Critical strengths
-
Effective self-distillation. Self-distillation yields substantial gains in cache translation quality over closed-form reconstruction, while preserving the lightweight linear architecture.
-
Including the appendix, the paper spans 80 pages and evaluates KV cache transfer across diverse scenarios, including different model architectures, tokenizers, and multi-turn conversations, providing useful insights into the practical applicability of the proposed method.
Critical weaknesses
- The paper compares different subsets of baselines under different evaluation settings and metrics. Figure 2 focuses on accuracy with selected translators, Figure 3 provides accuracy–latency comparisons for only a subset of methods, while Figure 5 mainly reports speedups over re-prefilling. Appendix M further compares DroidSpeak in terms of accuracy and analytical computation costs. Without a unified accuracy–latency evaluation covering closely related methods, like CacheBridge, it remains unclear whether KV-Lingo offers a better overall accuracy–efficiency trade-off than existing alternatives.
Questions for authors
-
Can the authors provide a unified accuracy–latency comparison with closely related methods, such as C2C, CacheBridge, and DroidSpeak, under consistent model pairs, benchmarks, and evaluation metrics? How does KV-Lingo compare with these methods in terms of the overall accuracy–efficiency trade-off?
-
How can KV-Lingo mitigate the substantial accuracy degradation observed for heterogeneous model pairs and long-context tasks? To what extent can this degradation be reduced without significantly increasing the translation overhead?
2 comments
Dear Reviewer,
We have now replied to each review in detail. Since the discussion window closes on November 18, after which we can no longer see or respond to comments, we are writing to invite discussion while it is still time for us to act on it. We would genuinely value hearing from you, even briefly.
Thank you for the work you have already put in. Reviewing takes real time, and we are grateful for it.
Authors
We thank the reviewer for their comments and careful observations, and clarify below the scope of each comparison we added.
W1 / Q1. "Can the authors provide a unified accuracy–latency comparison with closely related methods, such as C2C, CacheBridge, and DroidSpeak, under consistent model pairs, benchmarks, and evaluation metrics?"
Upon your remark, we added a common accuracy–latency evaluation for the implemented translation baselines, using consistent model pairs, benchmarks, and timing protocols (App. M.1, Fig. 33): the nine-benchmark mean, reasoning-off, against the on-device switch time on an Apple M5 Ultra. Each panel compares target re-prefill, the closed form of Heo et al., the head-wise MLP architecture of Chen et al. and a head-mixing MLP (both trained with our recipe), and KV-Lingo, head-mixing and head-wise, with 1 to 8 source layers per target layer (ν) on the cross-depth pair; CacheBridge is added in the two same-shape S→L panels. Where a baseline was missing from the original paper, we trained or fitted it and ran the full evaluation. Fig. 33 also provides a separate two-benchmark comparison with an adapted DroidSpeak baseline.
KV-Lingo has pareto dominating configurations in each panel (App. M.1).
CacheBridge. We now re-implement it and evaluate it on the same benchmarks at two capture points (Table 25, Fig. 33): after key normalisation, as specified, and before it, so that the target's own normalisation rescales each predicted key.
Moving the fit before key normalisation substantially improves CacheBridge: on 0.6B→1.7B from 0.246 to 0.490 (RULER 0.139 to 0.731), while KV-Lingo reaches 0.540; on 4B→8B from 0.752 to 0.784, within 0.01 of KV-Lingo (0.792). Timed as head-wise KV-Lingo at ν = 8 (same operator shape), its switch is about 3× slower than KV-Lingo in our implementation.
DroidSpeak. Our models have different hidden widths, so we evaluate a compatible restricted variant of DroidSpeak that recomputes a prefix of the target's layers and reuses the source cache above that boundary (App. M). On Qwen3-0.6B/1.7B, it approaches the target's accuracy only with 22 of 28 layers recomputed, at which point it costs as much as re-prefill on these short prompts.
C2C requires the target's native cache and therefore addresses a complementary cache-enrichment setting. It cannot replace target prefill in the handoff setting studied here, so we do not evaluate it; the shaded band in Fig. 33 only marks where its switch time would start.
We thank you for this unifying view, and hope that it clarifies the effectiveness of KV-Lingo.
Q2. "How can KV-Lingo mitigate the substantial accuracy degradation observed for heterogeneous model pairs and long-context tasks? To what extent can this degradation be reduced without significantly increasing the translation overhead?"
We evaluated four possible remedies:
More source layers per target layer (ν). On Qwen3-8B→0.6B (Table 15), widening the map substantially improves long-context retrieval, at a translation cost proportional to ν (Fig. 25):
LongMemEval thus reaches or exceeds the small model from ν = 2, while RULER keeps a large deficit. On the nine-benchmark mean, ν = 8 reaches the 0.6B's own score (0.511 against 0.506) with a 60 ms switch, against 157 ms for its re-prefill (App. M.1). In the other direction, 0.6B→8B lands above the 0.6B at every LongMemEval length (Fig. 3).
A non-linear translator. A head-mixing MLP is on par with KV-Lingo at ν = 2 on Qwen3-0.6B/8B (0.466 against 0.476 in L→S, 0.597 against 0.577 in S→L). On the same-shape pairs it gains at most 0.026, at 3–4× the switch time (App. M.1).
Reasoning. With reasoning on, the large model reading a small model's cache recovers 45–74% of the gap between the two models on four of six pairs; on the six benchmarks scored in both modes, at most 35% with reasoning off (matched subset, App. P). The cost is a longer decode (Table 34, App. P).
Hybrids. Recomputing the target's first j of 28 layers natively (0.6B→1.7B, App. M.2) recovers little until 20–24 layers are native, which gives back most of the prefill saving. Translating the first n − k tokens and letting the target prefill the remaining k = 128 or 512 does not substantially recover the RULER gap on 8B→0.6B (0.413 and 0.436 against 0.430; App. T).
We thank you again for your remarks which help us improve the paper.