acceptodds
Under review as a conference paper at ICLR 2027

KV-Lingo: Learning KV-Cache Translators with Distillation

15 views

Abstract

Large language models represent context with a key-value (KV) cache. Caches are model-specific: for the same text, models with different architectures or weights produce incompatible representations. This makes it costly to switch models over a shared context: although the context has already been processed by one model, the incoming model must process it again to build its own cache. We introduce KV-Lingo, a method for translating the KV cache of a source model into one that can be read by a target model. KV-Lingo consists of a collection of linear maps, typically one per layer of the target model, that are applied independently on all tokens' key and value representations. We train these maps using distillation, minimising the divergence between the target model’s predictions from its native cache and those from the translated cache. We consider several model pairs spanning multiple sizes and architectures, training one translator per pair on a generic text corpus. The resulting translators preserve strong downstream performance in both small-to-large and large-to-small transfers. Since a switch then costs a linear map and a single decoding step instead of a prefill, replacing re-prefill with cache translation reduces the time to first token after a model switch by 9.6× already on a 64-token prompt for Qwen models on an Apple M3 Ultra, and by up to 29× at 32k context length on an H100. These gains make KV-Lingo particularly useful for dynamic model routing: a context can be processed by one model and handed off to another only when needed, without re-prefilling the shared prefix. We finally show that KV-Lingo can be used for seamless model switching, staying close to re-prefill across repeated switches in our multi-turn evaluations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, cache translation, model switching, representation alignment, distillation, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Z5mJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes KV-Lingo, a framework that enables KV cache translation between large language models with fixed weights using linear mappings and self-distillation. Experiments across multiple model families and tasks demonstrate its effectiveness in KV cache transfer. Compared with re-prefilling the context, KV-Lingo achieves up to 29x faster model switching.

Critical strengths
  • Effective self-distillation. Self-distillation yields substantial gains in cache translation quality over closed-form reconstruction, while preserving the lightweight linear architecture.

  • Including the appendix, the paper spans 80 pages and evaluates KV cache transfer across diverse scenarios, including different model architectures, tokenizers, and multi-turn conversations, providing useful insights into the practical applicability of the proposed method.

Critical weaknesses
  • The paper compares different subsets of baselines under different evaluation settings and metrics. Figure 2 focuses on accuracy with selected translators, Figure 3 provides accuracy–latency comparisons for only a subset of methods, while Figure 5 mainly reports speedups over re-prefilling. Appendix M further compares DroidSpeak in terms of accuracy and analytical computation costs. Without a unified accuracy–latency evaluation covering closely related methods, like CacheBridge, it remains unclear whether KV-Lingo offers a better overall accuracy–efficiency trade-off than existing alternatives.
Questions for authors
  1. Can the authors provide a unified accuracy–latency comparison with closely related methods, such as C2C, CacheBridge, and DroidSpeak, under consistent model pairs, benchmarks, and evaluation metrics? How does KV-Lingo compare with these methods in terms of the overall accuracy–efficiency trade-off?

  2. How can KV-Lingo mitigate the substantial accuracy degradation observed for heterogeneous model pairs and long-context tasks? To what extent can this degradation be reduced without significantly increasing the translation overhead?

2 comments

We thank the reviewer for their comments and careful observations, and clarify below the scope of each comparison we added.

  • W1 / Q1. "Can the authors provide a unified accuracy–latency comparison with closely related methods, such as C2C, CacheBridge, and DroidSpeak, under consistent model pairs, benchmarks, and evaluation metrics?"

    Upon your remark, we added a common accuracy–latency evaluation for the implemented translation baselines, using consistent model pairs, benchmarks, and timing protocols (App. M.1, Fig. 33): the nine-benchmark mean, reasoning-off, against the on-device switch time on an Apple M5 Ultra. Each panel compares target re-prefill, the closed form of Heo et al., the head-wise MLP architecture of Chen et al. and a head-mixing MLP (both trained with our recipe), and KV-Lingo, head-mixing and head-wise, with 1 to 8 source layers per target layer (ν) on the cross-depth pair; CacheBridge is added in the two same-shape S→L panels. Where a baseline was missing from the original paper, we trained or fitted it and ran the full evaluation. Fig. 33 also provides a separate two-benchmark comparison with an adapted DroidSpeak baseline.

    • KV-Lingo has pareto dominating configurations in each panel (App. M.1).

    • CacheBridge. We now re-implement it and evaluate it on the same benchmarks at two capture points (Table 25, Fig. 33): after key normalisation, as specified, and before it, so that the target's own normalisation rescales each predicted key.

      nine-benchmark mean, S→Lclosed form (Heo et al.)CacheBridge, before key norm.CacheBridge, as specifiedKV-Lingosmall model alone
      0.6B→1.7B0.2150.4900.2460.5400.508
      4B→8B0.5910.7840.7520.7920.782

      Moving the fit before key normalisation substantially improves CacheBridge: on 0.6B→1.7B from 0.246 to 0.490 (RULER 0.139 to 0.731), while KV-Lingo reaches 0.540; on 4B→8B from 0.752 to 0.784, within 0.01 of KV-Lingo (0.792). Timed as head-wise KV-Lingo at ν = 8 (same operator shape), its switch is about 3× slower than KV-Lingo in our implementation.

    • DroidSpeak. Our models have different hidden widths, so we evaluate a compatible restricted variant of DroidSpeak that recomputes a prefix of the target's layers and reuses the source cache above that boundary (App. M). On Qwen3-0.6B/1.7B, it approaches the target's accuracy only with 22 of 28 layers recomputed, at which point it costs as much as re-prefill on these short prompts.

    • C2C requires the target's native cache and therefore addresses a complementary cache-enrichment setting. It cannot replace target prefill in the handoff setting studied here, so we do not evaluate it; the shaded band in Fig. 33 only marks where its switch time would start.

    We thank you for this unifying view, and hope that it clarifies the effectiveness of KV-Lingo.

  • Q2. "How can KV-Lingo mitigate the substantial accuracy degradation observed for heterogeneous model pairs and long-context tasks? To what extent can this degradation be reduced without significantly increasing the translation overhead?"

    We evaluated four possible remedies:

    • More source layers per target layer (ν). On Qwen3-8B→0.6B (Table 15), widening the map substantially improves long-context retrieval, at a translation cost proportional to ν (Fig. 25):

      8B→0.6B0.6B, own cacheν = 1ν = 2ν = 4ν = 8
      RULER0.8340.2860.4310.4530.576
      LongMemEval0.3330.2890.3370.3680.392

      LongMemEval thus reaches or exceeds the small model from ν = 2, while RULER keeps a large deficit. On the nine-benchmark mean, ν = 8 reaches the 0.6B's own score (0.511 against 0.506) with a 60 ms switch, against 157 ms for its re-prefill (App. M.1). In the other direction, 0.6B→8B lands above the 0.6B at every LongMemEval length (Fig. 3).

    • A non-linear translator. A head-mixing MLP is on par with KV-Lingo at ν = 2 on Qwen3-0.6B/8B (0.466 against 0.476 in L→S, 0.597 against 0.577 in S→L). On the same-shape pairs it gains at most 0.026, at 3–4× the switch time (App. M.1).

    • Reasoning. With reasoning on, the large model reading a small model's cache recovers 45–74% of the gap between the two models on four of six pairs; on the six benchmarks scored in both modes, at most 35% with reasoning off (matched subset, App. P). The cost is a longer decode (Table 34, App. P).

    • Hybrids. Recomputing the target's first j of 28 layers natively (0.6B→1.7B, App. M.2) recovers little until 20–24 layers are native, which gives back most of the prefill saving. Translating the first n − k tokens and letting the target prefill the remaining k = 128 or 512 does not substantially recover the RULER gap on 8B→0.6B (0.413 and 0.436 against 0.430; App. T).

We thank you again for your remarks which help us improve the paper.

Dear Reviewer,

We have now replied to each review in detail. Since the discussion window closes on November 18, after which we can no longer see or respond to comments, we are writing to invite discussion while it is still time for us to act on it. We would genuinely value hearing from you, even briefly.

Thank you for the work you have already put in. Reviewing takes real time, and we are grateful for it.

Authors

Reviewer_P21NOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces KV-Lingo, a KV cache translation method for switching between language models of different sizes & capabilities without incurring the cost of re-prefilling the entire context. The method is a two-stage process: 1) per-layer linear projectors are first initialized using closed-form squared error minimization between the KV cache of the two models, and 2) are futher trained using next-token KL divergence. During inference, the learned translator is used to seamlessly switch between served models without heavy re-prefilling cost. The paper evaluates the proposed method on a comprehensive set of benchmarks and settings.

Critical strengths
  • The central idea is well-motivated, and the idea of applying KL loss to train the projector seems novel.
  • The proposed self-distillation idea produces large gains over closed-form reconstruction.
  • Experiments are comprehensive, including various benchmarks, diverse model pairs, cross-tokenizer translation, etc. Further, it seems the authors show the complete results, including negative ones, rather than hiding them, which I believe is very valuable.
Critical weaknesses
  • Translators need to be trained for each direction: with N models, N^2 set of projectors need to be trained. Further, new translators would have to be trained when either of the model pair is updated.
  • The results are not very strong. It seems in most cases, small-to-large translation is not useful. In Figure 2 for both shown settings, the performance after the S-L translation is not better than the small model performance, suggesting that translation does not yield any gain. Also, reading per-benchmark scores in Table 9, 14, etc., it seems the improvement is not consistent across benchmarks, and often the scores are worse than not translating at all.
  • In Figure 3, I believe showing the “small model, own cache” would make the benefit clearer. Also, I wonder whether showing TTFT only here is appropriate, since a big part of the cost that needs to be paid by translating from small to larger model is the increased decoding latency.
  • Large-to-small translation results seem stronger and practical in real scenarios, but the gain is still not consistent. Unless I’m mistaken, we should expect the L-S scores to be better than the small model’s own cache score, since otherwise there should be no reason to use the larger model at all.

Some of my concerns may stem from a misunderstanding of the paper’s intended setting or interpretation of the results, and I am open to discussing further with the authors, and updating my assessment accordingly.

Questions for authors

Please see the weaknesses.

3 comments

We thank the reviewer for their comments and careful observations, and for their openness to discussion. We clarified the intended setting, added the requested baselines and costs, and toned down some claims.

  • W1. "Translators need to be trained for each direction"

    We agree, and the Limitations now say so (p. 10). In practice, however, deployment needs translators only for the directed switches it actually uses, which can be far fewer than N(N − 1). In our experiments, it takes <19 GPUh of self-distillation to train a translator, which is orders of magnitude lower than the cost of training the models. Composing translators through a shared model and warm-starting them after an update is an interesting future work direction.

Still, we believe the N=2N=2 case we, and several other related works, tackle here is of practical relevance.

  • W2. "small-to-large translation is not useful."

    Thank you for this remark. We now clarify that the useful S→L cases are clearest when the large model reasons (Table 34, App. P):

    • With reasoning on, S→L recovers 66 ± 6% of the gap between the two models on 0.6B→1.7B and 74 ± 3% on 0.6B→8B (45–74% on four of six pairs). On the six benchmarks scored in both modes, it recovers 26% and 35% with reasoning off (matched-subset analysis, App. P).
    • Without reasoning, the gain is partial: 0.6B→8B scores 0.577 on the nine-benchmark mean, against 0.506 for the 0.6B and 0.811 for the 8B (§4.3). On LongMemEval, it lands above the 0.6B at every length (Fig. 3).
    • Some settings do not benefit: RULER from the 0.6B (Fig. 44), and several heterogeneous pairs, where reasoning-off S→L falls below the small model (Table 34). We toned down the abstract and the introduction accordingly, and per-benchmark results remain mixed.

    We also clarified the role of S→L (§2.2): a router decides when the large model is needed, and KV-Lingo makes that switch cheap. The switch pays off only where the translated large model beats staying small, as in the cases above.

  • W3. "In Figure 3, I believe showing the "small model, own cache" would make the benefit clearer. Also, I wonder whether showing TTFT only here is appropriate, since a big part of the cost [...] is the increased decoding latency."

    This is a great idea. Fig. 3 now shows the small model on its own cache (open circles), on LongMemEval. RULER, where Qwen3-4B is already at the ceiling and 0.6B→8B stays below the 0.6B, is in Fig. 44 and these results are explained in §4.3. Fig. 45 (App. Y) also charges decoding, as a fixed-output-length estimate: the time to first token plus m − 1 decoding steps, measured on device.

    • Staying with the small model is the fastest option in every cell.
    • KV-Lingo's first token comes 18.6–77.3× sooner than re-prefilling for 4B→8B, and 13.7–43.7× sooner for 0.6B→8B.
    • It saves 17–72% of the total time for a 128-token answer, and 3–24% for a 1024-token one.

    Switching therefore pays off when the large model's quality is worth its slower decoding. That holds for LongMemEval from Qwen3-0.6B, but not for RULER, nor from Qwen3-4B.

    With reasoning on, we also estimated the total time of each scored generation from its realized output length and on-device timings (new Table 47). For 0.6B→8B, decoding dominates, so the translated 8B costs about as much as the re-prefilling 8B while keeping most of its gain: on GSM8K, 0.871 in 22.2 s against 0.895 in 21.9 s (0.699 in 5.4 s for the 0.6B, at the 4k-token budget of Fig. 23). On LongMemEval's 15k-token prompts, it scores 0.543 against 0.763 (8B) and 0.383 (0.6B), in 9.9 s against 13.5 s (8B) and 2.1 s (0.6B), trading-off accuracy for latency, as expected.

[Continued response]

  • W4. "we should expect the L-S scores to be better than the small model's own cache score, since otherwise there should be no reason to use the larger model at all."

    Our initial wording invited this reading, and we fixed it in §2.2. In L→S, the large model has already done useful work on the prefix: earlier turns, a plan, or a shared context. Generation then moves to the small model because it is cheaper per token or because the large one is needed elsewhere. These reasons exist independently of KV-Lingo, which only removes the small model's re-prefill. The baseline is therefore the small model on its own cache, and the goal is to match it. On the shape-preserving pairs, the benchmark-average score stays within one percentage point of native-small decoding (Table 34), although task-specific losses remain, e.g. RULER 0.782 against 0.839 and IFEval 0.530 against 0.578 into Qwen3-0.6B (Table 10).

    Gains over the small model appear once the large model has reasoned (Table 40). After the 1.7B has produced a reasoning trace, KV-Lingo largely preserves the benefit of passing that trace to the 0.6B, while avoiding the second prefill:

    0.6B answering after the 1.7B's traceMMLU-ProARC-CGSM8K
    native prefill of prompt + trace0.5080.8750.848
    KV-Lingo translation of prompt + trace0.4730.8700.862
    0.6B alone, reasoning on0.3190.6900.641

On MMLU-Pro, both trace rows also benefit from a format effect worth about 6 points (Table 40), so they do not show better reasoning than the 1.7B itself (0.473). GSM8K uses a 2k-token budget here, against 4k in Table 47.

We thank you again for your insights, which help us improve the paper. Please let us know if anything is still unclear.

Dear authors,

Thank you for the thorough and thoughtful response. The updated manuscripts resolves many of my concerns.

I see that in practical scenarios, N tends to be small (2~3), and that training a translator is not very costly. Also, the clarification and additional results showing when each direction of KV-Lingo is beneficial, and when it is not, make the strengths and limitations of the method much clearer. Overall, although I think the overall performance of the method is somewhat mixed, it does show clear benefits in specific scenarios, and the experimental scope of the paper is very extensive, which makes it a valuable scientific contribution for the community and future work. I therefore increase my score to 3.

Reviewer_ydq4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces KV-Lingo, to translate a source model's KV cache into a target LLM's format. This helps in situations where e.g. the conversation might switch from one LLM to another of a similar format, and offers efficiency gains over re-prefilling the context. This can help in small to large / large to small LM cache transation, translation between different models of similar architectures, dense to MoE cache translation in compute-bound settings. KV-Lingo uses per-layer, token-wise linear maps for keys and values, initialised with a closed-form least-squares fit and then fine-tuned by self-distillation against the target’s own next-token distribution. The authors train one translator per model pair on generic chat data. They evaluate it across a wide range of settings, including Qwen3 and Gemma pairs, unequal depths, a cross-tokenizer pair, MLA, dense->MoE, and multi-turn switching. They report that translated caches roughly match the smaller model’s quality and that time to first token after a switch drops by 5–29×.

Critical strengths

(S1): The reconstruction vs behaviour finding is useful. A translation is useful insofar as it leads to the target model decoding similar sequences as it would with its own prefilled cache. This relaxes the problem from reconstructing the specific target model cache to providing a cache that allows the target model to decode similarly. The experiments support this, demonstrating that closed-form translation stages can fall short.

(S2): Extensive and thorough empirical analysis. The experiments go beyond a single model family. They include a from-scratch pair to ablate Qwen3-specific artefacts, analyses on head-alignment that explain when head-wise maps suffice, and multilingual out-of-distribution tests. The paper is also honest about the strengths and limitations of the approach, noting seed collapses and negative results openly. The authors are also upfront about the cost constraint (the translator must be cheaper than the prefill it replaces) and is stated up front and actually shapes the design.

Overall the method is simple and the analysis is thorough, which leads me to lean towards acceptance. The insights are valuable even if in total the contribution is incremental over existing work.

Critical weaknesses

(W1): Some quality results don’t support the main use case. The paper motivates S->L as "hand off to the stronger model when needed." With reasoning off, the large model reading a translated small cache mostly performs like the small model. E.g. on 0.6B->8B, the mean is 0.58, against 0.51 for the small model and about 0.8 for the large one (fig 2b). On several 0.6B->1.7B benchmarks, translation falls below the small model itself (table 9). The Gemma and cross-tokenizer pairs show the same pattern.

Long-context retrieval is a bigger problem on unequal-depth pairs. Going 8B->0.6B, RULER falls to 0.29–0.58 depending on ν, while the 0.6B on its own prefill scores 0.83 (table 13). That is a poor trade, given that the small model's prefill is cheap anyway. The reasoning-on gains are interesting but rest on fewer benchmarks. I'd like to see the fraction of the native gap recovered reported as a headline metric. The abstract’s claims may need to be toned down.

(W2): Some efficiency claims are overstated, and key baselines are missing.

  • MoE speedup. The 9.6× M3 Ultra figure assumes the 4B model already holds the cache. Section 4.6 pitches the setup as a prefill accelerator, where Table 24 shows only 1.8× at 64 tokens and no gain at 1k. In that same setting, the MoE decoding from a translated cache scores below the dense 4B on all six short-context "other" benchmarks with reasoning off (table 22). It seems that just using the 4B would be faster and better?

  • The H100 speedups divide vLLM TTFT by PyTorch matmul (timed outside vLLM). The cost of writing the translated cache into paged blocks or re-applying RoPE and k-norm and any cross-device transfer is not included.

  • The multi-turn comparison is against full re-prefill of the whole history. Under the memory budget (one cache per model), the natural baseline may be an incremental prefill of only the new span since that model last spoke.

  • Simple hybrids, such as translating and then natively re-prefilling the last few hundred tokens, could also close much of the long-context gap. They aren’t explored.

The other point is that the novelty seems incremental. Per-layer linear maps from another model’s cache, trained through the frozen target, closely resemble Horton et al. (2024). While stage 1 is the closed-form approach of Heo et al. and Qu et al. Two-phase reconstruct->behavioural training mirrors Chen et al. (2026).

I think the genuine contributions are showing the use of KL self-distillation instead of CE, the demonstration that linear maps work well, and the breadth of evaluation. That is acceptable for an empirical paper, but the contribution should be positioned that way.

Questions for authors

(Q1): For each pair, what fraction of the native S->L gap is recovered, with reasoning off and on? Can the authors localise the loss? For example, if the target receives translated caches for only some layers and native caches for the rest, which layers matter most?

(Q2): How does KV-Lingo compare, in quality and switch cost, against incremental prefill of only the new span in the multi-turn experiments?

2 comments

We thank the reviewer for their comments and careful observations. We ran the requested measurements, corrected the timing claims and narrowed some quality claims.

  • W1. "fraction of the native gap recovered reported as a headline metric."

    Table 34 (App. P) now reports, for every pair, the fraction of the gap between the two models that S→L recovers:

    • With reasoning on, 45–74% on four of six pairs: 66 ± 6% on 0.6B→1.7B and 74 ± 3% on 0.6B→8B. The translated cache scores more than one SE above the small model in 27 of 40 pair–benchmark evaluations (a descriptive count).
    • With reasoning off, at most a third on the full nine-benchmark suite, and less than the small model on several heterogeneous pairs. On the six benchmarks scored in both modes (App. P), reasoning still raises it from 26 to 66% (0.6B→1.7B) and from 35 to 74% (0.6B→8B).
    • Long context is the weakest task group everywhere, at most 42% even with reasoning on.
  • W2, MoE. "just using the 4B would be faster and better?"

    Indeed, we agree for the 4B: it prefills almost as slowly as the MoE. Since the goal is only a faster MoE prefill, we rewrote §4.6 with new experiments with a Qwen3-1.7B prefiller (Table 1; App. N, Tables 30–31):

    • On an M5 Ultra, the MoE's first token comes 2.2–2.9× sooner than after its own prefill (64 to 8k tokens), and 4.7–14× sooner when the 1.7B already holds the context.
    • With reasoning on, the MoE reading the 1.7B's cache beats the 1.7B alone by 7–13 points on ARC-C, GSM8K and IFEval, and matches the MoE on its own cache on GSM8K (0.859) and RepLiQA (0.935 against 0.938). It stays below it on IFEval and CoQA.
    • With reasoning off, it falls 9 points below the 1.7B, so the use case is reasoning-on (§4.6).

    We think this is a clearer use case, and thank you for the suggestion.

  • W2, H100. "The cost of writing the translated cache into paged blocks or re-applying RoPE and k-norm [...] is not included."

    We now include fused normalization/RoPE, paged KV writes, and the client-observed cost of a vLLM cache-hit request; the last component is measured with repeated prompts, as a proxy for serving from the imported cache (App. Y, Fig. 43, Table 48).

    • Translation plus the write take 2.8–3.6 ms at 2k tokens and 22–29 ms at 32k.
    • The switch reaches the first token 1.5–3.2× sooner than an uncached re-prefill at 2k, and 4.9–17.6× sooner at 32k.
    • An NVLink copy adds 0.6–12 ms; reloading the cache from host memory removes the gain on the 0.6B/1.7B pair.

    The main text now uses on-device times, which have no serving overhead.

  • W2, hybrids. "Simple hybrids, such as translating and then natively re-prefilling the last few hundred tokens [...] aren't explored."

    We ran this on the pair you mention, Qwen3-8B→0.6B (ν = 2): KV-Lingo translates the first n − k tokens, then the 0.6B prefills the remaining k ∈ {0, 128, 512} tokens (App. T, Table 41). Recomputing up to 512 suffix tokens does not substantially recover the RULER gap. On device at 32k, the first token takes about 235 ms for either k, against 142 ms for translation alone and 1.9 s for re-prefill.

  • W3. "Per-layer linear maps [...] closely resemble Horton et al. (2024)"

    We agree this is an empirical paper and position it as such (§1). Horton et al. train an auxiliary model whose predictions are never used (Table 2, App. A). We also narrowed the objective claim: matching the frozen target's output distribution makes the linear maps effective, and is more robust than cross-entropy on our continuation corpus (App. U), while Chen et al. train on target-generated traces. Chen et al., Heo et al., Li et al. and Qu et al. are concurrent (App. A).

  • Q1. "If the target receives translated caches for only some layers and native caches for the rest, which layers matter most?"

    New App. M.2 (Table 26), Qwen3-0.6B→1.7B, reasoning-off: the 1.7B recomputes its first j of 28 layers and KV-Lingo translates the rest.

    j = 0j = 12j = 20j = 241.7B
    MMLU-Pro0.2380.2690.3260.3670.379
    ARC-C0.4670.4820.6670.7750.770
    GSM8K0.6560.7110.8120.7730.820
    RULER 8k0.8460.8590.9490.9800.981

    Recomputing only early layers gives limited recovery, and not monotonically. Most recovery occurs once the native prefix extends through 20–24 layers, so this repair gives back most of the prefill saving (j/28 is only its recompute share).

  • Q2. "How does KV-Lingo compare, in quality and switch cost, against incremental prefill of only the new span [...]?"

    See §4.7 and App. Q (Tables 35–36). On a cache kept native, incremental prefill matches full re-prefill up to numerics. On device, translation beats incremental prefill from 16-token spans, and the latter protocol cuts switch latency by 4%, 20% and 40% on the three dataset.

We thank you again for your comments and insights, which help us improve the paper.

Thank you for addressing my comments. I will maintain my score since I am leaning towards acceptance and I believe the paper's empirical findings are interesting and should be presented at ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.