- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes KVRelay, a lightweight translator that allows a target model to reuse another model’s KV cache without re-prefilling the context. Both models remain frozen, and the translator is trained by matching the target’s attention outputs rather than its KV tensors. Experiments across five model pairs show improved accuracy over direct cache reuse and Ridge mapping, with reduced time to first token.
Critical strengths
The paper provides useful evidence that a lightweight translator can make existing KV caches usable by another frozen model. Attention-output supervision consistently outperforms direct KV reconstruction across the tested settings. The five-seed results and measured translation costs support the findings within the evaluated scope.
Critical weaknesses
1.The comparison with existing learnable cache translation methods is limited. Previous work has also explored using transferred caches without requiring the receiver to re-encode the source context.¹ ² Although these methods differ in architecture and training objectives, a clearer discussion and controlled comparisons would help establish what the proposed design adds.
2.The length evaluation is limited. Main experiments cap contexts at 512 tokens, while the tests up to 4096 tokens repeat short passages. These settings do not fully reflect the complexity of long documents, ongoing conversations, or agent handoffs discussed in the paper. This scope is understandable for an exploratory study, but the evidence for these applications remains preliminary.
3.The supervision ablation lacks a clean comparison with language-modeling cross-entropy or output KL distillation. Earlier work already uses language-modeling or generation losses to train cache translators.¹ ² The mixed objective in this paper combines attention alignment, KL, and context contrast, so its lower performance does not establish an advantage over either alternative alone. Comparing these objectives with the same translator and training budget would better isolate the contribution of attention-output supervision.
References: [1] Lucio M. Dery et al. Latent Space Communication via K-V Cache Alignment. arXiv:2601.06123, 2026. https://arxiv.org/abs/2601.06123v1 [2] Siyi Chen et al. See What I See, Know What I Think: Dense Latent Communication Across Heterogeneous Agents. arXiv:2606.13594, 2026. https://arxiv.org/abs/2606.13594v1
Questions for authors
-
Could you clarify the main advantages over the prior cache translation methods cited above and provide a comparison under the same model pairs and evaluation protocol, including translation cost?
-
How does attention-output supervision compare with standalone language-modeling cross-entropy and output KL distillation when the translator architecture and training budget are fixed?