- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, a framework CachePort is proposed for reusing KV caches across independently pretrained LLM families without requiring a full prefill by the receiver model. CachePort tackles three challenges: tokenizer mismatches are handled by character-boundary alignment; representation differences are addressed through CKA-guided layer routing and affine KV translation; and residual errors are mitigated by receiver adaptation and selective recomputation. Experiments across Llama, Qwen, and Mistral demonstrate lower perplexity than prior transfer methods, improved downstream task performance, and serving efficiency gains for sufficiently long contexts.
Critical strengths
- This paper targets an important and challenging systems problem: cross-family KV cache replacement without a full receiver prefill and without modifying pretrained model weights. Its combination of content-anchored token alignment, representation translation, and receiver-side adaptation offers a practical and reasonably complete solution.
- The experiments cover multiple model families and show consistent perplexity gains over the evaluated transfer baselines. The component ablations provide useful evidence for the key design choices, and the vLLM integration demonstrates practical latency benefits for sufficiently long contexts.
Critical weaknesses
- As acknowledged in Appendix E, several baselines are simplified reimplementations rather than faithful reproductions. Some comparisons also rely on different execution paths, and the results do not include multiple training seeds or comparable training-cost accounting. As a result, the claimed superiority over existing methods is not yet fully established under a comprehensive and reproducible protocol.
- Query-selected refresh improves multi-hop QA performance but is evaluated only offline, without accounting for its probing and scheduling overhead. In addition, substantial quality gaps remain relative to full prefill in certain directions (e.g., HotpotQA F1 of 0.360 versus 0.500 for full prefill on Q7 to L8). The deployed latency gains are also context-length dependent, so the practical benefits for latency-sensitive or high-quality-demanding workloads remain insufficiently demonstrated.
Questions for authors
For cross-family pairs with non-monotonic layer correspondences, such as L8 to Q7, how much does CKA-guided routing improve over proportional-depth routing at equal refresh budgets?
2 comments
I only used AI tools to translate and polish my reviews. Therefore, the content of my reviews remains unchanged.
Thank you for your responses and additional experiments. I keep my score.
Rebuttal
We thank the reviewer for the insightful feedback, particularly the suggestion to test CKA routing on non-monotonic pairs. Below, we present this multi-pair routing ablation alongside controlled baseline comparisons, multi-seed variance and cost accounting (W1), and end-to-end vLLM latency and throughput measurements for both refresh modes (W2). We denote Llama3.1-8B, Qwen2.5-3B, Qwen2.5-7B and Mistral-7B-v0.3 as L8, Q3, Q7 and MI, respectively
Q1. How much does routing matter across model pairs?
CKA-guided routing recovered a larger fraction of the floor-to-native PPL gap at zero refresh (B=0) than both proportional-depth and terminal-alignment (C2C-style) heuristics on all three tested pairs:
The margin was 11.3 percentage points on the L8 → Q7 pair in your question and 16.0 points on L8 → Q3. Even on MI → L8 (equal 32 layers, where proportional routing is the identity map), CKA still improved recovery by 5.1 points. For each comparison, we fitted the maps for the chosen routing rule and held the training documents, distillation budget, evaluation examples, and refresh budget fixed.
W1. Baseline fairness, seeds, and training cost
W2. When is query-selected refresh useful, and how does the deployed path scale with length?
Query-selected refresh is useful for information-sensitive tasks where the output depends on a few exact mid-context tokens (e.g., extractive multi-hop QA), since contiguous tail refresh only updates the end of the prompt and never touches those decisive spans. On the short Q7 → L8 HotpotQA setting you identified, selecting query-attended windows raised F1 from 0.300 to 0.370 relative to tail refresh (and from 0.391 to 0.451 on L8 → Q7), though probing and sequential recomputation made native prefill faster at 1.4K. At longer prefixes 32K–64K, the same mechanism absorbs probing and scheduling overheads while still outperforming native prefill: in vLLM, it achieves 1.5–2.8× higher concurrent throughput across 16K–64K and 1.16–2.2× lower single-request latency at 32K–64K, while improving RULER QA accuracy over tail refresh.
Thank you again for your review and suggestion. We hope these results and clarifications fully address your concerns and allow you to view the paper more favorably.
Reference [1] Heo, Taekyung, et al. "Cross-model KV cache transfer in LLM families: A closed-form linear mapping for prefill reuse." arXiv preprint arXiv:2608.03893 (2026).