- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses an important problem in multi-model LLM serving: reusing the KV cache produced by a smaller model when a larger model takes over a request. RaReCache first translates the source cache using a calibrated linear mapping and then selectively recomputes a subset of token positions in the target model. Its main contribution is a target-free selection criterion, rank disagreement, which measures the energy of mapped representations outside the dominant subspace of the calibration predictions.
Critical strengths
- The problem is practically relevant. Model cascades and mid-conversation model switching can incur substantial repeated prefill costs, and selectively repairing a transferred cache is a plausible way to trade computation for accuracy.
- The method is relatively simple and avoids a target-model forward pass for selection at inference time. The decomposition of the score into a full-prediction norm and a low-rank projection norm is straightforward, and the implementation optimizations address meaningful overheads.
Critical weaknesses
- The score measures projection residual energy, but the paper does not establish that this reliably identifies mapping uncertainty or tokens with the greatest recomputation benefit.
- The selector may primarily benefit from avoiding template tokens. Comparisons against content-only random selection and template-filtered attention/error selectors are needed to isolate its contribution.
Questions for authors
- Does rank disagreement retain its advantage when all selectors exclude fixed template and instruction tokens?
- How much performance is lost when using one shared calibration corpus across tasks or changing prompt templates?
- Some figures have layout issues, e.g., Figure 2 extends below the bottom edge of the adjacent text block.
3 comments
Dear Reviewer,
We thank you again for your valuable feedback and insightful questions. Based on your suggestions, we have provided detailed experimental evaluations. We have also uploaded a revised PDF with the following changes:
- We have included a comprehensive selector comparison excluding template and instruction tokens (Table 3, Section 4.3)
- We demonstrated generalization across tasks using one shared calibration corpus for GSM8K and MATH-500 (Table 11, Appendix Section D.3) and LongBench-E (Table 1, Section 4.2)
- We fixed the layout issue in Figure 2
Since the author-reviewer discussion phase ends on November 18 (AoE), we would like to kindly ask whether our responses have addressed your concerns. We are happy to engage in further discussions or provide additional clarifications if needed.
Thank you again for your time and thoughtful review.
Sincerely,
Authors Paper-43343
Thanks for the detailed responses and additional experiments, which substantively address my empirical concerns; I will maintain my positive score.
We thank the reviewer MXsk for their appreciation of our work and insightful feedback. The following is our detailed response to their concerns and questions.
We present the selector comparison using Qwen3-0.6B → 14B in Table 12. TI denotes template and instruction tokens. Rank disagreement retains a significant advantage over other selectors for longer context lengths.
Table 12. Selector comparison for Qwen3-0.6B→14B
Rank Disagreement consistently outperforms content-only random and uniform selection across the evaluated LongBench-E length buckets. On short prompts such as GSM8K and MATH-500, template and instruction (TI) tokens constitute approximately 45–60% of the input, so excluding them concentrates recomputation on question tokens and provides a strong baseline. Content-only uniform selection achieves the highest GSM8K accuracy, though rank disagreement’s is close with 90.3% accuracy. On MATH-500, rank disagreement outperforms random and uniform selection excluding TI by 3.2 and 3.6 pp respectively.
Overall, these results support rank disagreement as an effective selector without explicit template annotations, and with clear long-context benefits over content-only random and uniform selection.
To analyze generalization of the calibration map across benchmarks, we evaluated MATH-500 with a map calibrated on GSM8K train set, MATH-500 train set, and a mix of both. At 30% recomputation with rank disagreement, mixed calibration achieves 73.6% compared with 75.4% for the MATH-500 calibrated map, suffering an accuracy drop of 1.8 pp. The GSM8K map suffers a much higher accuracy drop of 6.6 pp. Therefore, including the target prompt distribution during calibration helps the map generalize, though generalization costs a small drop in accuracy over task-specific calibration.
Table 13. MATH-500 evaluation on Qwen3-0.6B→14B using shared calibration corpus
We agree that projection residual energy alone does not establish mapping uncertainty or identify tokens with the greatest recomputation benefit. It measures the component of a token’s mapped KV representation outside the fitted calibration subspace and serves as a heuristic for prioritizing recomputation.
We provide empirical evidence to support this heuristic. Across multiple evaluated tasks and context lengths, the overall method retains 90–99% of target-model accuracy while recomputing fewer than 50% of tokens. These results show the heuristic is effective, and at a matched budget, it outperforms the alternatives.
Thanks for pointing this out. We will fix the layout issue of Figure 2 in the updated manuscript.