- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
CacheRepair trains a lightweight external network to predict cross-chunk KV residuals for a frozen LLM. It combines stale-cache features and token embeddings with block-causal attention, achieving competitive quality–TTFT trade-offs across three models and four datasets.
Critical strengths
Strengths
- The evaluation includes multiple models, quality–latency comparisons, uncertainty estimates, and system-cost measurements.
- KV-error and functional analyses support repair effectiveness; each checkpoint is reused across downstream datasets with explicit overlap screening.
Critical weaknesses
Weaknesses
-
Limited novelty. The method combines standard residual learning and Transformer components. Its advance over prior learned KV methods, such as KV Prediction and Cache-to-Cache, needs clarification.
-
Limited significance and insight. The experiments show that learned repair works in this setting, but offer limited insight into what makes cache errors learnable or how the findings generalize. The contribution remains narrow despite the thorough evaluation.
-
Unequal training budgets. CacheRepair uses 50,000 examples and a larger network; the main KV Packet baseline uses 512 examples. The appendix partly addresses this on one model. Comparisons with comparable training budgets are needed to separate method gains from scale gains.
-
Scaling behavior is not well demonstrated. The paper compares several repairer sizes, but does not clearly show how quality scales with training data, compute, and model capacity. The gains from further scaling remain unclear.
-
Limited long-context validation. Longer-context quality is tested on only one model, one dataset, and 128 requests. Broader length extrapolation beyond 100k and Needle-in-a-Haystack tests are needed to check whether repair preserves critical evidence.
Questions for authors
Questions
-
Clarify the differences from KV Prediction and Cache-to-Cache, and compare with a simpler residual-prediction network to show the value of the proposed architecture.
-
Analyze successful and failed repairs to identify which cross-chunk dependencies the network recovers and which remain difficult, and extract generalizable findings.
-
Extend the KV Packet training-budget comparison to the other target models, and report quality–TTFT curves under comparable data and compute budgets.
-
Vary training-set size, training steps, and repairer capacity, and report downstream-quality curves to demonstrate the gains from scaling each factor.
-
Extend long-context quality tests to more models and tasks. Add NIAH tests that vary context length and evidence position to check retrieval robustness beyond the training range.
2 comments
Dear Reviewer 9h1B,
We sincerely thank you for reviewing our clarifications and additional experiments, as well as for your valuable feedback in improving the manuscript. We clarify our novelty below:
-
CacheRepair is deliberately simple. It requires zero modifications to the frozen target LLM, no complex recomputation heuristics, and no task-specific tuning. It is a lightweight standalone model (e.g., 42M–159M parameters for a 14B model) trained once to repair cached KV states directly, making it a clean and easy-to-deploy technique.
-
CacheRepair is simple yet highly effective. It runs 1.30–6.76× faster than popular recomputation baselines, such as CacheBlend (EuroSys'25) and EPIC (ICML'25), at comparable quality, advancing the Pareto frontier in 11/12 evaluated settings. This delivers a decisive serving advantage over these widely adopted approaches.
Thank you again for your time and thoughtful consideration.
The Authors
Thanks for the clarification and the additional experiments. My main concern remains novelty. The method is a fairly direct application of residual prediction to cross-chunk KV repair, and I see the methodological advance over prior learned KV approaches as too limited to warrant acceptance. I’m keeping my score at 2.