acceptodds
Under review as a conference paper at ICLR 2027

CacheRepair: Learning to Repair Cross-Chunk Context in RAG for KV Cache Fusion

Abstract

Multi-document retrieval-augmented generation (RAG) requires a language model to process multiple retrieved text chunks before answering a question. Precomputing each chunk's KV cache independently and concatenating the caches when the chunks are retrieved can accelerate this step. However, the assembled cache lacks cross-chunk attention information, reducing answer quality. Selective recomputation methods recover the missing cross-chunk context by rerunning the target LLM on selected tokens, incurring substantial online computation. We introduce CacheRepair, a lightweight network that learns the difference between independently computed KV caches and those produced by processing the chunks together. The network combines compressed KV features with token embeddings and uses attention that is bidirectional within each chunk and flows from earlier to later chunks. Each repair block receives the compressed cache features, and the predicted residual is added to every document token's cache. Each repair network is trained for a specific frozen target LLM on a generic retrieval corpus and reused across downstream datasets. Our analysis shows that repair reduces KV errors both near chunk boundaries and throughout chunk interiors. Evaluation across three target LLMs and four downstream datasets places CacheRepair on the measured answer-quality–latency Pareto frontier in eleven of twelve model–dataset combinations. Reported time to first token (TTFT) includes online cache transfer and repair. Across all twelve combinations, the largest repairers achieve 1.69–4.61 speedups in median TTFT over full prefill and improve mean F1 by 2.1–26.1 percentage points over direct cache reuse.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, KV Cache Fusion, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9h1BOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Summary

CacheRepair trains a lightweight external network to predict cross-chunk KV residuals for a frozen LLM. It combines stale-cache features and token embeddings with block-causal attention, achieving competitive quality–TTFT trade-offs across three models and four datasets.

Critical strengths

Strengths

  • The evaluation includes multiple models, quality–latency comparisons, uncertainty estimates, and system-cost measurements.
  • KV-error and functional analyses support repair effectiveness; each checkpoint is reused across downstream datasets with explicit overlap screening.
Critical weaknesses

Weaknesses

  1. Limited novelty. The method combines standard residual learning and Transformer components. Its advance over prior learned KV methods, such as KV Prediction and Cache-to-Cache, needs clarification.

  2. Limited significance and insight. The experiments show that learned repair works in this setting, but offer limited insight into what makes cache errors learnable or how the findings generalize. The contribution remains narrow despite the thorough evaluation.

  3. Unequal training budgets. CacheRepair uses 50,000 examples and a larger network; the main KV Packet baseline uses 512 examples. The appendix partly addresses this on one model. Comparisons with comparable training budgets are needed to separate method gains from scale gains.

  4. Scaling behavior is not well demonstrated. The paper compares several repairer sizes, but does not clearly show how quality scales with training data, compute, and model capacity. The gains from further scaling remain unclear.

  5. Limited long-context validation. Longer-context quality is tested on only one model, one dataset, and 128 requests. Broader length extrapolation beyond 100k and Needle-in-a-Haystack tests are needed to check whether repair preserves critical evidence.

Questions for authors

Questions

  1. Clarify the differences from KV Prediction and Cache-to-Cache, and compare with a simpler residual-prediction network to show the value of the proposed architecture.

  2. Analyze successful and failed repairs to identify which cross-chunk dependencies the network recovers and which remain difficult, and extract generalizable findings.

  3. Extend the KV Packet training-budget comparison to the other target models, and report quality–TTFT curves under comparable data and compute budgets.

  4. Vary training-set size, training steps, and repairer capacity, and report downstream-quality curves to demonstrate the gains from scaling each factor.

  5. Extend long-context quality tests to more models and tasks. Add NIAH tests that vary context length and evidence position to check retrieval robustness beyond the training range.

2 comments

Thanks for the clarification and the additional experiments. My main concern remains novelty. The method is a fairly direct application of residual prediction to cross-chunk KV repair, and I see the methodological advance over prior learned KV approaches as too limited to warrant acceptance. I’m keeping my score at 2.

Dear Reviewer 9h1B,

We sincerely thank you for reviewing our clarifications and additional experiments, as well as for your valuable feedback in improving the manuscript. We clarify our novelty below:

  • CacheRepair is deliberately simple. It requires zero modifications to the frozen target LLM, no complex recomputation heuristics, and no task-specific tuning. It is a lightweight standalone model (e.g., 42M–159M parameters for a 14B model) trained once to repair cached KV states directly, making it a clean and easy-to-deploy technique.

  • CacheRepair is simple yet highly effective. It runs 1.30–6.76× faster than popular recomputation baselines, such as CacheBlend (EuroSys'25) and EPIC (ICML'25), at comparable quality, advancing the Pareto frontier in 11/12 evaluated settings. This delivers a decisive serving advantage over these widely adopted approaches.

Thank you again for your time and thoughtful consideration.

The Authors

Reviewer_sSSJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Multi-document retrieval-augmented generation (RAG) requires a language model to process multiple retrieved text chunks before answering a question. Precomputing each chunk's KV cache independently and concatenating the caches when the chunks are retrieved can accelerate this step. However, the assembled cache lacks cross-chunk attention information, reducing answer quality. To develop an efficient solution, the paper introduces CacheRepair. The paper identifies structured boundary-local and layer- persistent stale-KV error patterns that motivate repair across document positions. The paper further develop a learned network that combines stale-cache features, token identity, and chunk-structured attention to repair KV for a frozen LLM. The empirical results show that the developed approach can improve the quality at low latency.

Critical strengths

(1) The paper is related to a very interesting, well-motivated and practical setting. To address the challenges, the paper developed an approach with clear intuitions.

(2) The paper is very polished, with all the sections very well organized. It is very easy to understand the paper and the contributions.

Critical weaknesses

(1) It seems that more comprehensive evaluations can be done in this paper. It is not clear why the paper only evaluates 500 requests from each of four downstream datasets. The developed approach can be also tested using other model families. TTFT could be measured using various GPUs.

(2) From the perspective of using the approach in practice, developing a Lightweight Repair Network might make it more difficult to use this approach, compared to the baselines. For example, it seems that this Lightweight Repair Network can not be shared across different models. If the user fine-tunes the LLMs, it seems that the Lightweight Repair Network also needs to be updated. It is unclear how robust the develop approach is in the scenario when the LLMs needs to be updated. Some discussions or evaluations related to this could be very helpful.

Questions for authors

(1) It is not clear why the paper only evaluates 500 requests from each of four downstream datasets. The developed approach can be also tested using other model families. TTFT could be measured using various GPUs. It would be great to add some discussions or evaluations.

(2) How robust is the develop approach in the scenario when the LLMs needs to be updated?

Reviewer_EZLuOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CacheRepair, a lightweight network that predicts residual corrections to independently precomputed KV caches, restoring cross-chunk context without online target-LLM recomputation. It combines compressed KV features, token embeddings, and block-causal attention while keeping the target LLM frozen. Experiments on three LLMs and four QA datasets demonstrate improved answer quality over direct cache reuse and competitive quality–TTFT tradeoffs against existing KV fusion methods.

Critical strengths
  1. A practical and well-motivated approach to KV cache fusion. The paper addresses an important efficiency problem in multi-document RAG. Replacing online target-LLM recomputation with learned residual prediction offers a meaningful alternative to existing selective recomputation methods. The analysis of cross-chunk KV errors provides useful motivation for repairing representations beyond chunk boundaries.

  2. Comprehensive evaluation with supporting analysis. The paper evaluates three target LLMs, four downstream datasets, and several relevant baselines. Beyond answer quality and TTFT, it analyzes KV reconstruction errors, attention-output deviations, and latency components. These results provide evidence that the proposed repair mechanism improves both intermediate representations and downstream generation.

Critical weaknesses
  1. The efficiency benefits under strict quality constraints remain insufficiently established. Although CacheRepair contributes Pareto-frontier points in 11 of 12 settings, this comparison is based on point estimates over the tested configurations. Under the half-prefill latency budget, only four settings show statistically supported advantages over all eligible baselines (Table 3). Moreover, the largest Qwen2.5-14B repairer loses 7.21 and 8.38 F1 points relative to Full Prefill on MuSiQue and HotpotQA, respectively (Table 11). These results demonstrate useful quality–latency tradeoffs, but leave unclear how much acceleration remains when tighter quality requirements are imposed.

  2. The latency advantage may be sensitive to baseline implementation and cache-transfer overhead. Some recomputation configurations are slower than Full Prefill, while their dense computation is substantially lower. For example, at 16K tokens, CacheBlend 0.1 uses approximately 13% of Full Prefill's dense MACs but reaches 52% of its TTFT (Table 19). The original CacheBlend system also overlaps layer-wise KV loading with recomputation, but it is unclear whether comparable optimizations are preserved in the evaluated implementation. Since CacheRepair requires KV features from all target-LLM layers, its relative advantage may also depend on cache residency and transfer scheduling. A clearer accounting of these implementation differences is needed to establish how robust the reported efficiency gains are.

Questions for authors
  1. Could the authors summarize the minimum achievable TTFT under fixed F1-loss budgets of 1, 2, and 5 percentage points relative to Full Prefill, using the existing configurations? This would help clarify the benefits under stricter quality requirements, particularly on MuSiQue and HotpotQA.

  2. Why do some CacheBlend and InfoFlow configurations incur relatively high latency despite much lower dense computation than Full Prefill? Does the implementation preserve layer-wise overlap between KV loading and recomputation, as in the original CacheBlend system? Could the authors clarify the relevant overheads and their impact on the Pareto comparison?

  3. Table 2 shows only small differences among block-causal, token-causal, and bidirectional attention, as well as between full reinjection and entrance-only conditioning. Are these differences consistent across training seeds? Furthermore, Appendix F shows that the block-diagonal variant substantially perturbs the first chunk. Would preserving its original first-chunk KV recover some of the lost answer quality, helping distinguish missing cross-chunk interaction from this additional perturbation?

  4. APE provides another training-free approach to combining independently encoded contexts by modifying attention behavior. Is it excluded because such attention modifications fall outside the intended deployment interface? Clarifying this distinction would help establish the comparison boundary relative to other frozen-weight cache reuse methods.

Comments on the paper

1 comment

Dear Reviewers,

The authors have provided their rebuttal, but no follow-up comments have been posted yet. As the discussion period is ending soon, please review and acknowledge the rebuttal, and engage in further discussion if any concerns remain unresolved.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.