- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses the prefix-dependence bottleneck of KV cache reuse in Retrieval-Augmented Generation (RAG), where directly concatenating independently precomputed document KV caches leads to substantial accuracy degradation. The authors decompose this performance loss into two distinct sources — missing cross-attention between documents, and context-dependent artifacts such as the "attention sink" effect caused by the starting position and empty prefix — noting that existing solutions either incur inference latency via online recomputation or require extra training cost via model fine-tuning or link-token learning, often without clear theoretical justification. To solve this, this paper proposes KVPrior, a fully offline, training-free method that generates prefix-independent KV caches by averaging the KV caches of the same document under multiple randomly sampled irrelevant prefixes from the training set. Grounded in the Bayesian prior of document independence, this approach treats prefix-induced bias as removable noise and cancels out context-dependent artifacts while preserving document-specific semantic information. The paper claims empirical results show KVPrior nearly recovers full-prefill accuracy on document-independent RAG tasks, outperforming training-based baselines like KVPacket and online recomputation methods with no inference overhead. It also claims that on multi-hop reasoning benchmarks such as HotpotQA, KVPrior delivers moderate but notable improvements alone, achieving state-of-the-art performance when combined with existing recomputation approaches.
Critical strengths
(1) Simple but elegant solution: The decomposition of prefix dependence into cross-attention loss and contextual artifacts provides a clearer analytical framework for the field. KVPrior is elegantly simple: averaging KV caches over random prefixes yields a prefix-invariant cache representation without model modification, extra training, or inference-time computation cost. In document-independent RAG scenarios, its accuracy approaches full prefill, outperforming training-based methods like KVPacket and latency-incurring recomputation methods. This training-free approach has strong practical value for large-scale industrial RAG deployment and establishes a simple yet powerful baseline for future work.
(2) Composable design: KVPrior is fully complementary to existing recomputation-based methods: it removes contextual artifacts offline, while recomputation restores cross-document attention online. Their combination further boosts accuracy on multi-hop reasoning tasks. The paper unifies this logic under a Bayesian “prior + posterior update” perspective, offering a general conceptual framework where different technical routes can be stacked and optimized. The evaluation covers 4 mainstream models and 5 datasets with varied characteristics, including data-leakage controls and empirical validation of the theoretical hypothesis, supporting robust conclusions.
Critical weaknesses
(1) Lack of experimental data: Key quantitative results are either presented as vague placeholders or entirely missing from the paper. For instance, Table 1, which is designed to quantify the relative impact of the Hessian term across four models and five benchmarks, only contains placeholder labels with no actual numerical values to validate the core theoretical claim. Most performance comparisons, such as the accuracy gaps between KVPrior, KVPacket, and recomputation-based methods, are only described qualitatively in prose rather than reported with exact F1 scores, TTFT measurements, or statistical significance metrics. This absence of detailed, reproducible quantitative data materially undermines the credibility of the paper’s empirical conclusions.
(2) Theoretical justification lacks rigor and mechanistic depth: The Taylor expansion-based proof rests on the claim that the Hessian term’s influence is negligible, but this is only supported by indirect empirical metrics rather than rigorous theoretical derivation. The paper does not explain why noise from random prefixes is orthogonal to the Hessian direction, nor does it characterize the model configurations, document lengths, or layer depths at which this approximation breaks down, leaving the underlying mechanism not fully understood.
(3) Offline precomputation and storage overhead is not systematically quantified: While the paper highlights zero inference-time overhead, it fails to measure or analyze the offline compute and storage costs of generating multiple KV cache copies per document. For large-scale RAG systems with massive document corpora, repeated prefills linearly increase preprocessing time and memory footprint. The paper does not discuss the trade-off between offline cost and online latency savings, nor compare the total cost of ownership against methods that only require a single prefill per document.
Questions for authors
(1) Please first refine and complete the paper by supplementing all missing experimental result data. Sufficient detailed, reproducible quantitative data is critical for readers to independently verify the claimed improvements and assess the robustness of the proposed method.
(2) The paper argues that averaging KV caches removes prefix noise while preserving document semantics. Is there direct quantitative evidence of hidden-state similarity (e.g., cosine similarity) between the averaged cache and the corresponding document’s KV cache from full prefill? How do semantic preservation and noise reduction scale with the number of sampled prefixes?
(3) Control experiments show that using out-of-distribution prefixes degrades KVPrior’s performance, implying that prefix distribution match matters. Have the authors explored more targeted prefix sampling strategies (e.g., sampling from the target task’s document or query distribution)? Would such strategies further improve cross-domain performance?
(4) Please address points (2) and (3) in the Critical Weaknesses part.
Clarification: an incorrect PDF was submitted
Dear Reviewer 2rzx,
Thank you for reading our submission and kindly giving us a detailed review despite finding its incomplete version.
Regarding question (1), as we stated in the general reply to all reviewers, the initial submission you saw (the PDF with "BBB!") is the wrong PDF file we accidentally uploaded. Please find the latest PDF that we uploaded when the rebuttal phase opened. We appreciate it if you could reevaluate our submission based on this intended version.
Regarding question (2),
In our intended submission, we did not exhaustively and quantitatiely evaluate such similarity across all model&benchmark combinations, but we performed these experiments: First, in the section 5.2, we empirically measured the total variation distance between these distributions:
The result shows that TV(A∣∣B), TV(A∣∣AFP8), TV(A∣∣AINT8) are comparable, thus the effect of second-order term is within an acceptable magnitude.
While this experiment is originally intended to justify the approximation in Section 4.1, TV(A∣∣B) also answers your question: Since A is the original full-prefill that is then marginalized over the prefix, TV(A∣∣B) being small indicates that the semantics is preserved.
Second, in the appendix section A2, we made a t-SNE plot that visualizes
We did not include the exhaustive set over different model x benchmarks, but we have these results that all show the similar characteristics: Individual prefill tends to have a KV cache that is far from the mean.
To see how the number of samples affect this, you could subsample 10 points from the 30 points in the plot, and imagine their average --- the average of 10 points would not be subsantially different from the average of 30 points.
One direction for future work is to dynamically adjust the number of samples online until it passes the confidence interval for the Gaussian distribution. This is possible via anytime-valid statistical methods (e.g. confidence sequences).
Regarding question (3),
The answer is yes, but this is outside the scope of the paper. We explored replacing the uniform sampling with a simple bm25-based and embedding-based scorer, and observed a minor accuracy improvement.
Regarding question (4) --- Weakness (2),
The approximation does not catastrophically break down, as we discussed in the TV distance comparison experiment in Section 5.2.
Appendix section A2 (t-SNE plot of the KV cache) shows that no-recompute (prefill with empty prefix) tends to deviate further away from the population mean as it goes deeper in the layer, but the number of tokens from the beginning of the document does not appear to influence it in a sense that it always makes it deviate from the population mean regardless of the token position.
Regarding question (4) --- Weakness (3),
To compute the average over N prefixes, we do not need N resident copies in the GPU memory. The only extra temporary storage necessary is one additional copy of the KV cache that stores the running mean μ; After obtaining the KV cache Ck for the k-th prefix, we can update μ as μ←μ⋅kk−1;μ←μ+k1⋅Ck. In addition, this can also be done in the CPU.
After generating the KVPrior cache, we free all prefills generated with random prefixes. Thus, the long-term storage cost is equivalent to individual prefill.
Thank you, Authors