acceptodds
Under review as a conference paper at ICLR 2027

No Passage Is an Island: Orphaned KV Caches, Phantom Peaks, and Warm Encoding for Retrieval-Augmented Generation

10 views

Abstract

Retrieval-Augmented Generation improves Large Language Models by enabling document-grounded generation, however it requires recomputing the per-passage prefill at each retrieval time. As such, the systems cache each passage's KV states once and assemble the retrieved ones at query time. Current methods reduce this overhead by reusing the per-passage KV cache of prior prefills, so that generation can skip the prefill of retrieved passages. The assembled cache, however, reduces prediction quality, and prior systems repair it online by recomputing roughly a quarter of the tokens on every query. In this work, we take a different approach. We identify where the error comes from and remove most of it offline. We find that position is not the problem, but rather that a passage encoded alone is encoded cold, meaning that its keys are wrong in the low-frequency rotary bands, exactly where the question attends. The question's attention therefore collapses and lands on phantom peaks, tokens that the mismatched question favours and the fresh prefill does not. We introduce Raftel, which works by giving the phantom peaks their true keys, so the tokens to recompute are the ones the mismatched question attends, not the ones the true question would. We therefore warm the passages instead of repairing the cache. Each passage is encoded once against the stored caches of its most similar passages and iterated twice, which removes most phantom peaks before any question arrives. We evaluate our method on LongBench and RULER and find that the bare warm cache matches the prior state of the art's best operating point at 6.6x the speed of full prefill.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
KV cache reuse, KV cache repair, RAG

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UUzQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The Raftel method is proposed in this paper to remove the accuracy loss when per-passage KV caches are reused in RAG serving. The analysis points to errors in the low-frequency RoPE bands of independently cached keys. These errors lead to spurious attention peaks for the question tokens, and replacing the keys with fresh ones recovers the accuracy. Each passage is encoded offline against the stored caches of its neighbours before any query arrives, at the same storage cost as a cold cache, and a small online recompute step is kept for the residual peaks. The authors test the method on seven models and three multi-hop QA benchmarks.

Critical strengths

I think the diagnosis part is also a contribution, not just the method. The authors find that the error from isolated encoding is in the low-frequency rotary bands, and the oracle swap show that keys are the decisive component. Another interesting finding is about recompute: if recompute the tokens which cold question really attend to, half of the gap is recovered, but recompute the tokens which fresh question would attend to give almost no improvement. Also they observe an inverse correlation, when the repaired cache become more accurate, its attention match the fresh attention less. In my opinion this is useful for other works in this direction, because it means using "match the fresh attention" as objective may be wrong.

Critical weaknesses

I feel the neighbourhood setting in main table is quite close to best case. In the benchmark, the corpus is just the twenty passages retrieved for the same question, so neighbours of one passage are exactly the other passages of this question, and the multi-hop chain is already inside the group. But in a real deployed corpus, the neighbours need to be found by token overlap or embedding similarity from a large corpus, and Table 7 show this only give small gain. My concern is, the headline result (bare warm cache match the best prior method with no online work) is exactly the number which depend on the neighbours most. The authors are honest about this point and they measure the gap by themselves, but the main table still use the benchmark protocol.

Questions for authors
  1. For a real deployed corpus, what is the corresponding setting of the benchmark neighbourhood protocol? For example, adjacent chunks from same document, neighbours retrieved together, or token-overlap neighbours from the whole corpus?

  2. How big is the difference between your reproduced results of CacheBlend, ProphetKV and QCFuse and the numbers reported in their original papers? And can the check with the official QCFuse data release explain this difference?

  3. Do you plan to release the code? The neighbour rule and recompute rule have many implementation details, and it is hard to verify them only from the text.

  4. The diagnosis is shown on Qwen3-8B, where the effect is most clear. Are the four observations (low-frequency error concentration, phantom peaks, repair asymmetry, inverse correlation) also consistent on the other six models? Is there any architecture where the situation is different?

3 comments
AuthorsOpenReview ↗

Author Response

Thank you for the careful reading and for seeing the diagnosis as a contribution in its own right. Summary:

  • Neighbourhood: the results hold with one fixed cache per passage from a corpus-wide index (W, Q1).
  • Published numbers: the gap to the original papers comes from the protocol (Q2).
  • Other models: all four observations are measured on two more families (Q4).

W (neighbourhood setting). The result does not rest on the benchmark protocol. We gave each passage one fixed cache warmed before any query with neighbours from a corpus-wide index. F1 on the HotpotQA questions of Table 1 (Qwen3-8B, fresh prefill 0.602):

cachebare+ 15% recompute
cold0.3180.452
warm with co-retrieved neighbours (submission)0.5310.588
warm with one fixed cache per passage (corpus-wide index)0.5240.574
warm, co-retrieved, no same-article neighbours0.5050.559

The revision makes the corpus-wide fixed cache the protocol of Table 1 and keeps co-retrieval as a labelled variant.

Q1 (deployment counterpart). In the benchmark protocol 72% of neighbour pairs are chunks of the same Wikipedia article (36% for random neighbours). The gain does not rest on them (last row above). A deployment finds neighbours in the same document (it reaches fresh prefill on NarrativeQA in Table 7), in links between store entries (A-MEM, Mem0, Zep) or in a corpus-wide index (W above).

Q2 (reproduced vs published baselines). The difference comes from the evaluation protocol and the recompute budget. Our main protocol is the one the QCFuse paper describes: 20 retrieved chunks of 512 tokens per request. It is the most recent published setup and uses the usual RAG chunk size. It is also the harder case. Across 20 chunk boundaries the cold cache loses 0.28 F1 on HotpotQA. QCFuse's revised public data release instead serializes each request into about 10 chunks of ~1,200 tokens, where the cold cache loses only 0.12. We ran every method in one harness on both. The table gives the share of full-prefill F1 on HotpotQA at 26% recompute, since absolute F1 depends on the model:

methodoriginal paperQCFuse data releaseour protocol
CacheBlendwithin 0.02 F1 of full prefill (15% recompute)93%65%
ProphetKVabout 95% on HotpotQA (20% recompute)96%82%
QCFuse"full-prefill-level quality" (50% recompute)92%79%

The data release explains most of the difference. There ProphetKV's rule reaches its reported level. The official CacheBlend code in its own MuSiQue setting also matches its paper (0.282 vs 0.281 for full prefill). On our MuSiQue data the same official code gives 0.228 vs 0.280, so the drop comes from the data, not our reimplementation. On our protocol the same budget must repair a cold-start loss more than twice as large. QCFuse's reported quality also uses twice our budget. App. F adds this table and Sec. 4 states why we use this protocol.

Q3 (code). Yes, upon acceptance: the HuggingFace reference used for every number, the scripts for each table and a vLLM connector. Algorithm 1 and the hyperparameter table in App. H specify both rules.

Q4 (other models). Raftel works on every model. With 15% recompute it is at or above every published rule with 26% recompute on all seven models of Table 2. We measured the four observations on Llama-3.1-8B and GLM-4-9B next to Qwen3-8B.

observation and comparisonQwen3-8BLlama-3.1-8BGLM-4-9B
1. key error: highest vs lowest frequency band0.27 vs 0.490.23 vs 0.350.30 vs 0.61 (rotated half)
2. attention match with fresh prefill: cold cache vs fresh low-frequency keys0.23 vs 0.940.49 vs 0.820.21 vs 0.74 (unrotated half)
3. F1 of a 26% recompute chosen by cold vs fresh attention0.521 vs 0.4690.491 vs 0.4540.424 vs 0.385
4a. attention match: fresh keys vs 26% recompute1.00 vs 0.500.89 vs 0.770.99 vs 0.59
4b. F1: fresh keys vs 26% recompute0.411 vs 0.5210.366 vs 0.4910.400 vs 0.424
  • (1) holds on all three. The key error grows toward low frequencies.
  • (2) The misdirection holds on all three. Fresh low-frequency keys alone restore most of the attention. On GLM-4 its unrotated half does this. On Qwen3-8B the misdirection concentrates on phantom peaks (0.44 of the question's attention) and warming removes most of it. On Llama it is spread more evenly.
  • (3) holds on all three. Recomputing where the cold question looks beats the oracle choice from fresh attention by 0.04 to 0.05 F1. The revision states Observation 3 with this gap.
  • (4) holds on all three, and we sharpened it. Fresh keys match fresh attention best but answer worse than a 26% recompute. Matching fresh attention is therefore the wrong objective, as you note. Observation 4 now states the part that holds for every cache construction: attention similarity does not rank caches.

Thank the authors to give detailed setting of the neighborhood and baseline difference. To claim that Raftel works on every model, you may need more experiments to prove it or at least give some concrete theoretical proof, especially regarding to various model architectures.

AuthorsOpenReview ↗

Author Response to Reviewer UUzQ

Thank you for the follow-up. You are right that "works on every model" was too broad. We meant the broad set of model types we evaluated on:

  • Five families from 7B to 32B: Qwen3 (8B, 14B, 32B), Llama-3.1-8B, Mistral-7B-v0.3, GLM-4-9B and Falcon3-10B.
  • Different rotary designs: GLM-4 rotates only half of each head. Qwen3 normalises queries and keys before the rotation.
  • With 15% recompute Raftel is at or above every published rule with 26% recompute on all seven (Table 2).
  • The four observations hold on three of the families (App. G).

Why we expect it to transfer. Raftel needs two properties. First, a stored key must move to a new position without loss. This holds for any rotary embedding, full or partial. Second, a chunk's keys and values must depend on the text before it. This holds for every causal decoder, and warming supplies that text offline. Neither property involves the feed-forward layers, so mixture-of-experts models meet both. This is an argument and not a proof.

Given the variety of architectures we cannot claim that Raftel works on every model. To the best of knowledge, our coverage is still wider than that of prior blending methods. CacheBlend and ProphetKV each report three models and QCFuse four, from at most three families.

Reviewer_fVwoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary

Raftel addresses KV cache reuse by encoding each passage offline against its neighbours' cached states. Two rounds of this warm encoding reduce the need for online recomputation. The paper attributes cold-cache failures to incorrect keys that redirect query attention toward "phantom peaks."

Critical strengths

Strengths

  • A useful distinction between tokens containing relevant information and tokens that interfere with attention.
  • A training-free design that moves repair offline without increasing the final cache size.
  • Broad model coverage and useful ablations of neighbour sources, encoding rounds, and selection rules. Offline costs and cache-transfer latency are also reported.
Critical weaknesses

Weaknesses

  1. The main gains depend on neighbour availability that is not sufficiently justified. Section 4 describes corpus-wide, query-blind selection, while Table 7 includes neighbours retrieved alongside the passage. In that ablation, bare-cache F1 rises from 0.273 to 0.444 with co-retrieved neighbours, but only to 0.294/0.322 with independent Wikipedia indices. The same-document NarrativeQA result is encouraging, but does not establish comparable gains for cross-document RAG. The main results need an unambiguous, deployment-consistent neighbourhood protocol.

  2. The causal claims exceed the evidence. Larger errors in low-frequency key components do not establish their causal importance. The fresh-Q/fresh-K interventions are described without sufficient quantitative results or implementation details. Moreover, Figure 9 shows correlation and accuracy increasing together along each recomputation curve. It supports attention similarity being insufficient to compare cache constructions, rather than a general inverse relationship between similarity and accuracy.

  3. The baseline comparison is incomplete and partly simplified. Table 13 uses a fixed recomputation set selected by first-layer KV deviation for CacheBlend, whereas the original method uses gradual filtering across layers (CacheBlend, Section 4.3). This limits conclusions about deviation-based repair. InfoFlow KV, a directly relevant query-attention-based recomputation method, is also missing.

  4. Evaluation size and implementation sensitivity weaken confidence in smaller gains. Several experiments report only a "major subset," and the reported largest-model setting uses 40 examples. Appendix A also reports approximately 0.05 F1 differences between logically identical implementations. Paired confidence intervals are helpful, but do not resolve this implementation discrepancy or establish quality equivalence to full prefill.

The offline approach and diagnostic observations are promising. However, the practical validity of the neighbourhood protocol, the strength of the causal conclusions, and the baseline comparison remain unresolved. I would consider raising my score if these concerns are addressed with convincing evidence.

Questions for authors

Questions

  1. Specify the neighbour construction for every main result. Evaluate one fixed cache per passage, built from a corpus-wide index before test queries arrive, and report the resulting quality-latency curves.
  2. Quantify K-only, V-only, and Q-only interventions and low-/high-frequency key replacement. Define phantom peaks precisely and measure how warming changes them across models.
  3. Compare with the original CacheBlend filtering procedure and InfoFlow KV under matched budgets, and test whether warm encoding improves these baselines too.
  4. Report exact evaluation subsets, expand the small-sample experiments, and explain and resolve the implementation discrepancy. Use a stated quality tolerance when claiming equivalence to full prefill.
4 comments
AuthorsOpenReview ↗

Author Response Part 1

Thank you for the precise review. We answer each point with new results:

  • Neighbour protocol: one fixed cache per passage, from a corpus-wide index built before any test query, with quality–latency curves (Q1).
  • Causal evidence: the numbers behind the Sec. 2 interventions, a precise definition of phantom peaks, and new Q-, K- and V-only and band-limited interventions on three model families (Q2).
  • Baselines: CacheBlend as released and with its paper's gradual filtering, and InfoFlow KV, on cold and warm caches (Q3).
  • Statistics: the full evaluation set everywhere, the source of the implementation spread, and equivalence at a stated margin (Q4).
  • Wording: Observation 4 is revised as you suggest.

W1/Q1 (neighbour protocol). In the submission every main result (Tables 1–4, Figs. 5–6) used one construction. Each passage's 4 neighbours were chosen by token overlap among the passages retrieved with it for that question. The choice reads no query text, but the candidate set is the retrieved set (the co-retrieval row of Table 7). The revision states this in Sec. 4.

The result holds under the protocol you describe. Each of the 3,778 passages gets one fixed cache, warmed offline in two rounds with neighbours from a corpus-wide index before any test query, evaluated on the questions of Table 1. The latency row is measured as in Table 3.

F1 at recompute budget0%10%15%26%
cold0.3180.4160.4520.521
warm, corpus-wide fixed cache0.5240.5680.5740.574
warm, co-retrieved (submitted protocol)0.5310.5750.5880.589
TTFT0.16 s0.42 s0.49 s0.68 s

Fresh prefill: 0.602 F1 at 1.05 s. Under this protocol, Raftel + 15% is statistically indistinguishable from fresh prefill and is significantly above every published method at 26% recompute. The revision makes this protocol the main one in Table 1 and Fig. 5, with co-retrieval as a labelled variant in the other tables.

The revised Table 7 runs on the full evaluation set. The cold cache scores 0.318, co-retrieval 0.531 and the two Wikipedia indices 0.355 and 0.358. A Wikipedia index is a stress test outside Raftel's setting. With a 26% recompute it still stays above the cold cache (0.545 vs 0.521).

W2/Q2 (causal claims). Fig. 1a is correlational. The causal evidence is the Sec. 2 interventions. Their numbers are below, and App. G gives the full table. The interventions show that cached keys misdirect the question, through their low-frequency band.

  • Where the keys matter (Qwen3-8B): fresh K and V at the 8 middle layers 18–25 alone lift F1 from 0.318 to 0.407. That is about as much as the 10 last layers (0.397) and more than the 18 first layers (0.355).
  • Phantom peaks (now defined in Sec. 2, with the thresholds in App. H): a passage's attractors are its top 5% of tokens by the attention they receive within the passage during its cold encoding (layers 18–25). They are query-free and computed offline. Phantom peaks are the attractors that fall inside the question's top 26% on the cold cache.

New: Q-, K- and V-only and band-limited interventions on three families, on the full evaluation set. Each cell is F1 / correlation of the question's attention with fresh attention.

intervention (all layers)Qwen3-8BLlama-3.1-8BGLM-4-9B
cold0.318 / 0.230.325 / 0.490.311 / 0.21
+ fresh Q (question tokens)0.419 / 0.230.439 / 0.580.306 / 0.22
+ fresh K0.411 / 1.000.366 / 0.890.400 / 0.99
+ fresh V0.333 / 0.220.428 / 0.560.309 / 0.22
+ fresh K, lowest-frequency quarter of the pairs0.408 / 0.940.404 / 0.820.301 / 0.58
+ fresh K, highest-frequency quarter0.331 / 0.210.337 / 0.490.318 / 0.21
+ fresh K, unrotated dimensions (frequency zero)——0.413 / 0.74
fresh K and V0.601 / 1.000.506 / 1.000.518 / 1.00
  • Keys decide where the question looks, and the low-frequency band is causal for it. Fresh keys restore the attention on all three. The lowest-frequency quarter alone restores most of it on Qwen3 and Llama, and the unrotated half does on GLM-4. The highest-frequency quarter changes nothing.
  • Accuracy needs keys and values together. No single component reaches fresh prefill. Only K and V together do. On Qwen3 and Llama fresh queries raise F1 while the attention barely moves. They carry the question's state from the fresh pass. Both repairs that work, recompute and warm encoding, rewrite keys and values together.
  • Phantom peaks and warming. On Qwen3-8B, phantom peaks take 0.44 of the question's attention on the cold cache, and both warming and fresh low-frequency keys remove most of it. On Llama the misdirected attention is spread more evenly, and fresh low-frequency keys restore it as well.
AuthorsOpenReview ↗

Author Response Part 2

Figure 9. Along each recompute curve, similarity and accuracy rise together, as you note. The observation concerns comparisons across cache constructions. In Fig. 9 the bare warm cache has a lower similarity than the bare cold cache (0.117 vs 0.220) and a much higher F1 (0.531 vs 0.318). The interventions show the same. On Qwen3-8B fresh keys reach a correlation of 1.00 with fresh attention and an F1 of 0.411. A 26% recompute of the cold cache reaches only 0.50 but an F1 of 0.521. Llama-3.1-8B and GLM-4-9B show the same pattern (App. G).

Observation 4 now reads "attention similarity to fresh prefill does not rank cache constructions", and "inverse relationship" is removed from Secs. 1 and 5.

W3/Q3 (baselines). Our CacheBlend baseline uses the paper's KV deviation with the released default schedule. The gradual filtering of the CacheBlend paper (§4.3) is not the released default, and we now add it. InfoFlow KV is added to Sec. 5 and to the comparison.

Raftel with a 15% recompute beats every published method, even when the method recomputes 26%. Qwen3-8B, HotpotQA, the questions of Table 1, fresh prefill 0.602. The cold-cache columns run each method as published, reproduced on our protocol. None of these methods builds a warm cache, so the warm-cache columns add our contribution to theirs (co-retrieved neighbours).

Methodas published (cold cache), 15%26%on Raftel's warm cache, 15%26%
CacheBlend, as released (layer 1)0.3500.3940.5290.546
CacheBlend, gradual filtering (§4.3)0.3630.3840.5300.539
InfoFlow KV, without chunk reordering0.4480.4960.5770.589
InfoFlow KV, with chunk reordering0.4950.5160.5520.562
Raftel (ours)——0.5880.589
  • Warm encoding lifts every method, by up to +0.16 F1, so the gain comes from the cache, not from the selection rule. Table 8 shows the same for QCFuse and ProphetKV.
  • Gradual filtering does not change the CacheBlend result.

W4/Q4 (sample size, implementation, equivalence).

  • Sample size: every table now runs on the full evaluation set of the QCFuse evaluation scheme (Sec. 4), the same questions as Table 1 on HotpotQA. This includes Tables 2, 5, 7, 8 and 9 and Qwen3-32B on all three QA tasks and RULER (Tables 12 and 13). The main results hold on the full set, and no caption says "a major subset" any more.
  • Implementation spread: the ~0.05 F1 in App. A was measured on a small pilot set between two recompute implementations that differ only in the bf16 SDPA tiling. A different query-block shape changes the reduction order, which flips near-tied greedy tokens. All arms of a comparison run in one implementation, and the paired intervals compare them question by question. The implementation we will release gives the same answers as the harness on every arm of a pilot set.
  • Equivalence: we replace "within noise" with two one-sided tests at one margin of δ = 0.05 F1 for all tasks, on the paired bootstrap 90% interval of Raftel + 15% minus fresh prefill (full evaluation set per task, co-retrieved neighbours):
TaskΔ F190% intervalverdict at δ = 0.05
HotpotQA−0.015[−0.039, +0.006]equivalent
MuSiQue+0.007[−0.041, +0.055]non-inferior
2WikiMQA+0.026[−0.026, +0.080]non-inferior

Thanks for testing one fixed cache per passage. The HotpotQA result addresses part of my concern, but the broader cross-model results still rely on co-retrieved neighbours. The evidence under the fixed-cache setting is therefore too limited to support the broader deployment claims. I’m keeping my score at 2.

Thank you for the reply. We agree that the cross-model accuracy in Table 2 uses co-retrieved neighbours. The fixed-cache evidence is broader than one result.

  • Two fixed-cache settings. We test a corpus-wide index on HotpotQA (W1/Q1) and the chunk's own document on NarrativeQA (Table 7). Both build one cache per passage before any query. The own-document cache brings the bare cache to fresh prefill (0.250 vs 0.247).
  • Fixed and co-retrieved caches are close. On HotpotQA the corpus-wide fixed cache stays within 0.015 F1 of co-retrieval at every budget.
  • The speed-up does not depend on the neighbours. Warming runs offline, and the online path is the same for every neighbour source. A 15% recompute is 2.0 to 2.7× faster than fresh prefill on all seven models (Table 3). Our vLLM connector serves 2.4× more requests per second under load (Table 18).

Our deployment claim covers corpora with related neighbours, as the Limitations section states. These are the main uses of cached retrieval:

  • Codebases of 100k to over 1M lines. A function's neighbours are its file and the files it imports.
  • Document stores such as libraries, legal archives and internal company documents. A chunk's neighbours are the rest of its document.
  • Memory systems such as Zep, A-MEM and Mem0, which store links between entries.

In each case the neighbours come from the corpus's own structure, as in the NarrativeQA setting. We do not claim gains without related neighbours (Table 7, Wikipedia rows).

Reviewer_HE5HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the accuracy degradation caused by independently prefilling and reusing per-passage KV caches in retrieval-augmented generation. The authors argue that the problem is not simply positional relocation. Instead, when a passage is encoded in isolation, its keys in the low-frequency RoPE bands of semantic layers differ from those produced in the complete context, causing question attention to collapse onto spurious "phantom peaks." Based on this diagnosis, the paper proposes Raftel. Offline, each passage is encoded with a small set of related passage caches as context for two rounds, producing a warm cache. Online, an optional repair step recomputes a small fraction of tokens selected according to the attention produced by the mismatched question. Experiments cover HotpotQA, MuSiQue, 2WikiMQA, RULER, and seven model configurations. On HotpotQA, the bare warm cache matches or exceeds strong cold-cache repair methods while achieving 6.6 times the TTFT speed of full prefill. With 15% recomputation, performance in the main paired comparison is statistically indistinguishable from full prefill.

Critical strengths

The strongest aspect of the paper is the close connection among the observed system failure, the proposed mechanistic explanation, and the resulting method. The analysis of low-frequency RoPE key errors, attention divergence in semantic layers, and phantom peaks provides a more interpretable account of cache-reuse failure than a purely heuristic selection rule. The finding that repair should recompute the tokens attended to by the mismatched cache, rather than those preferred by the fresh cache, is particularly interesting and counterintuitive. The experiments cover several QA benchmarks, RULER tasks, seven model configurations, and multiple recomputation budgets. Comparisons with CacheBlend, QCFuse, and ProphetKV are conducted within a shared harness. The paper additionally reports confidence intervals, TTFT both with HBM-resident caches and CPU transfer, and broad ablations of neighborhood construction and warming rounds.

Critical weaknesses

Random or generic-corpus neighbors provide little benefit, so its applicability to open, dynamic, or highly heterogeneous RAG corpora is uncertain. In the benchmark setting, neighbors are selected from the evaluation corpus using lexical overlap and may offer cleaner topical co-occurrence than would be available in a real retrieval index. Offline warming costs approximately 2.7 times a cold prefill per chunk, while the costs of cache updates, index changes, and continuous corpus ingestion are not fully evaluated. The efficiency evaluation uses a single-card, unbatched research harness and does not report throughput, concurrency, or end-to-end latency including retrieval and decoding. The main mechanistic figures are largely based on one model, and some largest-model experiments use only approximately 40 examples, limiting the statistical strength of the generalization claims.

Questions for authors
  1. For a real RAG index containing millions or billions of passages and receiving continuous updates, what are the amortized costs of neighbor search, two-round warming, and cache maintenance?
  2. How does the method perform when related passages are not from the same document or when the retrieval distribution shifts substantially?
  3. Could the authors report throughput and latency in a batched, concurrent, end-to-end serving environment that includes retrieval, CPU-to-HBM transfer, and decoding?
  4. Could the benchmark corpus structure make the neighborhood construction overly favorable? What happens when neighbors are selected from an independent external corpus or a fixed production index?
  5. Can the causal claim that low-frequency key errors produce phantom peaks be further tested through frequency-band interventions, additional model families, and larger sample sizes?
2 comments
AuthorsOpenReview ↗

Author Response

Thank you for the careful review. Summary:

  • Raftel runs in vLLM. Our KV connector for vllm serve will be open-sourced after acceptance. With the caches in host memory, Raftel + 15% cuts the time to first token 2.3× on Qwen3-8B and 3.4× on Qwen3-32B at 16k tokens, and serves 2.4× the requests per second of fresh prefill under load (W4).
  • Neighbours: the gain holds with one fixed cache per passage from a corpus-wide index, and with every same-article neighbour excluded (W2, Q2).
  • New experiments: a frequency-band intervention on three model families, and Qwen3-32B on the full evaluation set (W5).

Setting. Raftel targets RAG over a corpus already on hand (knowledge bases, codebases). Each chunk is encoded once, offline, and reused by every query that retrieves it, as in CacheBlend, ProphetKV and QCFuse. Sec. 1 now states this scope.

W1 (random neighbours). Related neighbours are the method's one assumption, and corpora on hand provide them: a chunk's own document (it reaches fresh prefill on NarrativeQA, Table 7) and the links a store keeps between entries, as in A-MEM, Mem0 and Zep. Even unrelated neighbours warm the cache a little (Table 5). The floor is then our selection on a cold cache, which is at or above the published rules on most models (Tables 1 and 2).

W2/Q4 and Q2 (where the neighbours come from). In the submission, neighbours were chosen without the question, by token overlap among the passages retrieved together. The revision gives each passage one fixed cache, warmed before any query with neighbours from an index over all passages of the benchmark. Sec. 4 now states this. We also excluded every neighbour from the same Wikipedia article. The gain holds in both (Qwen3-8B, the HotpotQA questions of Table 1, F1, fresh prefill 0.602):

Cachebare+ 15% recompute
cold0.3180.452
warm, co-retrieved neighbours (submission)0.5310.588
warm, one fixed cache per passage, corpus-wide index0.5240.574
warm, co-retrieved, same-article neighbours excluded0.5050.559

The fixed cache with 15% recompute also beats the cold cache with 26% (0.521), and the revision makes it the protocol of Table 1. An external index over 400k Wikipedia paragraphs, with the retrieved passages excluded, is a stress test outside Raftel's setting. With a 26% recompute it still stays above the cold cache (Table 7). Under a query shift, nothing offline depends on the queries, and the online recompute follows the actual question's attention.

W3/Q1 (offline cost, updates, scale). Ingestion is paid once, offline (Table 6). Warming takes 0.20 s per chunk, 0.13 s more than a cold encode. A million chunks warm in about 55 accelerator-hours, and warming pays for itself after about 5 retrievals of a chunk. Storage equals a cold cache. Neighbour search is free with document adjacency, or one k-NN query per new chunk. A new chunk is warmed against the stored caches of its neighbours, so an insert costs only the new chunks and existing caches stay valid. A chunk can be re-warmed when its neighbourhood changes, at the same 0.20 s.

W4/Q3 (serving). Table 3 already includes the host-to-HBM copy. vLLM copies the caches faster than our research harness (0.07 s vs 0.17 s on Qwen3-8B) and recomputes faster, so its speed-up is larger than in Table 3. The vLLM connector supports continuous batching and tensor parallelism. Under load the advantage grows (Qwen3-8B, HotpotQA, caches in host memory, fresh prefill → Raftel + 15%):

Concurrent clients1416
Requests per second1.03 → 2.141.09 → 2.541.09 → 2.65
Mean time to first token0.88 → 0.38 s2.26 → 0.72 s11.8 → 3.3 s
Mean end-to-end latency, 48 tokens0.97 → 0.47 s3.60 → 1.55 s13.2 → 5.9 s

Retrieval runs before the connector and is the same for every method. App. J gives the full results.

W5/Q5 (mechanism, sample size). Sec. 2 and App. G now report the causal interventions on the questions of Table 1. Fresh keys alone restore the question's attention (correlation with fresh attention 0.23 → 0.997), and fresh queries do not. New:

  • Band intervention. Fresh keys in only the lowest-frequency quarter of the rotary pairs remove most phantom-peak mass, restore the attention and give the same F1 as all fresh keys. The highest-frequency quarter changes nothing (Sec. 2).
  • Two more families. On Llama-3.1-8B and GLM-4-9B, the key error also grows toward low frequencies, and fresh low-frequency keys restore the attention (on GLM-4, its unrotated half). App. G gives the full table.
  • Qwen3-32B now runs on the full evaluation set of Tables 1 and 4, on all three QA tasks and RULER (revised Tables 2, 12 and 13).
  • Every table now runs on the full evaluation set, and the main results hold. Following your summary, the paper now says that Raftel + 15% is statistically indistinguishable from fresh prefill.

Thank you for conducting extra empirical studies. These additional results solved most of my concerns. I will raise my score to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment

We thank the reviewers for careful and constructive reviews. They valued the link between the observed failure, its mechanism and the method, and saw the diagnosis as a contribution in its own right. All three asked how the neighbors are chosen and whether the mechanism holds beyond one model. We ran the new experiments and revised the paper. All changes in the revised PDF are in blue.

Neighbors without the question (R1, R2, R3). Each passage now gets one fixed cache, warmed offline before any test query with neighbors from an index over all passages of the benchmark. The gain holds it even holds with every neighbor from the same Wikipedia article excluded. This corpus-wide protocol is now the main one in Table 1 and Fig. 5. Co-retrieval stays as a labelled variant in the other tables, and Sec. 1 states the setting Raftel targets.

Full evaluation set (R1, R2). Every table now runs on the full evaluation set, including Qwen3-32B on all QA tasks and RULER. The main results hold. With a 26% recompute, Raftel is statistically indistinguishable from fresh prefill on all seven models of Table 2. We test equivalence at one stated margin, and App. A explains the implementation spread.

Causal evidence on three families (R1, R2, R3). New interventions refresh only the queries, the keys or the values, or only one frequency band of the keys, on Qwen3-8B, Llama-3.1-8B and GLM-4-9B. Fresh keys restore where the question looks, and their low-frequency band alone restores most of it. Accuracy needs keys and values together. All four observations hold on the three families (App. G). Observation 4 now states that attention similarity does not rank cache constructions. Phantom peaks are now defined in Sec. 2, and Algorithm 1 and App. H give every rule and threshold.

Baselines (R2, R3). We add CacheBlend with the gradual filtering of its paper, and InfoFlow KV. Warm encoding lifts every selection rule. Raftel with a 15% recompute is at or above every published method with 26% on all seven models. App. F traces the gap to the published numbers to the evaluation protocol, not to our reimplementation.

Serving and cost (R1). A vLLM KV connector runs Raftel with continuous batching and tensor parallelism. It cuts the time to first token by up to 3.4× and serves more than twice the requests per second of fresh prefill under load (App. J). Warming is paid once, offline. A new chunk is warmed against the stored caches of its neighbours, so existing caches stay valid.

Code. The HuggingFace reference used for every number, the scripts for each table and the vLLM connector will be released upon acceptance.

Dear reviewers,

A reminder that the discussion period will end on November 18 at 11:59pm AoE. I encourage you to interact with the authors' rebuttal to clarify any remaining issue (and how well the rebuttal addressed your concerns) before the next phase.

Thanks, The AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.