- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The Raftel method is proposed in this paper to remove the accuracy loss when per-passage KV caches are reused in RAG serving. The analysis points to errors in the low-frequency RoPE bands of independently cached keys. These errors lead to spurious attention peaks for the question tokens, and replacing the keys with fresh ones recovers the accuracy. Each passage is encoded offline against the stored caches of its neighbours before any query arrives, at the same storage cost as a cold cache, and a small online recompute step is kept for the residual peaks. The authors test the method on seven models and three multi-hop QA benchmarks.
Critical strengths
I think the diagnosis part is also a contribution, not just the method. The authors find that the error from isolated encoding is in the low-frequency rotary bands, and the oracle swap show that keys are the decisive component. Another interesting finding is about recompute: if recompute the tokens which cold question really attend to, half of the gap is recovered, but recompute the tokens which fresh question would attend to give almost no improvement. Also they observe an inverse correlation, when the repaired cache become more accurate, its attention match the fresh attention less. In my opinion this is useful for other works in this direction, because it means using "match the fresh attention" as objective may be wrong.
Critical weaknesses
I feel the neighbourhood setting in main table is quite close to best case. In the benchmark, the corpus is just the twenty passages retrieved for the same question, so neighbours of one passage are exactly the other passages of this question, and the multi-hop chain is already inside the group. But in a real deployed corpus, the neighbours need to be found by token overlap or embedding similarity from a large corpus, and Table 7 show this only give small gain. My concern is, the headline result (bare warm cache match the best prior method with no online work) is exactly the number which depend on the neighbours most. The authors are honest about this point and they measure the gap by themselves, but the main table still use the benchmark protocol.
Questions for authors
-
For a real deployed corpus, what is the corresponding setting of the benchmark neighbourhood protocol? For example, adjacent chunks from same document, neighbours retrieved together, or token-overlap neighbours from the whole corpus?
-
How big is the difference between your reproduced results of CacheBlend, ProphetKV and QCFuse and the numbers reported in their original papers? And can the check with the official QCFuse data release explain this difference?
-
Do you plan to release the code? The neighbour rule and recompute rule have many implementation details, and it is hard to verify them only from the text.
-
The diagnosis is shown on Qwen3-8B, where the effect is most clear. Are the four observations (low-frequency error concentration, phantom peaks, repair asymmetry, inverse correlation) also consistent on the other six models? Is there any architecture where the situation is different?
3 comments
Thank the authors to give detailed setting of the neighborhood and baseline difference. To claim that Raftel works on every model, you may need more experiments to prove it or at least give some concrete theoretical proof, especially regarding to various model architectures.
Author Response to Reviewer UUzQ
Thank you for the follow-up. You are right that "works on every model" was too broad. We meant the broad set of model types we evaluated on:
- Five families from 7B to 32B: Qwen3 (8B, 14B, 32B), Llama-3.1-8B, Mistral-7B-v0.3, GLM-4-9B and Falcon3-10B.
- Different rotary designs: GLM-4 rotates only half of each head. Qwen3 normalises queries and keys before the rotation.
- With 15% recompute Raftel is at or above every published rule with 26% recompute on all seven (Table 2).
- The four observations hold on three of the families (App. G).
Why we expect it to transfer. Raftel needs two properties. First, a stored key must move to a new position without loss. This holds for any rotary embedding, full or partial. Second, a chunk's keys and values must depend on the text before it. This holds for every causal decoder, and warming supplies that text offline. Neither property involves the feed-forward layers, so mixture-of-experts models meet both. This is an argument and not a proof.
Given the variety of architectures we cannot claim that Raftel works on every model. To the best of knowledge, our coverage is still wider than that of prior blending methods. CacheBlend and ProphetKV each report three models and QCFuse four, from at most three families.
Author Response
Thank you for the careful reading and for seeing the diagnosis as a contribution in its own right. Summary:
W (neighbourhood setting). The result does not rest on the benchmark protocol. We gave each passage one fixed cache warmed before any query with neighbours from a corpus-wide index. F1 on the HotpotQA questions of Table 1 (Qwen3-8B, fresh prefill 0.602):
The revision makes the corpus-wide fixed cache the protocol of Table 1 and keeps co-retrieval as a labelled variant.
Q1 (deployment counterpart). In the benchmark protocol 72% of neighbour pairs are chunks of the same Wikipedia article (36% for random neighbours). The gain does not rest on them (last row above). A deployment finds neighbours in the same document (it reaches fresh prefill on NarrativeQA in Table 7), in links between store entries (A-MEM, Mem0, Zep) or in a corpus-wide index (W above).
Q2 (reproduced vs published baselines). The difference comes from the evaluation protocol and the recompute budget. Our main protocol is the one the QCFuse paper describes: 20 retrieved chunks of 512 tokens per request. It is the most recent published setup and uses the usual RAG chunk size. It is also the harder case. Across 20 chunk boundaries the cold cache loses 0.28 F1 on HotpotQA. QCFuse's revised public data release instead serializes each request into about 10 chunks of ~1,200 tokens, where the cold cache loses only 0.12. We ran every method in one harness on both. The table gives the share of full-prefill F1 on HotpotQA at 26% recompute, since absolute F1 depends on the model:
The data release explains most of the difference. There ProphetKV's rule reaches its reported level. The official CacheBlend code in its own MuSiQue setting also matches its paper (0.282 vs 0.281 for full prefill). On our MuSiQue data the same official code gives 0.228 vs 0.280, so the drop comes from the data, not our reimplementation. On our protocol the same budget must repair a cold-start loss more than twice as large. QCFuse's reported quality also uses twice our budget. App. F adds this table and Sec. 4 states why we use this protocol.
Q3 (code). Yes, upon acceptance: the HuggingFace reference used for every number, the scripts for each table and a vLLM connector. Algorithm 1 and the hyperparameter table in App. H specify both rules.
Q4 (other models). Raftel works on every model. With 15% recompute it is at or above every published rule with 26% recompute on all seven models of Table 2. We measured the four observations on Llama-3.1-8B and GLM-4-9B next to Qwen3-8B.