- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that KV cache compression has a blind spot. Existing methods, whether eviction, summary compensation, or offload and recall, all decide what to keep by content relevance to the current query. This is what the paper calls content addressing, and it says a second access mode is missing, temporal addressing along the position stream. The result is a failure named sequential forgetting. When a model must copy a span verbatim, eviction by content keeps the head of the span but discards its continuation, so the copy starts correctly and then breaks mid way.
KVFetch adds a temporal recall channel to any score based compressor without retraining. It has three components. Admission moves low predictability literal spans, such as hex digits and rare identifiers, into a quantized cold tier indexed by position. A trigger detects active copying through a monotone read pointer over attention peaks. Prefetch and swap then recalls positional successors from the cold tier into hot slots of fixed size, so the attention cost stays constant. On RULER 16K, under a control at the same budget, KVFetch raises the average over 13 tasks by 8.43 and recovers the verbatim copy task from 0.8 to 78.4. On LongBench, where no task needs sequential access, the channel stays dormant.
Critical strengths
The failure mode is real and precisely localized. The paper does not just assert sequential forgetting. It gives a diagnostic, sustained copy length, that shows the copy breaks mid span rather than at localization, and it formalizes the condition in Definition 1. In the corner it targets, the validation is careful: a control at the same budget, an ablation of each component, a bootstrap over tasks with confidence intervals that do not overlap, and a correlation of 0.924 between the gain per task and the cold tier recall hit rate. The mechanism clearly works where it is designed to work.
Critical weaknesses
Weakness 1: The problem only exists because the method deletes KV entries, and the fix saves only a tiny slice of them
-
The failure the paper targets only arises because the compressor deletes KV entries and never brings them back. If the entries are kept, the failure disappears. Schemes that choose entries fresh at every decoding step, or that offload them and recall on demand, delete nothing, so an ongoing copy can always reach the next token, and the paper's own numbers trace this gradient: 0.8 when entries are deleted, 8.8 when they are recallable, 99.2 with the full cache. The failure is therefore a consequence of the decision to delete, not a basic limit of content-based compression, yet the introduction frames it as a structural gap in all existing methods.
-
The repair covers very little of what was evicted. The cold tier keeps only 0.36% of the cache, while roughly 79.6% is still deleted for good, and admission is limited to spans that look like rare literal strings. Anything else that must be reproduced word for word, a common phrase or a code block for example, stays lost. The claim of recovering verbatim copying is thus too broad, since deleting entries will always leave some spans that cannot be recovered.
-
I looked for a comparison against simply keeping those spans and could not find one. The spans worth saving can already be identified at prefill, so a simpler design could keep them in the hot cache from the start, within the storage budget the baseline already receives, and no such comparison is reported. The ablations show that each component matters given the chosen design, but they do not show that the design beats this static alternative.
Weakness 2: Almost all the gain comes from one synthetic task, realistic benchmarks show no effect, and the key numbers do not add up
-
The +8.43 gain on RULER at 16K comes almost entirely from one task, copying a synthetic ID in niah single 3, which alone contributes about six of the eight points. The admission rule is tuned to exactly this kind of string. On LongBench the channel barely fires, trigger rates stay between 0.000 and 0.029, and the overall change is +0.33. The paper motivates the method with RAG identifiers, code completion, and legal citations, but reports no measurement of how often such verbatim demand actually occurs in real workloads.
-
The central numbers do not reconcile with the tables. The paper attributes 86% of the residual gap to this failure, but Sec 4.2 assigns it to nine tasks while Sec 4.5 assigns it to four, and no four-task subset reaches 86% under the paper's own Table 4. The vt score appears as 95.89 in Table 4 and 98.16 in Table 14 for the same configuration. The headline gain is also measured against the 22.9% budget control while the individual task numbers use the 20% host, which makes the composition of the gain impossible to verify.
-
The method scales poorly in the regime it targets. The gain over the host shrinks as context grows, from +14.3 at 4K to +10.6 at 16K to +6.9 at 32K, and KVFetch's own average drops from 64.88 at 16K to 49.87 at 32K. All efficiency numbers come from a single stream with no batch size and no serving engine, so the reported 4.8% overhead says little about deployment cost.
Overall, the motivation caught my interest, but the algorithm design and the evaluation cases left me disappointed. The idea might fit a different target better. Helping modern sparse attention methods that use indexers to improve model accuracy, for example, looks more promising than staying in the KV eviction corner.
Questions for authors
Question 1: The admission signal is unigram frequency rank, known at prefill, and the spans it selects occupy 0.36% of the context. A simpler baseline could keep those spans in the hot tier at the same total budget, with no trigger, pointer, or prefetch. This baseline is never compared. If static retention already recovers the copy tasks, the dynamic machinery is unnecessary.
Question 2: The method activates only on the synthetic needle tasks in RULER. On LongBench the trigger fires on at most 2.9% of decoding steps. What fraction of realistic long context traffic, for example RAG or code completion, actually enters copy mode? Without such a measurement there is no evidence the problem exists outside RULER.
Question 3: A few numbers in the paper do not add up. Section 4.2 says nine tasks account for 86% of the residual gap, while Section 4.5 credits the same 86% to four tasks, and Table 4 caps any subset of four tasks at 63%. The same default configuration reports vt as 95.89 in Table 4 but 98.16 in Table 14. The abstract also claims nine of thirteen tasks improve, though Table 4 shows eleven positive deltas. How should these be read?
Question 4: Table 9 shows the gain over the host shrinking as context grows, +14.3 at 4K, +10.6 at 16K, +6.9 at 32K, while long context is the stated target. What happens at 128K/256K?
Question 5: The paper states the overhead applies only during active copying, yet the attention peak is extracted on every decoding step and the measured average cost is +4.8%. What is the overhead on a workload that never enters copy mode?
Question 6: Sequential forgetting only occurs in a compressor that physically deletes entries and never brings them back. The paper's own numbers show that recallable offload largely avoids the failure, since Quest reaches 8.8 where plain eviction scores 0.8, and modern sparse attention with indexers deletes nothing at all. Why are recallable or dynamic reselection methods not used as baselines, and could the temporal channel instead help such systems recover accuracy?
I thank the authors for the new experiments, but my central concern remains.
At the same attention cost, static retention nearly matches KVFetch, and the authors describe it as a strong, simpler baseline. The dynamic machinery adds only about two points to the task average, which weakens the case that a temporal channel is necessary.
The gain still concentrates on copy-like tasks, and comprehension traffic stays dormant. I therefore maintain my rating of 2.