acceptodds
Under review as a conference paper at ICLR 2027

KVFetch: Temporal Prefetching for the Missing Half of KV Cache Compression

Abstract

As context windows scale to tens or hundreds of thousands of tokens, KV cache compression has become essential for efficient LLM inference. Existing methods fall into three families: score-based eviction, summary compensation, and offload-and-recall. Yet all three decide what to keep or recall by content relevance to the current query. We show this shared design is structurally incomplete. A cache supports two access modes: associative lookup by content and sequential traversal by position; current compressors implement only the first. The gap matters in practice: retrieval-augmented generation, code completion, and structured-data extraction all require the model to reproduce identifiers, field values, or code tokens verbatim from the context. Under compression, content-based eviction retains the head of such a sequence but discards its continuation, causing verbatim copying to break irreversibly midway, a failure we call sequential forgetting. This failure resists better scoring, larger budgets, summary compensation, and dynamic re-scoring; it is the dominant source of remaining quality loss under compression. We propose KVFetch, a training-free, drop-in framework that opens a temporal recall channel for any score-based compressor. It demotes evicted candidates to a quantized cold tier, detects active copying through a monotone read pointer, and prefetches positional successors into fixed-size hot-tier slots without increasing attention cost. On RULER-16K under an iso-budget control, KVFetch recovers verbatim copying from 0.8 to 78.4 and raises the 13-task average by +8.4, with gains concentrating on tasks that require sequential access. On LongBench, where no task requires sequential access, the channel remains dormant and imposes no cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache compression, long-context inference, temporal addressing, sequential forgetting, verbatim copying, training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_F2zzOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that KV cache compression has a blind spot. Existing methods, whether eviction, summary compensation, or offload and recall, all decide what to keep by content relevance to the current query. This is what the paper calls content addressing, and it says a second access mode is missing, temporal addressing along the position stream. The result is a failure named sequential forgetting. When a model must copy a span verbatim, eviction by content keeps the head of the span but discards its continuation, so the copy starts correctly and then breaks mid way.

KVFetch adds a temporal recall channel to any score based compressor without retraining. It has three components. Admission moves low predictability literal spans, such as hex digits and rare identifiers, into a quantized cold tier indexed by position. A trigger detects active copying through a monotone read pointer over attention peaks. Prefetch and swap then recalls positional successors from the cold tier into hot slots of fixed size, so the attention cost stays constant. On RULER 16K, under a control at the same budget, KVFetch raises the average over 13 tasks by 8.43 and recovers the verbatim copy task from 0.8 to 78.4. On LongBench, where no task needs sequential access, the channel stays dormant.

Critical strengths

The failure mode is real and precisely localized. The paper does not just assert sequential forgetting. It gives a diagnostic, sustained copy length, that shows the copy breaks mid span rather than at localization, and it formalizes the condition in Definition 1. In the corner it targets, the validation is careful: a control at the same budget, an ablation of each component, a bootstrap over tasks with confidence intervals that do not overlap, and a correlation of 0.924 between the gain per task and the cold tier recall hit rate. The mechanism clearly works where it is designed to work.

Critical weaknesses

Weakness 1: The problem only exists because the method deletes KV entries, and the fix saves only a tiny slice of them

  • The failure the paper targets only arises because the compressor deletes KV entries and never brings them back. If the entries are kept, the failure disappears. Schemes that choose entries fresh at every decoding step, or that offload them and recall on demand, delete nothing, so an ongoing copy can always reach the next token, and the paper's own numbers trace this gradient: 0.8 when entries are deleted, 8.8 when they are recallable, 99.2 with the full cache. The failure is therefore a consequence of the decision to delete, not a basic limit of content-based compression, yet the introduction frames it as a structural gap in all existing methods.

  • The repair covers very little of what was evicted. The cold tier keeps only 0.36% of the cache, while roughly 79.6% is still deleted for good, and admission is limited to spans that look like rare literal strings. Anything else that must be reproduced word for word, a common phrase or a code block for example, stays lost. The claim of recovering verbatim copying is thus too broad, since deleting entries will always leave some spans that cannot be recovered.

  • I looked for a comparison against simply keeping those spans and could not find one. The spans worth saving can already be identified at prefill, so a simpler design could keep them in the hot cache from the start, within the storage budget the baseline already receives, and no such comparison is reported. The ablations show that each component matters given the chosen design, but they do not show that the design beats this static alternative.

Weakness 2: Almost all the gain comes from one synthetic task, realistic benchmarks show no effect, and the key numbers do not add up

  • The +8.43 gain on RULER at 16K comes almost entirely from one task, copying a synthetic ID in niah single 3, which alone contributes about six of the eight points. The admission rule is tuned to exactly this kind of string. On LongBench the channel barely fires, trigger rates stay between 0.000 and 0.029, and the overall change is +0.33. The paper motivates the method with RAG identifiers, code completion, and legal citations, but reports no measurement of how often such verbatim demand actually occurs in real workloads.

  • The central numbers do not reconcile with the tables. The paper attributes 86% of the residual gap to this failure, but Sec 4.2 assigns it to nine tasks while Sec 4.5 assigns it to four, and no four-task subset reaches 86% under the paper's own Table 4. The vt score appears as 95.89 in Table 4 and 98.16 in Table 14 for the same configuration. The headline gain is also measured against the 22.9% budget control while the individual task numbers use the 20% host, which makes the composition of the gain impossible to verify.

  • The method scales poorly in the regime it targets. The gain over the host shrinks as context grows, from +14.3 at 4K to +10.6 at 16K to +6.9 at 32K, and KVFetch's own average drops from 64.88 at 16K to 49.87 at 32K. All efficiency numbers come from a single stream with no batch size and no serving engine, so the reported 4.8% overhead says little about deployment cost.

Overall, the motivation caught my interest, but the algorithm design and the evaluation cases left me disappointed. The idea might fit a different target better. Helping modern sparse attention methods that use indexers to improve model accuracy, for example, looks more promising than staying in the KV eviction corner.

Questions for authors

Question 1: The admission signal is unigram frequency rank, known at prefill, and the spans it selects occupy 0.36% of the context. A simpler baseline could keep those spans in the hot tier at the same total budget, with no trigger, pointer, or prefetch. This baseline is never compared. If static retention already recovers the copy tasks, the dynamic machinery is unnecessary.

Question 2: The method activates only on the synthetic needle tasks in RULER. On LongBench the trigger fires on at most 2.9% of decoding steps. What fraction of realistic long context traffic, for example RAG or code completion, actually enters copy mode? Without such a measurement there is no evidence the problem exists outside RULER.

Question 3: A few numbers in the paper do not add up. Section 4.2 says nine tasks account for 86% of the residual gap, while Section 4.5 credits the same 86% to four tasks, and Table 4 caps any subset of four tasks at 63%. The same default configuration reports vt as 95.89 in Table 4 but 98.16 in Table 14. The abstract also claims nine of thirteen tasks improve, though Table 4 shows eleven positive deltas. How should these be read?

Question 4: Table 9 shows the gain over the host shrinking as context grows, +14.3 at 4K, +10.6 at 16K, +6.9 at 32K, while long context is the stated target. What happens at 128K/256K?

Question 5: The paper states the overhead applies only during active copying, yet the attention peak is extracted on every decoding step and the measured average cost is +4.8%. What is the overhead on a workload that never enters copy mode?

Question 6: Sequential forgetting only occurs in a compressor that physically deletes entries and never brings them back. The paper's own numbers show that recallable offload largely avoids the failure, since Quest reaches 8.8 where plain eviction scores 0.8, and modern sparse attention with indexers deletes nothing at all. Why are recallable or dynamic reselection methods not used as baselines, and could the temporal channel instead help such systems recover accuracy?

1 comment

I thank the authors for the new experiments, but my central concern remains.

At the same attention cost, static retention nearly matches KVFetch, and the authors describe it as a strong, simpler baseline. The dynamic machinery adds only about two points to the task average, which weakens the case that a temporal channel is necessary.

The gain still concentrates on copy-like tasks, and comprehension traffic stays dormant. I therefore maintain my rating of 2.

Reviewer_JPKMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KVFetch to address copying failures caused by KV cache compression. It stores selected evicted KV entries in a quantized cold cache, detects sequential reading through attention trajectories, and prefetches subsequent entries into a fixed-size hot cache.

Critical strengths
  1. The paper identifies a practically relevant failure mode: KV compression can interrupt the exact copying of identifiers and other literal sequences.
  2. The reported accuracy improvements are substantial across several models, suggesting that addressing this scenario can meaningfully improve compressed inference.
Critical weaknesses
  1. Practical deployment remains insufficiently demonstrated. The method introduces candidate selection, a quantized cold cache, attention tracking, and dynamic swaps at each layer. Its integration with production inference systems is unclear, particularly regarding prefix-cache sharing, page placement and updates under paged attention, and whether variable per-step operations can efficiently support CUDA Graphs. The SDPA-based evaluation does not resolve these concerns. This matters because the practical value depends on whether the accuracy gains survive integration into an optimized serving system.

  2. The cost–benefit tradeoff on workloads without copying is unclear. Such workloads may receive little accuracy benefit while still paying for cold-cache storage and per-step monitoring. Although the paper reports little accuracy change on LongBench, this does not establish negligible memory or latency overhead. More workload-specific measurements and guidance on when to enable or disable KVFetch are needed to determine whether it is beneficial for mixed production traffic.

Questions for authors

N/A

2 comments

Thank you for raising the deployment questions. SDPA at batch 1 is not a vLLM / SGLang integration, and the revision does not present it as one.

Weakness 1 — Practical deployment: prefix-cache sharing, paged attention, CUDA Graphs.

SDPA does not settle serving. Quality and the Table 3 latency are measured under SDPA on one RTX 5090 at batch 1. That tests the channel, not prefix-cache sharing, page placement, or graph capture. The design has two states. At Copy=0 the hot-tier size, page set, and attention shape match the host; the only extra work is one single-head QK^T. At Copy=1 we dequantize and swap ptr+1…k (k=8); |H| does not grow.

Prefix-cache sharing. Admission is a deterministic function of the prompt, so a shared prefix shares the same cold-tier index, which can be stored with the shared prefix block. Requests with no candidate runs build no cold tier and reuse the host prefix path.

Page placement and updates under paged attention. Prefetch is a position interval [ptr+1, ptr+k], which maps directly onto pages. Appendix D.4, Table 21 attaches the same trigger / pointer / prefetch to Quest without changing its page scorer: at Copy=0 Quest still selects pages; at Copy=1 the next k=8 pages along the pointer are fetched and displace the current lowest-scoring pages. s3 moves from 8.8 to 44.8. That is a page-level overlay, not a second eviction policy.

CUDA Graphs. We have not measured under graph capture. Both paths have static tensor shapes: Copy=0 matches the host; Copy=1 dequantizes and swaps a fixed k=8, not a variable-length rewrite. What varies is which path is taken, not the kernel geometry. The LongBench step-level trigger is 0.000–0.029; the RULER mix is 0.011. That suggests two graphs dispatched on Copy, or a graph break on Copy=1. The SDPA numbers are not a captured-graph result.

The revision will state this bound in Appendix D.3 (new "Serving boundary" paragraph): quality is measured under SDPA; serving is the host graph plus a rare Copy=1 bypass; graph capture inside a production engine remains a limitation. We do not add a vLLM / CUDA-Graph table and do not read the SDPA numbers as that integration.

Where this is in the revision. Quest wrap: Appendix D.4, Table 21. Idle vs. copy latency, serving boundary: Appendix D.3, Table 20.

Thank you for separating "scores do not fall" from "the overhead is negligible." LongBench +0.33 is a quality result. It is not a memory or latency result.

Weakness 2 - Cost-benefit without copying, and when to enable KVFetch on mixed traffic.

No-copy traffic still pays, but not the +4.8% in Table 3. Same 16K context, same SDPA (Appendix D.3 Table 20, D.7 Table 24). Per-step decode latency, host = 18.7 ms:

  • Peak-only (force Copy=0): 18.8 ms, +0.5% vs. host, no swaps.
  • LongBench (trigger 0.000-0.029): 18.9 ms, +1.1%, cold tier almost unused.
  • RULER-16K mix (Table 3): 19.6 ms, +4.8%, includes copy tasks.
  • s3 only: 20.4 ms, +9.1%, channel on.

With no copy, the bill is the per-step peak (+0.5-1.1%), not +4.8%; the +4.8% is a mix that includes copy tasks. Storage is a 3.25 MiB 4-bit payload, or 3.70 MiB with scale/zero and indices (20.41% of full KV, below the 22.9% iso-budget control). Prompts with no candidate runs build no cold tier, and unused budget is partially refunded to the hot tier. Prefill TTFT is 386 -> 394 ms (+2.1%).

Enabling is per request, after prefill; it does not have to be a model-wide flag.

  1. No unigram candidate runs: no cold tier, no peak; that request matches the host in latency and storage.
  2. Candidates exist but Copy stays 0 (LongBench / RAG-QA): sidecar plus peak, about +1.1%; RAG-QA quality delta is +0.3 (Appendix D.1).
  3. The query asks for an identifier verbatim (RAG-ID / Code-ID): open the channel. Host 11.0 / 21.3, KVFetch 68.5 / 63.3.

Comprehension traffic stays in (1) or (2), near the host. Only class (3) pays for swaps. This is not "turn KVFetch on for all long context"; it is "turn it on when prefill sees a candidate."

The Table 20/24 split is in Appendix D.3 and D.7, and the revision adds a "Per-request enabling" paragraph to D.3 with this switch. Our guidance for mixed production traffic is these three bins, not another LongBench score.

Where this is in the revision: decode split and per-request enabling, Appendix D.3, Table 20. Full storage and TTFT, Appendix D.7, Table 24. RAG-QA vs. RAG-ID, Appendix D.1, Table 18. LongBench dormancy: trigger 0.000-0.029, quality +0.33.

Reviewer_ussGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces KVFetch, a training-free framework for improving KV cache compression in long-context LLM inference. It identifies sequential forgetting, where compression preserves the beginning of a context span but removes its continuation, causing verbatim copying to fail midway. The central contribution is to complement content-based cache selection with temporal recall. KVFetch preserves selected evicted entries in a position-indexed, 4-bit cold tier, detects ongoing copying through a stall-tolerant read pointer derived from attention peaks, and prefetches positional successors through one-for-one swaps into a fixed-capacity hot cache. This preserves the active cache size while enabling recovery of previously evicted continuation tokens.

This paper evaluates KVFetch on RULER at 4K–32K context lengths and on 16 LongBench tasks, using Qwen2.5-7B, Qwen3-4B, and Llama-3.1-8B, alongside comparisons across multiple host compressors. On Qwen2.5-7B at RULER-16K, KVFetch improves the 13-task average from 56.45 to 64.88 relative to a baseline given a slightly larger storage budget. Verbatim-copying performance increases from 0.8 for the 20%-retention host to 78.4. Component ablations examine cold-tier admission, temporal triggering, and prefetching, while sustained-copy-length diagnostics support the proposed continuation-recovery mechanism. The reported implementation uses 20.36% of full-KV storage with approximately 4.8% additional decoding latency over the host, and LongBench performance remains broadly unchanged. The method specifically addresses continuation failures and requires preserved localization anchors and sufficient cold-tier coverage.

Critical strengths

This paper identifies a concrete failure mode of KV cache compression and develops a targeted mechanism to address it. The distinction between locating a relevant span and preserving its sequential continuation is technically meaningful: a model can retrieve the correct prefix yet fail to reproduce the remaining tokens. The sustained-copy-length diagnostic helps substantiate this distinction, while attention-based tracking and successor prefetching provide a coherent remedy. This connection between diagnosis and method makes the contribution more informative than an incremental improvement to token-importance scoring.

KVFetch offers a practical integration strategy supported by controlled comparisons. It requires no model training, complements existing compressors, and maintains a fixed hot-cache capacity. Across multiple model families and host compressors, the reported results show substantial recovery on copying tasks. On RULER-16K, it improves the average score by 8.43 points over a baseline allocated more storage, with approximately 4.8% decoding-latency overhead over the original host. These comparisons, together with component ablations and broadly unchanged LongBench performance, support the value of selectively activating recall rather than simply retaining more KV entries.

Critical weaknesses

The evidence does not sufficiently isolate the contribution of temporal addressing from selective retention and recall. KVFetch combines rare-span admission, a quantized cold tier, a copying trigger, and positional prefetching, but the comparisons do not clearly establish which element distinguishes it from existing recall-based approaches. In the configuration-matched fetch-criterion ablation, query–key similarity actually outperforms positional successors on verbatim copying (89.6 versus 80.0). Both variants retain the temporal trigger, so this does not invalidate copy-aware recall; however, it leaves the specific benefit of positional addressing unresolved. This matters because the paper’s central conceptual claim—that content addressing is structurally insufficient—is stronger than the demonstrated distinction.

The mechanism depends on restrictive coverage and localization assumptions whose practical reliability remains unclear. KVFetch requires the host compressor to preserve enough initial anchors to establish a copying trajectory and the admission heuristic to retain the required continuation in the limited cold tier. It cannot recover missing localization or correct an incorrectly selected target, as illustrated by its limited improvement on Mistral. Furthermore, low-frequency token runs do not necessarily cover all spans requiring exact reproduction, particularly spans containing common tokens. Consequently, the strong results on targeted RULER tasks do not yet establish how reliably the method addresses the broader RAG, code, and structured-extraction workloads used to motivate it. This uncertainty limits the demonstrated scope of the contribution.

Questions for authors
  1. Could the authors clarify which part of the improvement specifically demonstrates the need for temporal addressing? The query–key similarity variant shares the temporal trigger and outperforms positional successors on the verbatim-copying task under the matched admission configuration. What evidence separates the benefit of tracking sequential access from that of preserving selected entries and making them recallable? This distinction affects the assessment of the paper’s central conceptual contribution.

  2. How is the default unigram-frequency admission rule specified and calibrated? Please identify the frequency source, percentile threshold, and how these settings transfer across tokenizers and models. The appendix also describes priority admission around high-scoring anchors. How much recovery comes from low-frequency candidate runs versus anchor-neighborhood preservation, and how often does the cold-tier cap exclude required continuation tokens? These details are important for assessing whether coverage generalizes beyond the evaluated literal formats.

  3. How reliably does the attention-based read pointer identify the source span actually being copied? Monotone attention movement could also occur during ordinary generation, while valid copying may involve pauses, backward attention, or switching between spans. Do the existing telemetry logs distinguish correct triggers, false triggers, and missed copying events? Since recoverable cache replacements can still change subsequent generation, please clarify what supports the claim that false triggers are benign.

  4. How is the claim that sequential forgetting accounts for approximately 86% of the remaining compression gap established independently of KVFetch’s observed improvements? The manuscript connects this statistic to the tasks that improve under the method, but improvement alone does not establish the original failure’s cause. Please clarify the task-classification criteria and what evidence distinguishes missing-continuation failures from localization errors, target-selection errors, or other effects of compression.

  5. Does the reported efficiency comparison include all costs introduced by KVFetch, including cold-tier construction, quantization metadata, position indices, attention-peak extraction, dequantization, and cache swaps? The appendix states that SDPA requires additional query–key computation to obtain attention peaks, so a fixed hot-cache capacity does not by itself imply unchanged computational cost. Please clarify the accounting scope and whether the reported latency compares identical attention backends and execution settings.

6 comments

Thank you for this question. The original text did not separate the trigger from the fetch key, and query–key similarity outperforming positional successors on s3 (89.6 vs 80.0) made that easy to misread. A cleaner split is: the cold tier makes entries recallable; the trigger opens the channel when copying starts; the fetch criterion chooses which entries to bring in. Temporal addressing is the second of these, not the third. We have revised §3.6 and §4.3 accordingly, and added the Static-Pin control to the appendix.

The existing ablations already cut the design this way (Qwen2.5-7B, RULER-16K, 20%):

SettingRecallable storeSequential trackings3Avg
Hostnono0.854.28
w/o Admissionnotrigger on0.854.31
never-fetchyesno0.854.52
always-fetchyesno (undirected)24.051.72
w/o Prefetchyesyes, but not into attention0.854.48
Instantaneous strideyesweak30.460.24
Positional successors (default)yesyes80.0 / 78.492.96 / 64.88
Query–key similarityyesyes (same trigger)89.693.52
Random successorsyesyes12.8—
Static-Pin (admitted set frozen in hot tier, no trigger)resident, not cold-recallno76.062.52
KVFetchyesyes78.464.88

(Positional 80.0 / avg4 92.96 is the literal-admission match to the QK row; the main-table unigram run is 78.4 / 64.88. Static-Pin is Appendix C.1, Table 17.)

  1. Preservation plus recallability is not enough. never-fetch leaves spans in the cold tier and never brings them back: s3 stays 0.8. w/o Prefetch detects copying and still does not write into attention: 0.8 again. always-fetch recalls every step without tracking a trajectory: s3 24.0 and Avg 51.72, below the host—undirected recall hurts. Static-Pin freezes the admitted set in the hot tier and reaches s3 76.0 on a short single span, but it displaces qa and does not go dormant on LongBench (39.21 vs the host’s 39.67). Saving the entries solves “they still exist.” It does not, by itself, follow the copy that actually starts under iso-attention and many candidates.

  2. What requires temporal addressing is when the channel opens. Instantaneous stride scores 30.4; the accumulated pointer scores 78.4 (+48.0), the largest single ablation gap. That is sequential-access tracking: Copy=1 only after two monotone advances in a window of 8. Turn the trigger off (never / always) or skip the write, and the channel is not there.

  3. QK beating positional fetch does not remove the channel. Both rows share the same trigger and pointer; only the fetch key changes. With the channel open, QK is better on s3 (+9.6) and almost tied on the four-task mean (+0.76). Random successors fall to 12.8, so “which entries” is not arbitrary, but it is small next to opening the channel (0.8 vs 80). We default to positional successors because they are an O(k) lookup with no per-step dot products, not because content fetch fails. The revised manuscript now states this: the conceptual contribution is a decode-time recall channel gated by the copy trajectory; positional successors are a cheap default, not the definition of temporal addressing.

The store is a necessary precondition (no cold tier → 0.8); tracking is what makes the channel work (no trigger → 0.8, or a harmful 24.0). The QK variant is another fetch rule on the same channel, not a different story.

Revised manuscript: fetch-criterion wording in §3.6 and §4.3; query–key vs positional successors in Appendix B.4, Table 13; Static-Pin in Appendix C.1, Table 17. never/always/instantaneous-stride remain Table 2.

For Q2: You are right that the default unigram rule was a symbol, not a spec: τ, the frequency source, and the transfer across tokenizers were never written down. The three-tier σ in the appendix is the cold-tier ranking, not a full spec of the default unigram rule. The revision states the following.

Default spec (same rule for Qwen2.5-7B, Qwen3-4B, and Llama-3.1-8B; not the same table).

  • Frequency source. A fixed English C4 slice (10^8 whitespace tokens). Each model retokenizes that slice with its own tokenizer and we count types. These are tokenizer unigrams, not English word frequencies and not within-prompt relative frequencies.
  • Percentile. τ=15: among types that occur at least once in that C4 slice, the rarest 15%. Those tokens form a candidate run when they occur consecutively, extended by δ=4 on each end (the same hyperparameter as in the appendix), and only if they lie outside the hot tier.
  • Transfer. What transfers is the percentile rule, not token ids. Going from Qwen to Llama, we rebuild the rank table on the same C4 text with the new tokenizer; τ stays 15. Table 1 is therefore one rule and three tables, not one table copied across models.

Low-frequency runs vs anchor neighbourhood. Cold-tier ranking uses the same three-tier σ: unigram runs first (c_cand=2.0), then a W_a=64 window after high-scoring hot-tier anchors, else ε_0 s_p. Qwen2.5-7B, RULER-16K, otherwise the main-table config:

Cold-tier memberships313-task AvgCold slots from unigram runs
Anchor neighbourhood only14.455.410%
Unigram runs only74.464.21100%
Both (default)78.464.8882%
w/o Admission0.854.31—

Almost all of the recovery is from low-frequency runs (74.4 vs the host’s 0.8). Anchors alone reach only 14.4, a SnapKV-style static window that does not cover the full continuation. Adding them lifts s3 by another 4.0 (74.4→78.4), the short tail after a host-retained span head; about 18% of cold slots are of that kind. Table 14’s unigram vs literal (78.4 vs 80.0) is the same point in another form: a different format prior still covers rare spans; anchors are not doing the main work.

How often the 5%L cap drops a required continuation. On s3 we mark gold-span tokens that were evicted from the hot tier and ask whether they entered C:

  • 16K: 94.1% admitted, 5.9% excluded by the cap (usually because other rare runs in the same prompt filled the quota).
  • 128K: 76.4% admitted, 23.6% excluded; s3 falls 78.4→68.0, matching appendix Table 19.
  • 256K: 68.1% admitted, 31.9% excluded; s3 64.0.

The cap does drop continuations, and more so as L grows. What it drops is competing candidate runs, not the single needle as a whole. A successor that never entered C cannot be prefetched, however accurate the pointer—that is part of why s3 is lower at 256K than at 16K, not a trigger failure.

Default admission is a per-tokenizer C4 unigram rank, τ=15, consecutive runs plus δ=4. Coverage is from low-frequency runs; the anchor neighbourhood is a small add-on; the cap misses about 6% of needed continuations at 16K and more at longer context.

Revised manuscript: τ / C4 / one rank table per model in §3.4 and Appendix F.1; run vs anchors in Appendix C, Table 15; cap misses in Table 16; 128K/256K s3 in Appendix D.2, Table 19. Unigram vs literal remains Table 14.

Thank you for this question. Recoverable is not the same as harmless, and the original logs did not distinguish them. They recorded the fraction of steps with Copy_t=1 and the cold-tier hit rate; they did not label correct triggers, false triggers, or missed copying against a gold span. The Safety paragraph overstated the case: a recoverable replacement means the entry is not deleted, not that the attention set—and the next token—is unchanged. always-fetch already shows the other extreme: undirected high-rate swaps score 51.72 on average, below the host’s 54.28. The revision labels steps against gold spans and restricts “benign” to the observed false-trigger rate.

Labels (Qwen2.5-7B, RULER-16K / LongBench, same 20% host). A gold copy step is one where the prefix matches gold and the next gold token belongs to a verbatim source span. The pointer is aligned if it lies in that span ± the pointer band of 4.

  • TP: Copy=1 and the pointer is on the gold span.
  • FP: Copy=1 and the pointer is on no gold verbatim span.
  • FN (missed recall): the next gold continuation’s KV is not in the hot tier and this step does not fetch it. Copy=0 while a previous k=8 prefetch still covers that token is not an FN.
WorkloadCopy=1TP / Copy=1FP (all steps)Needed cont. already in HFN
niah_single_30.06891.4%0.00688.0%6.1%
RULER 13-task mix0.01184.0%0.002——
LongBench0.000–0.029— (no gold span)= triggern/an/a

On s3, 5.9% of needed continuations never entered the cold tier (the cap in Q2) and 6.1% are FN; the remaining 88.0% are in the hot tier when used. The cap is not a pointer error; the FN is.

Pauses, backward attention, and span switches are the three branches of Eq. (trigger). On s3 gold-copy steps: advance 41%, stall (−Δ_max ≤ Δ ≤ 0, pointer held) 56%, jump/reset (|Δ|>Δ_max) 3.2%. A stall does not close the channel if two advances remain in the window of 8. After a jump the pointer must collect n_adv=2 again; the median re-arm is 3 steps, and those bursts are most of the s3 FN. Span switching is not a single-span tracking failure: on mk2/mk3 the host already picks the wrong needle, so there is no trajectory to repair (main-text applicability). When the model does switch, the same short FN burst appears; it is not why those tasks stay low.

Why false triggers are benign at the observed rate and not at saturation.

  1. Copy mode needs two monotone advances in a window of 8; a single forward step is not enough. On LongBench the trigger is 0.000–0.029 and essentially all of it is FP.
  2. About 63% of those FP steps have |R|=0 (successors already hot, no swap). The rest move at most k=8 slots, with protection T_prot=12. qa_1 / qa_2 stay at the host’s 56.00 / 34.40 (Δ=0); LongBench is +0.33 with no family-level regression.
  3. always-fetch is the control, not the observed FP: every-step undirected recall, s3 24.0, mean 51.72, below the host. Recoverability does not imply harmlessness. Harmlessness is measured at an FP rate of about 0.6% of steps, mostly no-ops—not a guarantee from the Safety paragraph.

On a single-span copy the pointer is on the gold span for 91.4% of Copy=1 steps; missed recall is about 6%, mostly the re-arm after a jump, not a stall treated as failure; monotone motion in ordinary generation is almost entirely filtered by n_adv=2. We no longer claim “recoverable ⇒ false triggers are benign.” At the observed false-trigger rate, non-copy tasks do not regress.

Revised manuscript: TP/FP/FN in Appendix D.5, Table 22; Safety paragraph (observed-rate qualifier) in §3.6. always-fetch remains Table 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.