acceptodds
Under review as a conference paper at ICLR 2027

Query-Aware KV Block Selection from Cache Landmarks

Abstract

Autoregressive language models repeatedly read a growing key-value (KV) cache when generating from long contexts. Query-aware block selection retains the cache but limits attention to blocks chosen for each new token; its quality depends on ranking those blocks well. We introduce KV-Selector, a small learned scorer for frozen language models that reuses the minimum and maximum key summaries stored by Quest, an earlier selection method. It trains from the model's own attention in minutes, uses fewer than 10,000 parameters per layer and adds no summaries beyond Quest's. Across eleven models, it better preserves next-token predictions than the compared training-free rules in 78 of 82 settings that restrict one layer at a time. On four models with a 640-token read budget, it lowers negative log-likelihood on PG19, a benchmark of language modelling over books, by 0.013 to 0.039 nats relative to the best training-free rule making one block choice per layer. Retrieval gains are less consistent. With the cache in host memory, selecting blocks accelerates decoding relative to synchronously copying the whole cache, and on PG19 at 131k tokens KV-Selector keeps a lower negative log-likelihood than the compared training-free rules. Our unfused implementation remains slower than full attention with the cache on the GPU.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV cache, sparse attention, long-context LLMs, efficient inference, query-aware block selection, distillation, KV cache offloading

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KrBeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes KV-Selector, a small learned scorer for query-aware KV block selection in frozen language models. It reuses Quest’s minimum and maximum key landmarks, forms two query–landmark products per attention head, and combines them with a per-layer MLP containing fewer than 10,000 parameters. The scorer is distilled from the frozen model’s block attention mass. Evaluation includes single-layer fidelity diagnostics across eleven models, downstream benchmarks on four models, design ablations, and host-memory offloading experiments. The strongest downstream result is a consistent PG19 NLL improvement over shared training-free selection rules. Retrieval gains are mixed, and the current unfused implementation is slower than GPU-resident full attention. Offloading experiments demonstrate gains over synchronous whole-cache copying, with a quality–latency trade-off relative to simpler selectors.

Critical strengths
  1. A lightweight method with consistent language-modelling gains. KV-Selector adds only a small per-layer scorer and reuses Quest’s existing summaries without modifying the language model. Its PG19 improvements over shared training-free rules persist across scorer seeds, budgets, and additional long-context evaluations. This provides substantive evidence that learning the scoring function can improve selection quality with limited training data.

  2. An unusually careful empirical account of limitations. The paper includes strong simple baselines, per-head configurations, a released learned gate, numerical correctness checks, and explicit reporting of retrieval failures and GPU-resident slowdowns. These comparisons provide useful knowledge about when learned scoring helps and prevent the contribution from resting solely on favourable comparisons against Quest.

Critical weaknesses
  1. The demonstrated advantage over stronger alternatives is narrow. The most consistent gains concern PG19 under shared block selection. RULER and LongBench improvements are generally small or mixed; Mean-key outperforms KV-Selector on Qwen3-8B retrieval, and per-head Corner scoring narrows or reverses its PG19 advantage on the two Qwen models at approximately matched reads. A learned mean-key scorer also nearly matches the default scorer in the diagnostic ablation. Together, these results leave the broader benefit of the proposed configuration insufficiently established.

  2. A compelling practical quality–cost advantage has not been demonstrated. The current implementation is slower than GPU-resident full attention. The large offloading speedups are against synchronous whole-cache copying and are also achievable with simpler block selectors. Table 24 usefully reports quality and latency for the fixed-statistics, all-layer configuration, but leaves Mean-key latency unmeasured. Corner is faster at the same budget, and the paper does not establish whether KV-Selector remains preferable when a simpler selector is given a larger budget at matched latency. This missing comparison is consequential because reduced parameter count and improved fixed-budget NLL do not alone establish practical efficiency.

Questions for authors
  1. Evidence needed to assess practical benefit: At matched decoding latency and GPU memory, does KV-Selector retain its PG19 advantage over Mean-key and Corner when their read budgets are adjusted? Please include the currently unmeasured Mean-key timing in the Table 24 configuration.

  2. Clarification of deployment-relevant breadth: How much of the reported 78/82 fidelity advantage persists when feature statistics are fixed from training data throughout? The existing causal-statistics check covers only Qwen3-1.7B, whereas the headline diagnostic spans eleven models and uses future query positions.

  3. Clarification of the learned component: When all configurations use the same causal normalization protocol and matched measured reads, how much of the advantage remains attributable to learned scoring rather than shared versus per-head block selection? The current per-head comparisons use a different normalization protocol from the main shared configuration.

1 comment

Thank you for a careful review and for questions that we could turn directly into experiments. All three are now answered with new runs; the answers are not uniformly in our favour, and the revision reports them as they are.

W1. The advantage over stronger alternatives is narrow. We agree, and the revision scopes the claims to that evidence: consistent fidelity gains, PG19 gains at equal budget, mixed RULER and LongBench results, the Mean-key rule ahead on Qwen3-8B retrieval and at blocks of 64, and a learned mean-key scorer matching the min/max scorer in the fidelity study (the case for min/max is reuse of Quest's summaries). The new per-KV-head comparison (Q3) adds the strongest per-head baselines at an exactly matched read.

W2 and Q1. Quality at matched decoding time and memory; Mean-key timing. We timed the Mean-key rule in the offloaded configuration (130k tokens, every layer selected, H200), and every rule at budgets of 640-2,048 tokens. We scored PG19 at 131k tokens at 1,024 and 1,280 tokens as well (Table 27).

  • At equal budget, KV-Selector has the lowest NLL at every budget on all three timed models.
  • At equal decoding time in our implementation, the answer depends on the model. Each further 1,000 tokens costs 8-19 ms per token, and the MLP costs KV-Selector 1.5-7.0 ms over Corner. On K2-Horizon-3.7B, whose cache is largest, Corner and Mean-key reach KV-Selector's time at about 720 and 930 tokens and stay 0.130 and 0.045 nats above it. On MiniCPM5-2B and Granite-4.2-3B, Mean-key reaches it at 1,320 tokens or more and is at least 0.037 and 0.021 nats lower; Corner is within 0.007 nats. So your concern holds on two of the three models.
  • Without launch overhead (CUDA graph, Tables 21-22), KV-Selector's scoring costs only 0.8-1.0 ms per token more than Corner's and 1.4-2.1 ms more than Mean-key's at 130k, so the equal-time budgets would move closer together. We have not timed offloaded decoding in that mode and say so.
  • GPU memory: KV-Selector and Corner reach the same peak (same landmarks). Mean-key keeps one landmark per block and peaks 0.5-1.6 GB lower at 128k (Table 29). The offloading speed-up itself comes from reading fewer tokens and is shared by the simpler rules, as you note. We also found that our whole-cache baseline was slowed by a non-contiguous copy; the corrected speed-ups at 128k are 2.0-6.4x (general response).

Q2. Fidelity with fixed training statistics on all eleven models. We reran the Table 1 study on all eleven models with KV-Selector's feature statistics fixed from its training windows, keeping windows, layers, block sizes and seeds (new Table 8, Section 4.1). The lead persists in full: KV-Selector beats all three training-free rules in 78 of 82 settings, as with window statistics. It trails in the same two Gemma-3-1B cells (by up to 4.7 points) and in two Qwen3.8-27B cells (0.6 and 0.7). Its gap closed changes by +0.05 points on average per setting (-3.2 to +4.5) and by at most 1.1 points in any model's mean. It has the highest mean on all eleven models, by 0.9 to 26.7 points over the strongest training-free rule, and closes 80-100 % of the gap against 49-94 % for Quest.

Q3. Learned scoring vs shared and per-head selection, same normalisation, matched reads. We added per-KV-head selection: each KV head keeps its own K blocks for its query heads, so the read equals the shared read exactly. Every learned scorer uses fixed statistics, including a per-KV-head KV-Selector trained like the shared one. PG19 at 640 tokens, four models, paired over 60 items (new Table 4, Section 4.4):

  • Learned vs training-free, shared choice: KV-Selector is 0.013-0.043 nats below the best shared rule in this run.
  • Learned vs training-free, per-KV-head choice: KV-Selector is 0.004 / 0.013 / 0.017 nats below the best training-free rule on Qwen3-8B / MiniCPM5-2B / K2-Horizon-3.7B, and 0.022 nats above Mean-key on Qwen3-1.7B.
  • Granularity: choosing per KV head lowers every rule's NLL on every model (0.003-0.178 nats). On the two Qwen3 models, per-KV-head training-free rules beat shared KV-Selector; on MiniCPM5-2B (8 query heads per KV head) and K2-Horizon-3.7B they do not. So on the Qwen3 models the granularity of the choice matters more than learning it; on the other two, learning matters more; and a learned per-KV-head scorer is the best configuration on three of four models.
Reviewer_6sTiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes KV-Selector, a learned query-aware KV block selection method to improve long-context inference efficiency. It reuses Quest's element-wise min/max key landmarks and computes the dot products between the landmarks and the query as the input feature to the MLP to predict the block importance scores. The method is evaluated on several long-context benchmarks, PG19, RULER, and LongBench against prior baselines.

Critical strengths
  • The paper is clearly written and easy to understand.
Critical weaknesses
  • The novelty of the paper is limited. The method seems quite similar to SeerAttention-R, with slightly different input features. The choice of input features (dot products between query and landmark keys) are also not clearly justified.
  • Since the method requires training, its practical usefulness is significantly limited compared to other training-free baselines.
  • The experimental results do not show the effectiveness of the proposed method. The LongBench and RULER scores suggest that the method is either on par or sometimes worse than the training-free baselines. I believe that the performance of a learned method should be significantly better than its training-free counterparts, in order for it to be useful despite its limited practicality.
  • The decoding speed comparison in section 4.5 seems to suggest that the decoding efficiency of the method is worse than full attention, and the paper does not validate its efficiency claims through an optimized implementation.
Questions for authors
  • What is the theoretical motivation for dot products between query and min/max keys as features?
  • Why is the MLP trained to select one set of block indices across all KV heads? Wouldn't selecting different indices per KV head perform better without any additional inference cost?
1 comment

Thank you for the review. We answer each point below.

W1. Novelty relative to SeerAttention-R; feature choice. Both distil a block scorer from a frozen model's attention. The difference is what each adds to the model. SeerAttention-R adds query and key projections in every layer and keeps a second cache of pooled block keys; its released Qwen3-8B gates have about 0.92M parameters per layer. KV-Selector adds no projection and no cache: it reads only the two landmarks Quest already stores, uses 4.8k-9.4k parameters per layer (110x fewer on Qwen3-8B) and trains in 8-21 minutes. The paper asks how much of the benefit of learned selection survives at that extreme. The answer is mixed: higher fidelity on all eleven models and lower PG19 NLL than every shared training-free rule, ties with the released gate on PG19 and LongBench, and a clear deficit on coarse-block retrieval.

W2. Training limits practicality. Training is a one-off step per model, comparable to the calibration step of post-training quantisation: 12 wikitext windows, 8-21 minutes on one GPU, model weights untouched. The same scorers then serve every benchmark, corpus and length in the paper without retraining, up to PG19 at 131k tokens from 4k-token training windows. The revision states this in Section 3.

W3. RULER and LongBench are on par. We agree that the downstream gain is concentrated on PG19 and the paper says so. LongBench at 640 tokens cannot separate the rules: full attention and the attention-mass oracle are within 1.3 points of each other on all four models (Table 2), so no selection rule can be significantly better there; the revision states this. RULER does leave headroom (3.5-7.8 points between the best training-free rule and the oracle) and KV-Selector closes part of it only on Qwen3-1.7B; on Qwen3-8B the Mean-key rule is better. Where quality is measured per token, the learned ranking saves reads: in a budget sweep on Qwen3-1.7B, the Mean-key rule needs 1.3-1.8x the read for KV-Selector's PG19 NLL at 320-2,048 tokens, and Quest 6.3x at 320.

W4. Slower than full attention; no optimised implementation. We now measure the decode step with all kernel launches removed (CUDA graph) for full attention and each rule up to 130k tokens on an H200, with a per-component profile (Tables 21-22). The earlier +18-29 ms per token were almost entirely launch overhead: launch-free, KV-Selector at 640 tokens is 0.9-2.2 ms per token slower than full attention (K2-Horizon-3.7B at 130k: 12.40 vs 11.55 ms). It is still not faster, because the unfused scoring pass over all blocks costs about as much as full attention's read of the cache. We report this plainly and no longer present GPU-resident speed as a strength; with the cache in host memory, selection does decode faster than copying the whole cache, 2.0-6.4x at 128k against a corrected baseline.

Q1. Theoretical motivation for the two products. Write the block's bounding box with centre c = (k_min + k_max)/2 and width w = k_max - k_min. Then q·k_min = q·c - q·w/2 and q·k_max = q·c + q·w/2: the two features are exactly the centre's logit and the box's width along the query. Quest's bound is q·c + (1/2) Σ_i |q_i| w_i and the Corner rule is q·c + (1/2)|q·w|. Both add a width term to the centre's logit; Quest's term grows with the box's extent in every dimension the query touches, even when no key lies near that corner, which over-ranks heterogeneous blocks. A block's attention mass, Σ_j exp(q·k_j/√d), depends on where its logits sit and how widely they spread, not on their worst-case maximum. The MLP learns a function of exactly these two quantities per head and combines heads, trained on block mass. The revision adds this as Eq. 2 in Section 2.

Q2. Why one block set for all KV heads; would per-KV-head selection not be better at no extra cost? A shared choice needs one index list per layer, which keeps the gather simple; per-KV-head selection reads the same bytes, so it is the right comparison. We ran it: every KV head keeps its own K blocks for its query heads, so the read equals the shared read exactly, for Quest, Corner, Mean-key and a learned per-KV-head scorer, all with fixed statistics, on PG19 for the four benchmark models (new Table 4, Section 4.4). You were right that it is better: it lowers the NLL of every rule on all four models, by 0.003-0.178 nats (15 of 16 paired intervals exclude zero), most on the Qwen3 models and least on MiniCPM5-2B, whose KV heads each serve eight query heads. KV-Selector trained per KV head keeps the lowest NLL on Qwen3-8B, MiniCPM5-2B and K2-Horizon-3.7B (0.004, 0.013 and 0.017 nats below the best per-KV-head training-free rule, intervals excluding zero); on Qwen3-1.7B the per-KV-head Mean-key rule is 0.022 nats lower. Our earlier per-query-head experiments read the union of a KV head's choices and so cost more; they move to Appendix C.9.

Reviewer_j9czOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes KV-Selector, a query-aware KV-cache block selector that keeps the full KV cache but lets each query read only a subset of blocks. For each block, it reuses Quest’s per-block min/max key landmarks and computes, for every query head, two dot products with these landmarks. These 2H features are standardized and passed to a small per-layer MLP that outputs a scalar block score. By default, each query always reads the first block and its own block, plus the top-K scored blocks; all heads in a layer share one block choice. The scorer adds no new query/key/hidden-state projections and no additional cache representation. It is trained while the language model is frozen, by distilling the block attention mass of the unrestricted model into a soft target distribution. The paper evaluates the method on a masked-window fidelity diagnostic across eleven models, on PG19, RULER, and LongBench for four main benchmark models, on per-head selection, on a comparison with released SeerAttention-R gates, and on decoding speed with the cache either in GPU memory or in host memory. The main claims are higher masked-window fidelity than training-free rules, lower PG19 NLL than shared training-free rules, mixed RULER/LongBench differences, and faster decoding than naively copying the whole cache when the cache is offloaded to host memory.

Critical strengths

The problem is clearly motivated, and reusing Quest’s existing summaries gives the design a degree of engineering simplicity.

Critical weaknesses
  • Per-dimension min/max landmarks are not a sufficient statistic for the distribution of keys inside a block. There are cases where two blocks have identical min/max landmarks but different true attention mass, and no MLP can distinguish them from these features alone. The authors’ own ablation also shows that min/max inputs are not better than the simpler mean-key product, and the method clearly fails on coarse-block multi-key retrieval. The learned component therefore mainly calibrates a restricted feature set rather than providing a superior ranking signal.

  • The speed comparison is incomplete. It provides no end-to-end latency comparison with host-memory sparse attention systems such as MagicPig or RetrievalAttention, nor with offloading systems that host Quest-style selection.

  • As a training-based method, it does not outperform SeerAttention-R on the key retrieval task. Moreover, the comparison is not trained on the same data, which further weakens the evidence for the method’s superiority. The paper mentions DSA/CSA/CSA2-style training-time sparse attention only in passing, without any mechanism-level discussion or experimental positioning relative to this main line of work.

  • The system-level optimization is incomplete. There is no fine-grained latency breakdown, and the method is not combined with system optimizations such as RetroInfer. The method is not integrated into production serving frameworks such as vLLM or SGLang; with the cache resident in GPU memory, the implementation is slower than full attention. This limits the practical relevance of the acceleration results and makes it difficult to identify the actual bottleneck.

Questions for authors
  • Can the authors add SeerAttention-R directly to the Qwen3-8B rows of Table 2 under the same block size and always-read policy, so that the comparison is more comparable?

  • Can the authors provide a latency breakdown by component—landmark reading, MLP scoring, top-K selection, and gather—and quantify the expected benefit of a fused kernel?

2 comments

(Part 1 of 2)

Thank you for the detailed review. We answer each point and say what changed in the revision.

W1. Min/max landmarks are not a sufficient statistic. We agree, and the revision states it (Section 2): two blocks with the same bounding box can hold different keys and receive different attention. The same holds for any fixed-size summary, including the mean key, SeerAttention-R's pooled block keys and DeepSeek-V4's compressed entries. Our claim is narrower: a learned function of the summary Quest already stores ranks blocks better than Quest's bound and the other training-free rules. Two observations suggest the summary is not the binding limit on our data. First, scorers given more of each block gain little: a convolution over all of the block's keys (26.9k parameters) reaches 96.4 / 88.1 % gap closed against 96.0 / 88.4 % for KV-Selector on Qwen3-1.7B / SmolLM2-1.7B, and a 2.3M-parameter recurrent scorer 96.3 / 89.6 % (Table 9). Second, the coarse-block retrieval deficit you point to does not close when the mean key is added to the learned scorer (71.3 % against 79.5 % for the fixed Mean-key rule at blocks of 64), so it comes from what the scorer learns from generic text rather than from the min/max summary alone. On "calibrates a restricted feature set": that is a fair description, and it is the contribution, since with identical inputs the learned ranking closes 81-99 % of the random-to-oracle gap against 51-94 % for Quest, and Quest needs 6.3x the read for KV-Selector's PG19 NLL at 320 tokens. We also added a centre/width decomposition (Eq. 2) showing what the two products encode and why Quest's bound over-rewards wide boxes.

W2. No end-to-end comparison with host-memory systems. KV-Selector is a scoring function over Quest-style block landmarks. MagicPIG, RetrievalAttention and RetroInfer differ mainly in the index they build (LSH tables, nearest-neighbour graphs, key clusters) and in how they overlap transfers; a faithful end-to-end comparison means porting the scorer into each system, which we could not do within the discussion period. The revision positions these systems explicitly (Section 5). In checking our own offload experiment we found that our whole-cache copy baseline was slowed by a non-contiguous copy; the corrected baseline is 3.9-6.3x faster, and the speed-up of selection at 128k falls to 2.0-6.4x (general response; Figure 3, Tables 24-26). That speed-up comes from reading fewer tokens and is shared by the Corner and Mean-key rules, which the revision now times beside KV-Selector; Table 27 compares the rules at equal decoding time.

(Part 2 of 2)

W3. Not better than SeerAttention-R on retrieval; not trained on the same data. Correct, and the paper says so: the released gates lead on RULER at blocks of 64 and 640 tokens (78.2 vs 72.2 %) and tie on PG19 and LongBench. The comparison is between released artefacts; the gates have about 110x KV-Selector's parameters and a cached pooled key per block. A same-data comparison with a full RULER evaluation at blocks of 64 (10+ GPU-hours per configuration) did not fit the discussion period; we state this limitation. On training-time sparse attention, the revision adds a mechanism-level paragraph (Section 5): NSA learns a block compressor and selects blocks with it; MoBA trains the model with a mean-key gate; DSA scores every token with a lightning indexer that has its own keys; DeepSeek-V4's compressed sparse attention merges token groups with learned weights and applies the indexer to the merged entries. In all of them the summary or scorer is trained jointly with the model, which adapts to it. KV-Selector targets the case they leave open: a trained model that stays frozen.

W4 and Q2. Latency breakdown and the benefit of a fused kernel. We added both (Tables 21-22, Section 4.5). The decode step is captured as a CUDA graph for full attention and every rule, with a per-component GPU profile, up to 130k tokens on an H200, on four models. Example, K2-Horizon-3.7B at 130k, GPU time per token: full attention 5.36 ms for its attention (19.2 GB read at about 3.6 TB/s); KV-Selector at 640 tokens reads landmarks and forms the products in 2.38 ms, runs the MLP in 1.02 ms, selects top-K in 1.75 ms, gathers in 0.51 ms and attends in 0.29 ms. End to end, the launch-free step takes 12.40 ms against 11.55 ms for full attention, and over all models and lengths KV-Selector is 0.9-2.2 ms per token slower: the +18-29 ms of the paper's eager timings were launch overhead, and the remaining gap is the unfused scoring pass. The Mean-key rule, with one landmark, is 1.2 ms faster than full attention there. An estimate from byte counts: with float16 landmarks, as Quest stores them, the scoring read is 1/16 of the cache, about 0.33 ms at the same bandwidth, so a fused scoring and top-K kernel would be needed for a GPU-resident speed-up. We have not built it and do not claim one. Integration into vLLM, SGLang or RetroInfer is outside what we could do in this period.

Q1. SeerAttention-R in Table 2 under the same block size and always-read policy. Table 3 contains exactly this comparison at the gates' own block size of 64: "SeerAttention-R, our fixed blocks" uses our always-read policy (first block, current block and the top-ranked others) with the released gates. At 640 tokens it scores 76.2 % on RULER against 72.2 % for KV-Selector and has the same PG19 NLL (2.666 nats). We did not run the gates at blocks of 16, because they were trained for 64-token blocks. The revision adds this pointer to Table 2's caption.

Comments on the paper

1 comment

We thank the reviewers for careful, specific reviews. Three themes recur: (i) whether a learned function of Quest's existing landmarks is a contribution beyond SeerAttention-R and training-time sparse attention, and why these features; (ii) whether the quality gains matter, given mixed RULER and LongBench results; (iii) whether the method is fast, with the cache on the GPU and in host memory. We ran new experiments on all three and revised the paper. We also found and corrected an error in our own offloading baseline, which we report first.

Correction: the whole-cache copy baseline was too slow. In Section 4.5 and the offloading tables, "copying the whole cache" copied a strided view of pinned host memory, which PyTorch first makes contiguous in pageable memory. Copying the same bytes as contiguous per-head slices is 3.9-6.3x faster (MiniCPM5-2B at 130k tokens: 517.9 ms per token with the strided copy, 516.8 in the submitted paper, against 125.4 ms contiguous). Against the corrected baseline, selecting 640 tokens decodes 2.0x (MiniCPM5-2B), 3.6x (Granite-4.2-3B) and 6.4x (K2-Horizon-3.7B) faster at 128k tokens, not 8.1-34.8x; under a real GPU memory limit 3.4-6.3x, not 19-41x; at 32k tokens on MiniCPM5-2B the whole copy is faster. The revision uses the corrected numbers everywhere, including the abstract, Figure 3 and Tables 24-26.

New experiments in the revision.

  1. Fidelity with fixed statistics, all eleven models (Table 8, Section 4.1). With feature statistics fixed from the training windows, KV-Selector again leads all three training-free rules in 78 of 82 settings; its gap closed changes by +0.05 points on average and by at most 1.1 points in any model's mean.
  2. Launch-free latency and a per-component profile (Tables 21-22, Section 4.5). With the decode step captured as a CUDA graph on an H200, KV-Selector at 640 tokens is 0.9-2.2 ms per token slower than full attention up to 130k tokens (the eager +18-29 ms was launch overhead). The remaining cost is scoring every block; we do not claim a GPU-resident speed-up.
  3. Quality at equal decoding time (Table 27). With the cache in host memory at 128k tokens, KV-Selector has the lowest PG19 NLL at every equal budget (640-2,048) on all three timed models. At equal time in our implementation the answer depends on the model: KV-Selector stays ahead on K2-Horizon-3.7B (by 0.045 nats over Mean-key at about 930 tokens), while on MiniCPM5-2B and Granite-4.2-3B the Mean-key rule with 1,320 or more tokens is at least 0.037 and 0.021 nats lower.
  4. Per-KV-head selection at an exactly equal read (Table 4, Section 4.4). Each KV head picks its own blocks for its query heads. This lowers every rule's PG19 NLL on all four models (by 0.003-0.178 nats). KV-Selector trained per KV head is then lowest on three models (0.004-0.017 nats ahead); on Qwen3-1.7B the per-KV-head Mean-key rule is 0.022 nats lower.

Revised text. A centre/width decomposition of the two landmark products, showing how Quest, the Corner rule and KV-Selector differ (Eq. 2). An explicit statement that no fixed-size summary determines a block's mass, with the evidence that richer block inputs change gap closed by -0.3 to +1.2 points (Section 2, Table 9). A mechanism-level account of selection trained with the model (NSA, MoBA, DSA's lightning indexer, DeepSeek-V4's compressed sparse attention, InfLLM-v2) and of host-memory systems (RetrievalAttention, MagicPIG, RetroInfer, ShadowKV) in Section 5. The training cost (one-off, 8-21 min on one GPU). The LongBench ceiling (full attention and the oracle within 1.3 points). A conclusion that names retrieval as the main weakness and gives the speed results as above.

What we claim after the revision. A learned ranking of Quest's existing landmarks, with under 10k parameters per layer and minutes of training on a frozen model, preserves next-token predictions better than the training-free rules across eleven models, with window or fixed statistics, and lowers PG19 NLL against every training-free rule at equal budget, with one block choice per layer and, on three of four models, per KV head. Retrieval gains are mixed, and coarse-block retrieval is a clear weakness. With the cache in host memory, reading fewer tokens gives a 2-6x speed-up at 128k that any rule with the same read shares. We no longer present decoding speed as a strength of the learned scorer itself.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.