- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes KV-Selector, a small learned scorer for query-aware KV block selection in frozen language models. It reuses Quest’s minimum and maximum key landmarks, forms two query–landmark products per attention head, and combines them with a per-layer MLP containing fewer than 10,000 parameters. The scorer is distilled from the frozen model’s block attention mass. Evaluation includes single-layer fidelity diagnostics across eleven models, downstream benchmarks on four models, design ablations, and host-memory offloading experiments. The strongest downstream result is a consistent PG19 NLL improvement over shared training-free selection rules. Retrieval gains are mixed, and the current unfused implementation is slower than GPU-resident full attention. Offloading experiments demonstrate gains over synchronous whole-cache copying, with a quality–latency trade-off relative to simpler selectors.
Critical strengths
-
A lightweight method with consistent language-modelling gains. KV-Selector adds only a small per-layer scorer and reuses Quest’s existing summaries without modifying the language model. Its PG19 improvements over shared training-free rules persist across scorer seeds, budgets, and additional long-context evaluations. This provides substantive evidence that learning the scoring function can improve selection quality with limited training data.
-
An unusually careful empirical account of limitations. The paper includes strong simple baselines, per-head configurations, a released learned gate, numerical correctness checks, and explicit reporting of retrieval failures and GPU-resident slowdowns. These comparisons provide useful knowledge about when learned scoring helps and prevent the contribution from resting solely on favourable comparisons against Quest.
Critical weaknesses
-
The demonstrated advantage over stronger alternatives is narrow. The most consistent gains concern PG19 under shared block selection. RULER and LongBench improvements are generally small or mixed; Mean-key outperforms KV-Selector on Qwen3-8B retrieval, and per-head Corner scoring narrows or reverses its PG19 advantage on the two Qwen models at approximately matched reads. A learned mean-key scorer also nearly matches the default scorer in the diagnostic ablation. Together, these results leave the broader benefit of the proposed configuration insufficiently established.
-
A compelling practical quality–cost advantage has not been demonstrated. The current implementation is slower than GPU-resident full attention. The large offloading speedups are against synchronous whole-cache copying and are also achievable with simpler block selectors. Table 24 usefully reports quality and latency for the fixed-statistics, all-layer configuration, but leaves Mean-key latency unmeasured. Corner is faster at the same budget, and the paper does not establish whether KV-Selector remains preferable when a simpler selector is given a larger budget at matched latency. This missing comparison is consequential because reduced parameter count and improved fixed-budget NLL do not alone establish practical efficiency.
Questions for authors
-
Evidence needed to assess practical benefit: At matched decoding latency and GPU memory, does KV-Selector retain its PG19 advantage over Mean-key and Corner when their read budgets are adjusted? Please include the currently unmeasured Mean-key timing in the Table 24 configuration.
-
Clarification of deployment-relevant breadth: How much of the reported 78/82 fidelity advantage persists when feature statistics are fixed from training data throughout? The existing causal-statistics check covers only Qwen3-1.7B, whereas the headline diagnostic spans eleven models and uses future query positions.
-
Clarification of the learned component: When all configurations use the same causal normalization protocol and matched measured reads, how much of the advantage remains attributable to learned scoring rather than shared versus per-head block selection? The current per-head comparisons use a different normalization protocol from the main shared configuration.
Thank you for a careful review and for questions that we could turn directly into experiments. All three are now answered with new runs; the answers are not uniformly in our favour, and the revision reports them as they are.
W1. The advantage over stronger alternatives is narrow. We agree, and the revision scopes the claims to that evidence: consistent fidelity gains, PG19 gains at equal budget, mixed RULER and LongBench results, the Mean-key rule ahead on Qwen3-8B retrieval and at blocks of 64, and a learned mean-key scorer matching the min/max scorer in the fidelity study (the case for min/max is reuse of Quest's summaries). The new per-KV-head comparison (Q3) adds the strongest per-head baselines at an exactly matched read.
W2 and Q1. Quality at matched decoding time and memory; Mean-key timing. We timed the Mean-key rule in the offloaded configuration (130k tokens, every layer selected, H200), and every rule at budgets of 640-2,048 tokens. We scored PG19 at 131k tokens at 1,024 and 1,280 tokens as well (Table 27).
Q2. Fidelity with fixed training statistics on all eleven models. We reran the Table 1 study on all eleven models with KV-Selector's feature statistics fixed from its training windows, keeping windows, layers, block sizes and seeds (new Table 8, Section 4.1). The lead persists in full: KV-Selector beats all three training-free rules in 78 of 82 settings, as with window statistics. It trails in the same two Gemma-3-1B cells (by up to 4.7 points) and in two Qwen3.8-27B cells (0.6 and 0.7). Its gap closed changes by +0.05 points on average per setting (-3.2 to +4.5) and by at most 1.1 points in any model's mean. It has the highest mean on all eleven models, by 0.9 to 26.7 points over the strongest training-free rule, and closes 80-100 % of the gap against 49-94 % for Quest.
Q3. Learned scoring vs shared and per-head selection, same normalisation, matched reads. We added per-KV-head selection: each KV head keeps its own K blocks for its query heads, so the read equals the shared read exactly. Every learned scorer uses fixed statistics, including a per-KV-head KV-Selector trained like the shared one. PG19 at 640 tokens, four models, paired over 60 items (new Table 4, Section 4.4):