- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors examine the problem of fixed-budget support selection in long-context LLM inference. Sparse attention and KV-cache methods typically score cached tokens independently, keep the Top-K, and renormalize softmax attention over the retained set. The authors ask whether this ranking abstraction recovers the best K-token support even when every candidate is scored exactly. To do so, they consider a single softmax-attention head at one decoding step. The sparse head output is , with . Support distortion is measured by the downstream next-token KL , and each candidate is scored by its exact singleton-deletion effect . They show that the answer is negative in general: in an explicit Top-2 example, exact-singleton Top-K incurs about 133× the KL of the optimal pair. From the identity , they show that the value of swapping for depends on the current retained support. They then prove support-conditioned preference reversals for every , meaning no context-independent scalar ordering can represent all fixed-budget exchange preferences.
Based on this idea, they propose:
-
SwapCert, a one-sided headroom diagnostic. It searches the one-swap neighborhood of a support, which contains supports instead of , for a lower-KL witness. The authors prove that whenever SwapCert fires, the support has at least that much headroom relative to the exact candidate-universe optimum.
-
Exhaustive support enumeration on real models. The authors intervene on the first and last heads of the deepest softmax layer of SmolLM2-1.7B, Gemma-2-2B, and Qwen3.5-4B, using 30 LongBench-E prompts. They evaluate all size-4 supports within a 24-token candidate universe. Exact-singleton Top-4 leaves at least 10% relative headroom in 55/60, 49/60, and 60/60 cells, although absolute KL gaps are small ( to ). The effect is robust to the candidate-universe construction but varies strongly across models and budgets $K \in $ {}.
-
Diagnostic and behavioral evaluation. At , SwapCert detects about 95% of headroom-positive cells. Top-1 next-token agreement with the dense model changes only rarely, and the few observed flips are corrected by both the best one-swap support and the exact oracle.
Critical strengths
-
Ranging problem is well-observed. The study tests the best case for any independent per-token scorer, not a particular heuristic. The verifiable 133× counterexample, the exact exchange identity, and the support-conditioned reversal result explain why the failure is structural.
-
The empirical protocol is careful and the claims are honestly scoped. The ε threshold, models, heads, prompts, and candidate-universe constructions are all examined. The authors explicitly limit their claims: headroom is relative to an M = 24 candidate universe, not the full context; results show replication rather than architecture invariance; SwapCert is a one-sided diagnostic rather than a serving method.
Critical weaknesses
-
Needs to be examined with related works more deeply. Current examination weaks from baselines.
-
As far as I understood from the paper:
a. Only four top-1 flips are observed across all models (Fig. 3, right) b. Median KL is relatively small (Table 1).
This means (as far as I see) that the practical significance and external validity are quite limited.
Questions for authors
- Weakness 2. All results also come from perturbing a single head at one decoding step with and . Can you show that the analysis matters in realistic settings? For example, you could apply sparse selection to all heads and layers at larger budgets (K in the hundreds) and report multi-step or task-level metrics such as LongBench accuracy or perplexity.
We thank the reviewer for the careful technical reading and for recognizing the strength of the objective-matched counterexample, exchange identity, and preference-reversal construction.
We substantially revised the full manuscript in response to all three reviews: the Introduction and contribution summary have been refocused (Section 1), related-work positioning sharpened (Section 2), theoretical and computational roles distinguished (Sections 3–4), experimental scope extended (Sections 5–6), and interpretation consolidated (Sections 7–8). The revision contains five clear figures and fully documented expanded experiments in Appendices L–M.
W1 — Related methods and comparative baselines.
Revised Section 2 now compares rank-based KV retention (H2O, StreamingLLM, SparQ, SnapKV, Quest), output-aware or value-aware selection (CriticalKV, AnDPro, DropKV), and set-aware optimization or budget allocation (BumbleBee, Trust the Mass, Ada-KV).
The new closest-comparison table distinguishes their fidelity objectives and mathematical roles.
In particular, DropKV studies a renormalized head-output criterion under geometric conditions, and Trust the Mass examines forced-weight subset reconstruction; our Proposition 1 tests the sufficiency of exact singleton-deletion scores for the same downstream KL objective used to evaluate the whole retained set.
The original controlled exact-ranking comparison is now prominent in Section 6.1: replacing attention Top-K with exact downstream-singleton Top-K reduces the joint-support KL in 38/120 cells, ties in 34/120, and increases it in 48/120, separating the accuracy of marginal importance information from the quality of its selected support.
W2 — Larger budgets, multiple heads/layers, and predictive outcomes.
The new controlled protocol (Section 5; Appendix L) evaluates K=128 and K=256 on 2,048-token inputs across SmolLM2-1.7B, Gemma-2-2B, and Qwen3.5-4B.
Dense prefill is followed by cached decoding with sparse active supports in all heads of the last three eligible full-softmax layers, over the same frozen 30-prompt panel per model.
We compare four matched-budget selectors—attention Top-K, geometric head-output singleton Top-K, recent, and sink+recent—and report absolute dense-reference KL in Section 6.3.
Paired KL and gold-answer NLL differences use 20,000 prompt-bootstrap resamples and are reported for all six model/budget configurations, including their confidence intervals.
For Gemma, geometric selection reduces mean dense-reference KL relative to attention selection by 6.68% at K=128 and 9.77% at K=256 (absolute differences −4.52×10⁻⁵ and −3.07×10⁻⁵).
Both exploratory paired KL intervals exclude zero; at K=128, the reference-answer NLL is also lower by 0.00411 with interval [−0.00696, −0.00157].
The full table reports model-dependent effects for SmolLM2 and Qwen as well.
Separate larger-budget exchange screens find positive same-budget improvements in 39/72 configurations, with matched randomized controls reported in Appendix L.
The K=128, 24-token generation diagnostic records actual greedy continuations, which agree between geometric and attention selection in 90/90 matched cases.
These data provide a more realistic characterization of distributional fidelity and observed decoding outcomes, complementary to the exact-support oracle analysis.
Decision-level bridge and broader exact audits.
The new Section 6.4 and Appendix M directly apply Proposition 3 to 180 exact behavioral cells: dense top-1 decisions are mathematically certified in 152/180 ranked supports, 159/180 best-one-swap supports, and 161/180 candidate-universe optima; the four observed ranked-support flips are all repaired by a best one-swap.
Appendix J additionally reports the existing approximately 2K- and 4K-context exchange screens over 1,512 heads / 48,384 exchanges and 504 heads / 16,128 exchanges, respectively.
Thus the revised paper combines exact global comparison inside M=24 candidate universes with broader-budget, multi-layer predictive evidence and clearly labeled behavioral tests.
We appreciate the concrete experimental suggestions: they motivated major additions to the main text and supplementary evidence.
We respectfully invite the reviewer to reconsider the rating based on the strengthened comparisons, larger-budget multi-layer protocol, paired NLL/KL evidence, and certified decision-level analysis.