- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Focus-KV, a decoding-time KV-cache compression method for long chain-of-thought reasoning that replaces the uniform per-head budget of R-KV with a static, head-wise budget. Motivated by the observation that reasoning ability is concentrated in a sparse set of attention heads (Sec. 3), Focus-KV (i) builds a 100-sample probe set from OpenR1-Math-220K ranked by reasoning length and difficulty, (ii) scores each query head by the increase in next-token cross-entropy on the probe set when that head is suppressed, (iii) aggregates scores to KV heads (max over the GQA group) and converts them into per-KV-head budgets via a uniform floor, a cap, and water-filling redistribution, and (iv) keeps these budgets fixed during decoding while reusing R-KV's scoring for token selection. On Qwen3-14B/8B and R1-Distill-Llama-8B, Focus-KV improves pass@1 over R-KV, ReasonAlloc, and Ada-KV on AIME24 and MATH-500, especially at small budgets, and transfers to LiveCodeBench, MuSR, ZebraLogic, and SATBench with the math-only probe set.
Critical strengths
-
A training-free way to set a static head-wise budget. Importance is estimated offline from a small sampled probe set via causal ablation, so no RL/learning stage is needed and no runtime re-estimation is exposed to the bias of an already-compressed cache. The proposed Budget Utilization Index (BUI) is a new and interesting lens for analyzing whether larger per-head budgets are actually used effectively.
-
Consistent accuracy gains. Focus-KV improves over the compared baselines across three models and multiple benchmarks, with the largest gains in low-budget regimes.
Critical weaknesses
-
Limited novelty of the contribution.
Analyses of head-wise roles (e.g., retrieval/streaming heads) and eviction or budget-allocation methods built on them are well established since the long-context literature (e.g., DuoAttention, Ada-KV, HeadKV), and reasoning-specific work already exists (e.g., RLKV, which explicitly identifies "reasoning heads"). Hence the claim that reasoning ability is concentrated in a sparse set of heads (Sec. 3) and the first two contributions cannot be presented as new findings. Moreover, the token-selection component of Focus-KV is taken directly from R-KV , so the method is effectively R-KV plus a head-importance score and a budget allocator. This is an incremental delta over existing work. It would also be much more convincing if the two factors, non-uniform budgets per se and the proposed importance signal, were separated by ablation, showing how each contributes to the gains. -
Insufficient and under-specified baselines.
The most closely related head-wise methods (RLKV, DuoAttention, and other recent reasoning-oriented KV methods) are mentioned in one sentence in Related Work but never compared. Moreover, the compared methods (R-KV, ReasonAlloc, Ada-KV) are described only in a sentence each: how they were implemented, how Ada-KV (a prefill method) was adapted to decoding, and whether window size, scoring hyperparameters, and compression interval match Focus-KV are not reported (Appendix B.2 lists only Focus-KV's hyperparameters). Without this, it is hard to understand why Focus-KV outperforms them, and the gains cannot be attributed to the proposed allocation.
Questions for authors
Clarifications
How were R-KV, ReasonAlloc, and Ada-KV implemented and tuned? Please specify the adaptation of Ada-KV to decoding, and whether local window, lambda, observation window, and compression interval are matched with Focus-KV. Could you compare against RLKV and DuoAttention, or at least a binary full/compressed variant using Focus-KV's own importance scores, and a control with shuffled importance under the same allocator?
Suggestions
- Since eviction-scoring comparisons on standard reasoning benchmarks are increasingly saturated, I would encourage showing robustness in more practical settings, e.g., long input + long decoding together, and whether the gains persist when combined with different token-scoring rules. Dedicated scoring methods tailored to reasoning have also appeared (e.g., BeaconKV-ICML26), and it would be worth referring to such work.
- HF-based efficiency measurements are useful, but results on vLLM (where ragged per-head KV lengths conflict with PagedAttention; e.g., Tangram addresses this for static head budgets) would make the paper stronger.