acceptodds
Under review as a conference paper at ICLR 2027

Focus-KV: Head-Wise KV Cache Compression Focus On Reasoning Heads

23 views

Abstract

Large language models rely on long chain-of-thought (CoT) to solve complex reasoning problems, but the long decoding trajectories make the key–value (KV) cache a major memory cost. Existing KV-cache compression methods reduce this cost by evicting tokens or adaptively reallocating cache budgets at runtime. However, they either assign the same budget to all attention heads, which gives substantial memory to heads that contribute little to reasoning, or rely on runtime attention statistics that can be biased by the already-compressed cache. To address this, we propose Focus-KV, a static head-wise KV-cache compression algorithm that allocates more budget to attention heads that are important for reasoning, and the per-head budgets remain unchanged at runtime. Experiments across multiple models and benchmarks show that Focus-KV consistently outperforms strong baselines, particularly under highly constrained budgets. We further demonstrate that the identified reasoning-critical heads capture reasoning ability that is general rather than confined to specific tasks: the importance signal obtained from a mathematical probe set can be directly applied to reasoning tasks in other domains without re-estimation. Focus-KV exploits this cross-domain generality to better preserve the model’s reasoning ability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression, Reasoning, Attention Heads.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_mCPPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Focus-KV, a decoding-time KV-cache compression method for long chain-of-thought reasoning that replaces the uniform per-head budget of R-KV with a static, head-wise budget. Motivated by the observation that reasoning ability is concentrated in a sparse set of attention heads (Sec. 3), Focus-KV (i) builds a 100-sample probe set from OpenR1-Math-220K ranked by reasoning length and difficulty, (ii) scores each query head by the increase in next-token cross-entropy on the probe set when that head is suppressed, (iii) aggregates scores to KV heads (max over the GQA group) and converts them into per-KV-head budgets via a uniform floor, a cap, and water-filling redistribution, and (iv) keeps these budgets fixed during decoding while reusing R-KV's scoring for token selection. On Qwen3-14B/8B and R1-Distill-Llama-8B, Focus-KV improves pass@1 over R-KV, ReasonAlloc, and Ada-KV on AIME24 and MATH-500, especially at small budgets, and transfers to LiveCodeBench, MuSR, ZebraLogic, and SATBench with the math-only probe set.

Critical strengths
  1. A training-free way to set a static head-wise budget. Importance is estimated offline from a small sampled probe set via causal ablation, so no RL/learning stage is needed and no runtime re-estimation is exposed to the bias of an already-compressed cache. The proposed Budget Utilization Index (BUI) is a new and interesting lens for analyzing whether larger per-head budgets are actually used effectively.

  2. Consistent accuracy gains. Focus-KV improves over the compared baselines across three models and multiple benchmarks, with the largest gains in low-budget regimes.

Critical weaknesses
  1. Limited novelty of the contribution.
    Analyses of head-wise roles (e.g., retrieval/streaming heads) and eviction or budget-allocation methods built on them are well established since the long-context literature (e.g., DuoAttention, Ada-KV, HeadKV), and reasoning-specific work already exists (e.g., RLKV, which explicitly identifies "reasoning heads"). Hence the claim that reasoning ability is concentrated in a sparse set of heads (Sec. 3) and the first two contributions cannot be presented as new findings. Moreover, the token-selection component of Focus-KV is taken directly from R-KV , so the method is effectively R-KV plus a head-importance score and a budget allocator. This is an incremental delta over existing work. It would also be much more convincing if the two factors, non-uniform budgets per se and the proposed importance signal, were separated by ablation, showing how each contributes to the gains.

  2. Insufficient and under-specified baselines.
    The most closely related head-wise methods (RLKV, DuoAttention, and other recent reasoning-oriented KV methods) are mentioned in one sentence in Related Work but never compared. Moreover, the compared methods (R-KV, ReasonAlloc, Ada-KV) are described only in a sentence each: how they were implemented, how Ada-KV (a prefill method) was adapted to decoding, and whether window size, scoring hyperparameters, and compression interval match Focus-KV are not reported (Appendix B.2 lists only Focus-KV's hyperparameters). Without this, it is hard to understand why Focus-KV outperforms them, and the gains cannot be attributed to the proposed allocation.

Questions for authors

Clarifications
How were R-KV, ReasonAlloc, and Ada-KV implemented and tuned? Please specify the adaptation of Ada-KV to decoding, and whether local window, lambda, observation window, and compression interval are matched with Focus-KV. Could you compare against RLKV and DuoAttention, or at least a binary full/compressed variant using Focus-KV's own importance scores, and a control with shuffled importance under the same allocator?

Suggestions

  • Since eviction-scoring comparisons on standard reasoning benchmarks are increasingly saturated, I would encourage showing robustness in more practical settings, e.g., long input + long decoding together, and whether the gains persist when combined with different token-scoring rules. Dedicated scoring methods tailored to reasoning have also appeared (e.g., BeaconKV-ICML26), and it would be worth referring to such work.
  • HF-based efficiency measurements are useful, but results on vLLM (where ragged per-head KV lengths conflict with PagedAttention; e.g., Tangram addresses this for static head budgets) would make the paper stronger.
Reviewer_LE6YOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Focus-KV assigns fixed, unequal KV-cache capacities to heads before inference:

  1. Select 100 long, difficult mathematical reasoning examples.
  2. Disable each query head individually and measure the increase in next-token cross-entropy.
  3. Aggregate query-head scores into KV-head scores using the maximum within each GQA group.
  4. Allocate a minimum budget to every head, distribute extra capacity according to importance, and cap individual allocations.
  5. During decoding, reuse R-KV’s token-scoring and eviction procedure within those budgets. Experiments cover Qwen3-8B, Qwen3-14B, and DeepSeek-R1-Distill-Llama-8B, primarily on MATH-500 and AIME24, with additional cross-domain evaluations.
Critical strengths
  • The accuracy improvements are meaningful in several settings.
  • Measuring the effect of suppressing a head provides direct evidence of its contribution to the evaluated loss
Critical weaknesses
  • "reasoning ability is concentrated in a sparse subset of attention heads,". This statement resonates with another similar work, ThinKV [1]. It would be useful for the authors to compare and contrast against it as related work.

  • The novelty and baseline coverage are insufficiently established. RLKV already studies reasoning-critical heads; HeadKV-R2 already combines reasoning-oriented profiling with nonuniform allocation; LU-KV [2] already supports offline, static head-wise allocation. RLKV and HeadKV are cited but not evaluated, while LU-KV is absent. The experiments establish improvements over the chosen baselines, but do not isolate the advance over the closest alternatives.

  • Removing a head measures the value of its entire output. It does not establish how much that head benefits from retaining another 128 or 512 historical tokens. A crucial head could work with a short window, while a less influential head could require distant context. Direct cache-restriction interventions or loss-versus-budget curves would better justify allocation.

  • Focus-KV requires offline profiling through individual query-head ablations on a curated reasoning dataset, but the associated computational cost is not reported. This limits assessment of its practical advantage over other offline head-identification methods. Although the paper demonstrates cross-domain transfer and stability between 100 and 300 probes, robustness to independently sampled probe sets, difficulty distributions, and reasoning lengths remains unclear. Since the resulting budgets remain fixed during decoding, calibration errors could systematically underallocate cache to heads needed by particular inputs or reasoning stages.

  • Although Focus-KV demonstrates accuracy improvements over R-KV, its end-to-end efficiency evaluation is limited to Full-KV and ReasonAlloc. Throughput and latency comparisons against R-KV would clarify whether the proposed allocation improves practical inference performance beyond accuracy. Comparison with reasoning-aware compression methods such as ThinKV would further strengthen the evaluation.

[1] Ramachandran, Akshat, et al. "Thinkv: Thought-adaptive kv cache compression for efficient reasoning models." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Tang, Ziyao, et al. "Predicting Future Utility: Global Combinatorial Optimization for Task-Agnostic KV Cache Eviction." arXiv preprint arXiv:2602.08585 (2026).

Questions for authors

Please see weakness section.

Reviewer_Q6u4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Focus-KV, a decoding-stage KV cache compression method for reasoning models that gives each KV head a fixed, non-uniform token budget. Head importance is estimated offline by suppressing each query head on a 100-sample math probe set and measuring the increase in next-token loss. Budgets are then set as a uniform floor plus an importance-proportional share, capped at κ·B̄. Within each head, tokens are evicted with the R-KV scoring rule. On Qwen3-8B/14B and R1-Distill-Llama-8B, Focus-KV improves over R-KV, ReasonAlloc and Ada-KV on AIME24 and MATH-500, mainly at small budgets, and the math-derived budgets transfer to code and logic benchmarks.

Critical strengths
  • The method is simple and training-free, and it plugs into an existing eviction rule. The head budgets are computed once and reused across tasks.
  • The accuracy evaluation is broad: three models, 6–8 budgets, and 32 samples per question, with full numbers reported (Table 3). Gains at low budgets are consistent.
Critical weaknesses
  • The efficiency claims are not supported at budgets where accuracy is preserved.

In Table 1 and Appendix C, each method runs at its own maximum batch size, and accuracy is not reported alongside throughput. "Memory Saving" is computed as 1 − budget / generation length rather than measured, and it ignores the padding to the per-layer maximum budget that the implementation introduces. At budget 4096, where AIME24 is close to Full-KV (71.92 vs. 74.60), throughput is lower than Full-KV at the same batch size (190.4 vs. 215.6 tok/s). Please report a same-memory comparison, measured peak memory, and accuracy in the same table, and add R-KV to the efficiency table.

  • It is unclear how often compression is actually applied.

All budgets are fixed absolute token counts, and the paper does not report output-length distributions or the fraction of samples that ever exceed the budget. On MATH-500 at budget 2048, Focus-KV, R-KV and Full-KV are within 0.04 points of each other (96.13 / 96.10 / 96.14), which suggests that many samples are never compressed. The reported gains therefore come mostly from budgets of 128–768, where every method is 10–40 points below Full-KV. Please report length distributions and trigger rates, or use length-relative budgets.

  • Positioning against head-level methods is inaccurate, and they are missing as baselines.

The introduction states that head-wise allocation during decoding is "insufficiently explored", but the paper's own baseline ReasonAlloc and the cited HARD-KV are both decoding-time head-adaptive methods. Static, offline head-level allocation has also been studied. HeadKV scores heads offline and allocates a base budget plus an importance-proportional pool, which is close to Eq. 1, yet it is described here as a runtime attention-based method. Other examples are DuoAttention, RazorAttention, PruLong, KVZip, FastKVZip (which also uses next-token loss to select heads), and RLKV (reasoning-oriented head selection). The statement that these methods "provide less flexible budget control" is not tested. Focus-KV also needs offline profiling (L × H_q forward passes over the probe set), and its cost is not reported. Please compare against at least one static head-level method and report the probe cost.

  • The evidence for "reasoning heads" is weak.

Figure 3 compares only the most and least important heads, with no random-head or retrieval-head control. There is no ablation that keeps the same floor and cap but uses random importance scores, which would separate the effect of the importance signal from the effect of a non-uniform budget shape.

Questions for authors

Please address the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.