- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GRIK (Grouped Relevance-Integrated Key pruning), a training- and calibration-free method for reducing KV-cache memory in grouped-query attention (GQA) language models. The paper identifies two structural conditions required to realize an exact reduced-width dense key cache under GQA: channel masks must be shared among query heads belonging to the same KV group and remain consistent across cached tokens.
Based on these constraints, GRIK constructs a prompt-adaptive channel mask for each KV head by combining three importance signals: output-projection influence, key-projection weight norms, and prompt-dependent query/key activation energy. An iterative refinement procedure further adjusts the channel selection.
Experiments on three GQA language models demonstrate substantial K-cache memory reduction with relatively small accuracy changes on LongBench. The method also integrates with token eviction and KV-cache quantization. A fixed-memory H100 serving experiment reports increased concurrent capacity and approximately 1.10× aggregate throughput.
Overall, the paper addresses a practically relevant problem and provides extensive empirical evaluations. However, the algorithmic novelty beyond existing channel-pruning approaches remains limited, and the practical advantages are not yet fully established given the missing comparison with closely related methods, substantial prefill overhead, and inconsistent accuracy preservation across benchmarks.
Critical strengths
S1. Relevant problem formulation and practical storage benefits.
The paper identifies a meaningful limitation of existing channel-pruning implementations when applied to GQA architectures. The distinction between nominal channel sparsity and realized memory savings is important for practical deployment. Enforcing group-shared and token-consistent channel masks provides a straightforward way to obtain a reduced-width dense key cache without duplicating KV representations across query heads.
S2. Extensive empirical evaluation and promising memory–accuracy trade-offs.
The evaluation covers multiple GQA backbones, LongBench, RULER, component ablations, KV-cache memory measurements, and serving performance. The results show that GRIK can reduce total KV-cache storage to approximately 0.80× at a 40% K-channel pruning ratio while largely maintaining LongBench performance. The approximately matched-memory experiments further demonstrate improvements of 1.55–1.62 points over H2O alone. The integration with token eviction and KV quantization also supports the practical relevance of the approach.
Critical weaknesses
W1. Limited Algorithmic Novelty and Insufficient Comparison with Closely Related Methods
The primary contribution of GRIK is enforcing a shared channel mask across query heads and cached tokens, together with a heuristic channel-importance score. However, the two structural conditions largely follow directly from the requirement of representing the pruned keys using a fixed-width dense tensor. The associated memory theorem therefore provides a formal characterization of the storage layout rather than a fundamentally new compression principle.
Similarly, the proposed scoring function combines familiar ingredients, including weight norms, activation statistics, and attention-head importance. The iterative refinement procedure also follows a relatively standard progressive-pruning strategy. The paper does not establish a theoretical connection between the combined score and attention-output approximation error.
More importantly, LeanK (EMNLP 2025) already investigates K-cache channel pruning using shared, static channel masks and demonstrates practical memory and decoding improvements. Although the paper discusses LeanK in related work, it does not provide a direct empirical comparison. The distinction between training-free prompt adaptation and learned static pruning is meaningful, but the benefits of this distinction remain insufficiently quantified.
As a result, it is unclear how much improvement comes from a genuinely new channel-selection principle rather than applying existing importance metrics to a GQA-compatible storage layout.
W2. Insufficient Isolation of the Proposed Scoring Components
Although the paper includes a storage-controlled ThinK-mean baseline, the ablation studies do not completely isolate the contributions of the proposed scoring components.
In particular, Table 7 reports that removing the head-importance weighting term αh reduces performance from 40.12 to 38.11 at B=1024. However, the reported result is numerically identical to the ThinK-mean baseline in Table 6. According to the methodology, ThinK-mean differs from GRIK not only in head weighting but also in its scoring formulation, static prior, and iterative refinement. Therefore, it is unclear whether the reported ablation truly removes only αh while holding the other components fixed.
Moreover, the paper does not adequately establish the importance of prompt adaptation relative to simpler fixed-mask alternatives. A comparison against static masks derived from weight norms or aggregated activation statistics would help determine whether the prompt-dependent scoring mechanism is necessary.
These issues make it difficult to assess which elements of GRIK are responsible for its improvements over simpler group-shared channel-pruning strategies.
W3. Significant Performance Degradation on Some Long-Context and Reasoning Tasks
The near-lossless LongBench results do not consistently extend to other workloads.
In Table 8, GRIK falls behind Full KV by approximately 6.90–8.59 points on the 13-task RULER benchmark, despite the absence of token eviction. Although the eight needle-in-a-haystack tasks retain near-perfect accuracy, the paper reports substantial degradation on tasks requiring global information aggregation, particularly common-word extraction.
The GSM8K evaluation also shows a non-negligible accuracy reduction, from 77.8 for Full KV to 73.5 for GRIK. This suggests that the effects of channel pruning may depend strongly on the type of information processing required.
These findings raise concerns about the generality of the proposed channel-selection criterion. The paper would benefit from a clearer characterization of the workloads for which the proposed pruning strategy can reliably preserve accuracy, especially when the target applications involve reasoning or global context aggregation rather than localized retrieval.
W4. Limited End-to-End Efficiency Advantages and Substantial Prefill Overhead
Although GRIK achieves meaningful KV-cache storage reduction, its practical inference-efficiency advantages are less convincing.
Table 5 reports approximately 1.10× higher aggregate request throughput under a fixed GPU-memory budget. However, this improvement is obtained at increased concurrency, where per-user decoding throughput decreases from 95.2 to 70.4 tokens/s and decoding latency increases from 10.50 to 14.21 ms/token. Since these measurements use different concurrency levels, they do not establish an improvement in decoding efficiency at matched concurrency.
Furthermore, Appendix G reports substantial prefill overhead for the N=32 configuration used in the main experiments, ranging from 53% to 148%. At 16 generated tokens, its end-to-end throughput is only 0.62–0.74× that of Full KV. Although the authors recommend N=4 as a more practical configuration, its prefill latency is not directly measured in the reported experiment.
These results suggest that the practical benefit is primarily increased memory-limited concurrency rather than faster individual inference. A more complete end-to-end evaluation that includes mask computation, refinement, cache packing, and decoding under matched operating conditions is necessary to establish the deployment advantages of GRIK.
Questions for authors
Q1. Comparison with existing shared-mask channel pruning.
Could the authors provide an empirical comparison with LeanK or a simpler static per-KV-head pruning baseline under matched memory budgets? In particular, how much accuracy improvement comes specifically from prompt-dependent channel selection compared with a fixed mask constructed from weight norms or calibration statistics?
Q2. Isolation of head weighting and iterative refinement.
Could the authors clarify the implementation of the “w/o αh” ablation in Table 7? Why are its results identical to ThinK-mean, given that ThinK-mean also differs from GRIK in other scoring components? A controlled ablation that changes only αh, together with a comparison of one-shot and iterative selection, would help establish the independent contribution of each component.
Q3. End-to-end serving efficiency.
Does the fixed-memory serving experiment in Table 5 include the full cost of prompt-dependent channel scoring, iterative refinement, and cache packing? Which refinement configuration is used? Could the authors report end-to-end throughput and decoding latency at matched concurrency for the recommended N=4 configuration, including both short- and long-generation workloads?
Q4. Robustness across different long-context workloads.
Given the substantial degradation on the RULER-13 benchmark despite near-perfect needle-retrieval performance, could the authors provide a more detailed analysis of why the proposed channel-importance score fails on global aggregation tasks? Are these degradations primarily caused by the shared-mask constraint or by the specific scoring criterion? A controlled comparison on the affected tasks could materially clarify the limitations of the method.
4 comments
(2/3)
W2. … Table 7 reports that removing the head-importance weighting term αh reduces performance from 40.12 to 38.11 at B=1024. However, the reported result is numerically identical to the ThinK-mean baseline in Table 6. … it is unclear whether the reported ablation truly removes only αh while holding the other components fixed. Moreover, … A comparison against static masks derived from weight norms or aggregated activation statistics would help …
Q2. Could the authors clarify the implementation of the "w/o αh" ablation in Table 7? … A controlled ablation that changes only αh, together with a comparison of one-shot and iterative selection, would help establish the independent contribution of each component.
The reviewer is right; we ran the requested one-factor ablations. The "w/o α_h (uniform mean)" row of Table 7 (35.45 / 38.11) repeats the ThinK-mean run of Table 6. That run removes α_h, the prior and refinement together, so it is not a one-factor ablation, and we apologize for the mislabeling. We have now run controlled ablations on LLaMA-3-8B at r=0.4, B=1024, changing one factor at a time from GRIK N=4:
| Variant | LongBench-16 |
|---|---|
| GRIK (α_h + prior + N=4) | 40.10 (second GPU: 40.01†) |
| − α_h only (uniform weights; prior and N=4 kept) | 39.76 |
| − prior only | 39.81 (Table 7, N=32) |
| − query energy only | 39.20† |
| − key energy only | 40.26† |
| one-shot (N=1) / N=4 / N=32 | 39.85† / 40.10 / 40.12 |
| additive instead of multiplicative score | 40.12 |
| none of the three (ThinKPress form) | 39.33 |
† Run on a second GPU; the same-GPU GRIK reference is 40.01.
These ablations isolate each component:
- the query energy is the largest single factor (−0.81);
- the head weights α_h contribute independently (−0.34 when removed alone);
- iterative selection helps over one-shot (−0.16 at N=1);
- the static weight-norm alone cannot replace the activation signal (14.54; response 1/3).
The gain over the ThinKPress form (+0.77) therefore comes from GRIK's query-aware, head-weighted score, not from any single familiar ingredient. Two components do not help on LongBench: the key energy (+0.25 when removed) and the product form (additive is equal). We no longer claim either (Sections 4 and 5.4). Table 7 now contains these rows, and the mislabeled row is removed.
W3. In Table 8, GRIK falls behind Full KV by approximately 6.90–8.59 points on the 13-task RULER benchmark, despite the absence of token eviction. … The GSM8K evaluation also shows a non-negligible accuracy reduction, from 77.8 for Full KV to 73.5 for GRIK. …
Both drops disappear under a correct and complete evaluation. To check them, we reran RULER with Full KV in our own runner, and GSM8K on the full test set:
| 4K | 8K | 16K | 32K | GSM8K (1,319 q.) | |
|---|---|---|---|---|---|
| Full KV | 94.74 | 94.03 | 92.41 | 86.16 | 76.65 |
| GRIK r=0.4 | 94.85 | 94.01 | 92.13 | 86.08 | 75.74 (N=4), 75.97 (N=32) |
The RULER gap was an evaluation error in our runner, which omitted RULER's answer_prefix. That zeroes variable tracking for every method, while the Full KV row came from the official harness. With the fix, GRIK is within 0.3 points of Full KV at every length. On GSM8K, the difference is not significant (exact McNemar p=0.38 / 0.51); the 234-question subset (77.8 vs 73.5) overstated it. GRIK therefore preserves both global-context and multi-step reasoning accuracy at r=0.4.
(3/3)
Q4. … could the authors provide a more detailed analysis of why the proposed channel-importance score fails on global aggregation tasks? Are these degradations primarily caused by the shared-mask constraint or by the specific scoring criterion? A controlled comparison on the affected tasks could materially clarify the limitations of the method.
After correcting the evaluation, there is no aggregation failure at r=0.4; at higher ratios, the loss comes from removing channels, not from the shared mask.
- Common-word extraction is 98.8 vs 98.9 (4K) and 93.65 vs 93.6 (8K) for GRIK vs Full KV.
- At 16K it is 70.8 vs 73.95 (−3.2), which accounts for most of the 0.28 gap in the 16K average. All other 16K tasks are within 0.5.
- At 32K the task is near floor for Full KV itself (7.2 vs 5.15), a limit of the base model.
Higher ratios do hurt aggregation. At r=0.7, GRIK scores 62.20 / 58.53 at 4K / 8K. Common-word extraction falls from 98.8 to 26.6 and variable tracking from 99.8 to 25.2 (4K), while the needle tasks stay high.
On shared mask vs score: per-query-head ThinK, which violates C1 and lets every head choose its own channels, scores 58.30 / 39.20 at 4K / 8K at r=0.7 (LeanK's numbers, official harness). Giving each head its own mask therefore does not prevent the loss, and the shared-mask GRIK degrades less. The loss tracks how many channels are removed, not the sharing constraint. We recommend r ≤ 0.4 for retrieval and aggregation workloads and state this in the limitations.
W4. … this improvement is obtained at increased concurrency, where per-user decoding throughput decreases from 95.2 to 70.4 tokens/s … Although the authors recommend N=4 as a more practical configuration, its prefill latency is not directly measured … A more complete end-to-end evaluation that includes mask computation, refinement, cache packing, and decoding under matched operating conditions is necessary …
Q3. Does the fixed-memory serving experiment in Table 5 include the full cost of prompt-dependent channel scoring, iterative refinement, and cache packing? Which refinement configuration is used? Could the authors report end-to-end throughput and decoding latency at matched concurrency for the recommended N=4 configuration, including both short- and long-generation workloads?
Table 5 excluded mask selection, so we built a matched engine and measured every cost.
- What Table 5 includes. Table 5 compares a mature FA3 path for Full KV with our first asymmetric path, at different concurrency, as the reviewer notes. It did not include scoring, refinement or packing. The serving integration used one fixed mask per layer and KV head computed from the weights, because a per-prompt mask cannot be kept consistent in a paged cache shared across requests. N therefore does not apply to Table 5.
- New engine. To answer directly, we built one engine in which Full KV and GRIK share the paged cache, the attention kernel (a GQA-shared split-K kernel; at D_k=128 it matches PyTorch SDPA within 4%) and the model code. Masks are computed from the prompt inside the engine, and outputs match HF greedy decoding.
Equal concurrency, per-user decode tokens/s (LLaMA-3.1-8B, GB10, r=0.5):
| Context × batch | Full KV | GRIK | Δ |
|---|---|---|---|
| 4K × 32 | 6.89 | 7.70 | +12% |
| 16K × 16 | 4.68 | 5.51 | +18% |
| 32K × 8 | 4.68 | 5.53 | +18% |
Equal KV memory (48 GiB), each at its largest batch: GRIK fits 1.34× more sequences, with +27% / +31% / +32% aggregate throughput at 4K / 16K / 32K and −3% to −5% per-user speed.
End to end, N=4 (request time vs Full KV, including prefill, mask selection, refinement and packing):
| r, context, batch | 16 output tokens | 128 | 512 |
|---|---|---|---|
| r=0.5, 4K, 1 | +6.9% | +2.1% | +0.9% |
| r=0.5, 4K, 8 | +11.8% | +4.3% | −0.9% |
| r=0.5, 16K, 8 | +9.3% | +3.7% | −3.5% |
| r=0.4, 16K, 8 | +9.9% | +7.9% | +5.3% |
- Short generations. The N=4 prefill (+27% / +17% at 4K / 8K; response to KiiC) dominates, and GRIK is slower.
- r=0.5. A decode step is 4–11% faster at batch 8, so the cost is repaid after a few hundred output tokens.
- r=0.4. K is stored at 80 channels, which is not aligned for this kernel, so decode is 1–3% slower and requests stay slower. r=0.5 (64 channels) is therefore the latency setting; its LongBench is within 0.3–0.5 of r=0.4.
These measurements answer the matched-concurrency question directly:
- At the same batch, GRIK decodes faster per user (+12–18%).
- Under a memory limit, it serves 1.34× more sequences with +27–32% throughput.
- With the full prefill cost included, it is slower only for short outputs.
The deployment benefit is thus established under matched conditions. We state the short-output regime as a limitation. All engine results are in Appendix M (highlighted in red).
Dear Reviewer ewjC,
Thank you for raising your score; we are glad that our responses were helpful.
As the discussion period is coming to an end, we would like to ask whether any concerns remain, in particular regarding the matched-memory comparison with LeanK and static masks (Q1), the controlled one-factor α_h ablation (Q2), the RULER gap, which we traced to an evaluation error (GRIK is now within 0.3 points of Full KV from 4K to 32K), and the full GSM8K test set (Q4), and the serving results that include the full cost of mask selection and packing (Q3). If these points are now resolved, we would appreciate it if you could consider reflecting this in your final assessment.
Otherwise, we would be happy to clarify any remaining points.
Best regards, Authors
(1/3) We sincerely thank Reviewer ewjC for the detailed analysis and constructive feedback, and for finding the formulation relevant and the evaluation extensive. Below, we provide detailed point-by-point responses; all revisions are highlighted in red in the manuscript.
We first note one correction that affects a strength the reviewer listed. The matched-memory gain over H2O is +0.43 to +0.53, not +1.55 to +1.62. The old H2O-only numbers came from a different run; rerun in the same harness they are 0.9–1.4 points higher (Shared Results, item 3). The new gain is significant at the example level (95% CI [+0.13, +0.87] at B=512 and [+0.16, +0.91] at B=1024).
We agree on the theory and quantified both comparisons the reviewer asks for. C1/C2 follow from requiring a fixed-width dense tensor, and the theorem characterizes a layout rather than a new compression principle. The revision says so. We have no theorem linking the score to attention-output error, so we present it as a heuristic and test it empirically. To quantify the two open questions (learned static vs training-free, and prompt-dependent vs fixed), we ran the following comparisons at matched memory.
Prompt-dependent vs fixed masks (LLaMA-3-8B, r=0.4, B=1024, same layout, bytes and GPU):
A weight-only fixed mask fails, so the activation signal is essential. A calibration mask recovers most of the quality (+0.14 below GRIK), but only with calibration data. Prompt adaptation therefore gives the best score while requiring neither training nor data. We state this in Section 5.4 and no longer claim that prompt adaptation is necessary for accuracy.
LeanK in the same harness (LeanK's released LLaMA-3.1-8B mask and cache policy, equal K bytes, ≈69% of K channels pruned):
With the per-head widths held equal, GRIK's training-free channels are not significantly different from LeanK's learned ones. The benefit of learning therefore lies mainly in LeanK's per-head width allocation (87% of its lead over uniform-width GRIK), not in the channel choice. GRIK's own training-free allocation recovers most of that width effect (45.64). On RULER-13 at 4K, GRIK choosing channels and widths within LeanK's per-layer budget scores 93.95, vs 93.76 for LeanK (Appendix M).