acceptodds
Under review as a conference paper at ICLR 2027

GRIK: Grouped Relevance-Integrated Key Pruning for Grouped-Query Attention LLMs

Abstract

The key–value (KV) cache is a major memory bottleneck in long-context inference, and existing K-channel pruning does not shrink it under grouped-query attention (GQA): in the evaluated implementations, ThinK materializes K and V per query head ( total KV) and SparK reconstructs a full-width cache (). We identify two structural conditions, a mask shared across the query heads of each KV group and across all cached tokens, under which channel pruning yields a dense shared-key cache whose width equals the retained channel count. We introduce GRIK, a training- and calibration-free method that satisfies both conditions by construction and derives a prompt-adaptive mask per KV head from output-projection influence, key-projection norms, and attention energy. At 40% K-channel pruning, GRIK stores K at and total K+V at the bf16 baseline. Without token eviction, its LongBench average is within points of Full KV on three GQA backbones; with H2O eviction it is the best non-Full-KV method in 19 of 24 (model, budget, ratio) cells, and at approximately matched total-KV memory it improves over H2O alone by – points. The reduction composes with other axes: adding GRIK to SnapKV, FP8, or int4 KV changes LongBench by at most points while cutting total KV to as low as . On RULER, its eight exact-retrieval tasks average at least through 32K, although the 13-task average trails Full KV by – points. In a fixed-memory H100 serving test at 50% pruning, the smaller cache admits more concurrent sequences and raises aggregate request throughput by , with per-user decode latency that of the baseline at the higher concurrency. Code and configurations are available at https://anonymous.4open.science/r/GRIK/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ewjCOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRIK (Grouped Relevance-Integrated Key pruning), a training- and calibration-free method for reducing KV-cache memory in grouped-query attention (GQA) language models. The paper identifies two structural conditions required to realize an exact reduced-width dense key cache under GQA: channel masks must be shared among query heads belonging to the same KV group and remain consistent across cached tokens.

Based on these constraints, GRIK constructs a prompt-adaptive channel mask for each KV head by combining three importance signals: output-projection influence, key-projection weight norms, and prompt-dependent query/key activation energy. An iterative refinement procedure further adjusts the channel selection.

Experiments on three GQA language models demonstrate substantial K-cache memory reduction with relatively small accuracy changes on LongBench. The method also integrates with token eviction and KV-cache quantization. A fixed-memory H100 serving experiment reports increased concurrent capacity and approximately 1.10× aggregate throughput.

Overall, the paper addresses a practically relevant problem and provides extensive empirical evaluations. However, the algorithmic novelty beyond existing channel-pruning approaches remains limited, and the practical advantages are not yet fully established given the missing comparison with closely related methods, substantial prefill overhead, and inconsistent accuracy preservation across benchmarks.

Critical strengths

S1. Relevant problem formulation and practical storage benefits.

The paper identifies a meaningful limitation of existing channel-pruning implementations when applied to GQA architectures. The distinction between nominal channel sparsity and realized memory savings is important for practical deployment. Enforcing group-shared and token-consistent channel masks provides a straightforward way to obtain a reduced-width dense key cache without duplicating KV representations across query heads.

S2. Extensive empirical evaluation and promising memory–accuracy trade-offs.

The evaluation covers multiple GQA backbones, LongBench, RULER, component ablations, KV-cache memory measurements, and serving performance. The results show that GRIK can reduce total KV-cache storage to approximately 0.80× at a 40% K-channel pruning ratio while largely maintaining LongBench performance. The approximately matched-memory experiments further demonstrate improvements of 1.55–1.62 points over H2O alone. The integration with token eviction and KV quantization also supports the practical relevance of the approach.

Critical weaknesses

W1. Limited Algorithmic Novelty and Insufficient Comparison with Closely Related Methods

The primary contribution of GRIK is enforcing a shared channel mask across query heads and cached tokens, together with a heuristic channel-importance score. However, the two structural conditions largely follow directly from the requirement of representing the pruned keys using a fixed-width dense tensor. The associated memory theorem therefore provides a formal characterization of the storage layout rather than a fundamentally new compression principle.

Similarly, the proposed scoring function combines familiar ingredients, including weight norms, activation statistics, and attention-head importance. The iterative refinement procedure also follows a relatively standard progressive-pruning strategy. The paper does not establish a theoretical connection between the combined score and attention-output approximation error.

More importantly, LeanK (EMNLP 2025) already investigates K-cache channel pruning using shared, static channel masks and demonstrates practical memory and decoding improvements. Although the paper discusses LeanK in related work, it does not provide a direct empirical comparison. The distinction between training-free prompt adaptation and learned static pruning is meaningful, but the benefits of this distinction remain insufficiently quantified.

As a result, it is unclear how much improvement comes from a genuinely new channel-selection principle rather than applying existing importance metrics to a GQA-compatible storage layout.

W2. Insufficient Isolation of the Proposed Scoring Components

Although the paper includes a storage-controlled ThinK-mean baseline, the ablation studies do not completely isolate the contributions of the proposed scoring components.

In particular, Table 7 reports that removing the head-importance weighting term αh reduces performance from 40.12 to 38.11 at B=1024. However, the reported result is numerically identical to the ThinK-mean baseline in Table 6. According to the methodology, ThinK-mean differs from GRIK not only in head weighting but also in its scoring formulation, static prior, and iterative refinement. Therefore, it is unclear whether the reported ablation truly removes only αh while holding the other components fixed.

Moreover, the paper does not adequately establish the importance of prompt adaptation relative to simpler fixed-mask alternatives. A comparison against static masks derived from weight norms or aggregated activation statistics would help determine whether the prompt-dependent scoring mechanism is necessary.

These issues make it difficult to assess which elements of GRIK are responsible for its improvements over simpler group-shared channel-pruning strategies.

W3. Significant Performance Degradation on Some Long-Context and Reasoning Tasks

The near-lossless LongBench results do not consistently extend to other workloads.

In Table 8, GRIK falls behind Full KV by approximately 6.90–8.59 points on the 13-task RULER benchmark, despite the absence of token eviction. Although the eight needle-in-a-haystack tasks retain near-perfect accuracy, the paper reports substantial degradation on tasks requiring global information aggregation, particularly common-word extraction.

The GSM8K evaluation also shows a non-negligible accuracy reduction, from 77.8 for Full KV to 73.5 for GRIK. This suggests that the effects of channel pruning may depend strongly on the type of information processing required.

These findings raise concerns about the generality of the proposed channel-selection criterion. The paper would benefit from a clearer characterization of the workloads for which the proposed pruning strategy can reliably preserve accuracy, especially when the target applications involve reasoning or global context aggregation rather than localized retrieval.

W4. Limited End-to-End Efficiency Advantages and Substantial Prefill Overhead

Although GRIK achieves meaningful KV-cache storage reduction, its practical inference-efficiency advantages are less convincing.

Table 5 reports approximately 1.10× higher aggregate request throughput under a fixed GPU-memory budget. However, this improvement is obtained at increased concurrency, where per-user decoding throughput decreases from 95.2 to 70.4 tokens/s and decoding latency increases from 10.50 to 14.21 ms/token. Since these measurements use different concurrency levels, they do not establish an improvement in decoding efficiency at matched concurrency.

Furthermore, Appendix G reports substantial prefill overhead for the N=32 configuration used in the main experiments, ranging from 53% to 148%. At 16 generated tokens, its end-to-end throughput is only 0.62–0.74× that of Full KV. Although the authors recommend N=4 as a more practical configuration, its prefill latency is not directly measured in the reported experiment.

These results suggest that the practical benefit is primarily increased memory-limited concurrency rather than faster individual inference. A more complete end-to-end evaluation that includes mask computation, refinement, cache packing, and decoding under matched operating conditions is necessary to establish the deployment advantages of GRIK.

Questions for authors

Q1. Comparison with existing shared-mask channel pruning.

Could the authors provide an empirical comparison with LeanK or a simpler static per-KV-head pruning baseline under matched memory budgets? In particular, how much accuracy improvement comes specifically from prompt-dependent channel selection compared with a fixed mask constructed from weight norms or calibration statistics?

Q2. Isolation of head weighting and iterative refinement.

Could the authors clarify the implementation of the “w/o αh” ablation in Table 7? Why are its results identical to ThinK-mean, given that ThinK-mean also differs from GRIK in other scoring components? A controlled ablation that changes only αh, together with a comparison of one-shot and iterative selection, would help establish the independent contribution of each component.

Q3. End-to-end serving efficiency.

Does the fixed-memory serving experiment in Table 5 include the full cost of prompt-dependent channel scoring, iterative refinement, and cache packing? Which refinement configuration is used? Could the authors report end-to-end throughput and decoding latency at matched concurrency for the recommended N=4 configuration, including both short- and long-generation workloads?

Q4. Robustness across different long-context workloads.

Given the substantial degradation on the RULER-13 benchmark despite near-perfect needle-retrieval performance, could the authors provide a more detailed analysis of why the proposed channel-importance score fails on global aggregation tasks? Are these degradations primarily caused by the shared-mask constraint or by the specific scoring criterion? A controlled comparison on the affected tasks could materially clarify the limitations of the method.

4 comments

(1/3) We sincerely thank Reviewer ewjC for the detailed analysis and constructive feedback, and for finding the formulation relevant and the evaluation extensive. Below, we provide detailed point-by-point responses; all revisions are highlighted in red in the manuscript.

We first note one correction that affects a strength the reviewer listed. The matched-memory gain over H2O is +0.43 to +0.53, not +1.55 to +1.62. The old H2O-only numbers came from a different run; rerun in the same harness they are 0.9–1.4 points higher (Shared Results, item 3). The new gain is significant at the example level (95% CI [+0.13, +0.87] at B=512 and [+0.16, +0.91] at B=1024).

W1. … the two structural conditions largely follow directly from the requirement of representing the pruned keys using a fixed-width dense tensor. … The paper does not establish a theoretical connection between the combined score and attention-output approximation error. More importantly, LeanK (EMNLP 2025) … does not provide a direct empirical comparison. The distinction between training-free prompt adaptation and learned static pruning is meaningful, but the benefits of this distinction remain insufficiently quantified.

Q1. Could the authors provide an empirical comparison with LeanK or a simpler static per-KV-head pruning baseline under matched memory budgets? In particular, how much accuracy improvement comes specifically from prompt-dependent channel selection compared with a fixed mask constructed from weight norms or calibration statistics?

We agree on the theory and quantified both comparisons the reviewer asks for. C1/C2 follow from requiring a fixed-width dense tensor, and the theorem characterizes a layout rather than a new compression principle. The revision says so. We have no theorem linking the score to attention-output error, so we present it as a heuristic and test it empirically. To quantify the two open questions (learned static vs training-free, and prompt-dependent vs fixed), we ran the following comparisons at matched memory.

Prompt-dependent vs fixed masks (LLaMA-3-8B, r=0.4, B=1024, same layout, bytes and GPU):

MaskTraining / dataLongBench-16
Fixed, weight norms ‖W_k‖none14.54
Fixed, calibration-averaged GRIK score (40 synthetic prompts)calibration prompts39.81
Prompt-adaptive GRIK, N=4none39.95

A weight-only fixed mask fails, so the activation signal is essential. A calibration mask recovers most of the quality (+0.14 below GRIK), but only with calibration data. Prompt adaptation therefore gives the best score while requiring neither training nor data. We state this in Section 5.4 and no longer claim that prompt adaptation is necessary for accuracy.

LeanK in the same harness (LeanK's released LLaMA-3.1-8B mask and cache policy, equal K bytes, ≈69% of K channels pruned):

TrainingLongBench-16
Full KV—46.07
LeanK (released mask)yes46.20
GRIK, uniform widthsnone44.24
GRIK channels + LeanK's per-head widthsnone for channels45.95 (95% CI [−0.57, +0.08])
GRIK channels + GRIK-allocated widths (equal per-layer budget)none45.64

With the per-head widths held equal, GRIK's training-free channels are not significantly different from LeanK's learned ones. The benefit of learning therefore lies mainly in LeanK's per-head width allocation (87% of its lead over uniform-width GRIK), not in the channel choice. GRIK's own training-free allocation recovers most of that width effect (45.64). On RULER-13 at 4K, GRIK choosing channels and widths within LeanK's per-layer budget scores 93.95, vs 93.76 for LeanK (Appendix M).

(2/3)

W2. … Table 7 reports that removing the head-importance weighting term αh reduces performance from 40.12 to 38.11 at B=1024. However, the reported result is numerically identical to the ThinK-mean baseline in Table 6. … it is unclear whether the reported ablation truly removes only αh while holding the other components fixed. Moreover, … A comparison against static masks derived from weight norms or aggregated activation statistics would help …

Q2. Could the authors clarify the implementation of the "w/o αh" ablation in Table 7? … A controlled ablation that changes only αh, together with a comparison of one-shot and iterative selection, would help establish the independent contribution of each component.

The reviewer is right; we ran the requested one-factor ablations. The "w/o α_h (uniform mean)" row of Table 7 (35.45 / 38.11) repeats the ThinK-mean run of Table 6. That run removes α_h, the prior and refinement together, so it is not a one-factor ablation, and we apologize for the mislabeling. We have now run controlled ablations on LLaMA-3-8B at r=0.4, B=1024, changing one factor at a time from GRIK N=4:

VariantLongBench-16
GRIK (α_h + prior + N=4)40.10 (second GPU: 40.01†)
− α_h only (uniform weights; prior and N=4 kept)39.76
− prior only39.81 (Table 7, N=32)
− query energy only39.20†
− key energy only40.26†
one-shot (N=1) / N=4 / N=3239.85† / 40.10 / 40.12
additive instead of multiplicative score40.12
none of the three (ThinKPress form)39.33

† Run on a second GPU; the same-GPU GRIK reference is 40.01.

These ablations isolate each component:

  • the query energy is the largest single factor (−0.81);
  • the head weights α_h contribute independently (−0.34 when removed alone);
  • iterative selection helps over one-shot (−0.16 at N=1);
  • the static weight-norm alone cannot replace the activation signal (14.54; response 1/3).

The gain over the ThinKPress form (+0.77) therefore comes from GRIK's query-aware, head-weighted score, not from any single familiar ingredient. Two components do not help on LongBench: the key energy (+0.25 when removed) and the product form (additive is equal). We no longer claim either (Sections 4 and 5.4). Table 7 now contains these rows, and the mislabeled row is removed.

W3. In Table 8, GRIK falls behind Full KV by approximately 6.90–8.59 points on the 13-task RULER benchmark, despite the absence of token eviction. … The GSM8K evaluation also shows a non-negligible accuracy reduction, from 77.8 for Full KV to 73.5 for GRIK. …

Both drops disappear under a correct and complete evaluation. To check them, we reran RULER with Full KV in our own runner, and GSM8K on the full test set:

4K8K16K32KGSM8K (1,319 q.)
Full KV94.7494.0392.4186.1676.65
GRIK r=0.494.8594.0192.1386.0875.74 (N=4), 75.97 (N=32)

The RULER gap was an evaluation error in our runner, which omitted RULER's answer_prefix. That zeroes variable tracking for every method, while the Full KV row came from the official harness. With the fix, GRIK is within 0.3 points of Full KV at every length. On GSM8K, the difference is not significant (exact McNemar p=0.38 / 0.51); the 234-question subset (77.8 vs 73.5) overstated it. GRIK therefore preserves both global-context and multi-step reasoning accuracy at r=0.4.

(3/3)

Q4. … could the authors provide a more detailed analysis of why the proposed channel-importance score fails on global aggregation tasks? Are these degradations primarily caused by the shared-mask constraint or by the specific scoring criterion? A controlled comparison on the affected tasks could materially clarify the limitations of the method.

After correcting the evaluation, there is no aggregation failure at r=0.4; at higher ratios, the loss comes from removing channels, not from the shared mask.

  • Common-word extraction is 98.8 vs 98.9 (4K) and 93.65 vs 93.6 (8K) for GRIK vs Full KV.
  • At 16K it is 70.8 vs 73.95 (−3.2), which accounts for most of the 0.28 gap in the 16K average. All other 16K tasks are within 0.5.
  • At 32K the task is near floor for Full KV itself (7.2 vs 5.15), a limit of the base model.

Higher ratios do hurt aggregation. At r=0.7, GRIK scores 62.20 / 58.53 at 4K / 8K. Common-word extraction falls from 98.8 to 26.6 and variable tracking from 99.8 to 25.2 (4K), while the needle tasks stay high.

On shared mask vs score: per-query-head ThinK, which violates C1 and lets every head choose its own channels, scores 58.30 / 39.20 at 4K / 8K at r=0.7 (LeanK's numbers, official harness). Giving each head its own mask therefore does not prevent the loss, and the shared-mask GRIK degrades less. The loss tracks how many channels are removed, not the sharing constraint. We recommend r ≤ 0.4 for retrieval and aggregation workloads and state this in the limitations.

W4. … this improvement is obtained at increased concurrency, where per-user decoding throughput decreases from 95.2 to 70.4 tokens/s … Although the authors recommend N=4 as a more practical configuration, its prefill latency is not directly measured … A more complete end-to-end evaluation that includes mask computation, refinement, cache packing, and decoding under matched operating conditions is necessary …

Q3. Does the fixed-memory serving experiment in Table 5 include the full cost of prompt-dependent channel scoring, iterative refinement, and cache packing? Which refinement configuration is used? Could the authors report end-to-end throughput and decoding latency at matched concurrency for the recommended N=4 configuration, including both short- and long-generation workloads?

Table 5 excluded mask selection, so we built a matched engine and measured every cost.

  • What Table 5 includes. Table 5 compares a mature FA3 path for Full KV with our first asymmetric path, at different concurrency, as the reviewer notes. It did not include scoring, refinement or packing. The serving integration used one fixed mask per layer and KV head computed from the weights, because a per-prompt mask cannot be kept consistent in a paged cache shared across requests. N therefore does not apply to Table 5.
  • New engine. To answer directly, we built one engine in which Full KV and GRIK share the paged cache, the attention kernel (a GQA-shared split-K kernel; at D_k=128 it matches PyTorch SDPA within 4%) and the model code. Masks are computed from the prompt inside the engine, and outputs match HF greedy decoding.

Equal concurrency, per-user decode tokens/s (LLaMA-3.1-8B, GB10, r=0.5):

Context × batchFull KVGRIKΔ
4K × 326.897.70+12%
16K × 164.685.51+18%
32K × 84.685.53+18%

Equal KV memory (48 GiB), each at its largest batch: GRIK fits 1.34× more sequences, with +27% / +31% / +32% aggregate throughput at 4K / 16K / 32K and −3% to −5% per-user speed.

End to end, N=4 (request time vs Full KV, including prefill, mask selection, refinement and packing):

r, context, batch16 output tokens128512
r=0.5, 4K, 1+6.9%+2.1%+0.9%
r=0.5, 4K, 8+11.8%+4.3%−0.9%
r=0.5, 16K, 8+9.3%+3.7%−3.5%
r=0.4, 16K, 8+9.9%+7.9%+5.3%
  • Short generations. The N=4 prefill (+27% / +17% at 4K / 8K; response to KiiC) dominates, and GRIK is slower.
  • r=0.5. A decode step is 4–11% faster at batch 8, so the cost is repaid after a few hundred output tokens.
  • r=0.4. K is stored at 80 channels, which is not aligned for this kernel, so decode is 1–3% slower and requests stay slower. r=0.5 (64 channels) is therefore the latency setting; its LongBench is within 0.3–0.5 of r=0.4.

These measurements answer the matched-concurrency question directly:

  • At the same batch, GRIK decodes faster per user (+12–18%).
  • Under a memory limit, it serves 1.34× more sequences with +27–32% throughput.
  • With the full prefill cost included, it is slower only for short outputs.

The deployment benefit is thus established under matched conditions. We state the short-output regime as a limitation. All engine results are in Appendix M (highlighted in red).

Dear Reviewer ewjC,

Thank you for raising your score; we are glad that our responses were helpful.

As the discussion period is coming to an end, we would like to ask whether any concerns remain, in particular regarding the matched-memory comparison with LeanK and static masks (Q1), the controlled one-factor α_h ablation (Q2), the RULER gap, which we traced to an evaluation error (GRIK is now within 0.3 points of Full KV from 4K to 32K), and the full GSM8K test set (Q4), and the serving results that include the full cost of mask selection and packing (Q3). If these points are now resolved, we would appreciate it if you could consider reflecting this in your final assessment.

Otherwise, we would be happy to clarify any remaining points.

Best regards, Authors

Reviewer_KiiCOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

GRIK studies when key-channel pruning actually reduces KV-cache memory in GQA models. The paper argues that all query heads in a KV group must share one channel mask and that this mask must be the same for all cached tokens. Only then can K be stored as a smaller dense tensor. GRIK selects one mask per KV head at prefill without training. The score combines a static key-projection norm prior, prompt query and key energies, and per-head weights based on output contribution, refined over N steps. Only K is pruned, so at r=0.4 the K cache is 0.60x and total K+V is 0.80x. Experiments on three 8B GQA models cover LongBench with and without H2O eviction, a same-layout baseline (ThinK-mean), approximately memory-matched runs, combination with SnapKV, FP8 and int4, RULER, prefill time on a GB10 GPU, and an H100 serving test at r=0.5. The results support real memory savings and higher aggregate throughput when KV memory limits concurrency, with a clear RULER loss and higher per-user latency.

Critical strengths
  1. The paper clearly separates channel sparsity from memory that is actually saved. The two conditions are simple, and the byte accounting matches the stored tensors.

  2. The method is simple, works for any checkpoint, and keeps a dense cache, so it combines directly with token eviction and quantization.

  3. The evaluation includes useful controls: a no-eviction run that isolates the channel effect, a same-layout baseline, an approximately memory-matched comparison, and a real serving test.

Critical weaknesses
  1. The novelty is narrower than claimed, and static masks are not compared. A group-shared, token-constant key mask already exists in NVIDIA kvpress ThinKPress [4], which scores each KV head by group-averaged query energy times key energy. This is Eq. 2 with uniform head weights and no norm prior. ThinK's channel score [1] also equals ||q_j|| * ||k_j|| [1], so E_q * E_k is its square. LeanK [3] already stores a packed per-KV-head K cache. The new parts are therefore the head weights, the norm prior, and the refinement. LeanK also finds K-channel importance largely static across inputs, and reports near Full-KV RULER scores at 70% pruning on LLaMA-3.1-8B, while GRIK loses 6.9 to 8.6 points at 40% pruning. A static mask has no prefill cost, so it is the natural check on whether per-prompt masks are worth +53% to +148% prefill time.

  2. The cost of the recommended N=4 setting is not shown. Appendix G states that N=4 prefill was not measured, but Section 5.3 says N=4 "lies between". Table 5 does not state N, and its 4% TTFT increase at 2K input is below even the N=1 overhead in Table 13 (+8% to +14%, on a different GPU). It is unclear whether mask selection is included. Since ThinK-mean uses the same layout, cache size and decode speed are identical, and the open trade-off is quality versus prefill time.

  3. Several baselines are borrowed or use non-default settings. On LLaMA-3-8B, the Full KV and ThinK rows match Table 2 of [1] exactly, while the other methods are the authors' runs on different hardware and software. GRIK then beats Full KV in all four cells by 0.54 to 1.71 points, mostly through TREC (71.0 vs. 53.0 at B=2048, Table 14). The RULER Full KV row (Table 8) and ThinK r=0.7 row (Table 21) are identical to LeanK's table [3], without attribution. SparK [2] is run with Gaussian recovery, but the SparK paper's default is mean-only recovery, under which SparK stays close to the eviction baseline and above ThinK.

Minor: Table 20 lists ThinK (r=0.6, B=1024) as 38.57, while Tables 2 and 14 and [1] give 37.58. App. I gives a gap range of [-0.27, -0.19], but Table 2 gives -0.21 to -0.27. App. E.3 cites thunlp/ThinK, while the official code is SalesforceAIResearch/ThinK. The N=1 row in Table 7 is empty.

Reference:

[1] Y. Xu et al. ThinK: Thinner Key Cache by Query-Driven Pruning. ICLR 2025. arXiv:2407.21018.

[2] H. Liao et al. SparK: Query-Aware Unstructured Sparsity with Recoverable KV Cache Channel Pruning. AAAI 2026. arXiv:2508.15212.

[3] Y. Zhang et al. LeanK: Learnable K Cache Channel Pruning for Efficient Decoding. EMNLP 2025. arXiv:2508.02215.

[4] NVIDIA. kvpress, ThinKPress (kvpress/presses/think_press.py). https://github.com/NVIDIA/kvpress

Questions for authors
  1. Which N does Table 5 use, and is mask selection included in TTFT? Please report N=4 and ThinK-mean prefill time on the same GPU, and compare quality and end-to-end latency against ThinK-mean on one workload at the same concurrency.

  2. How does GRIK compare with kvpress ThinKPress [4], a static mask (w_k only, or energies averaged on calibration prompts), and LeanK [3]? A one-factor alpha_h ablation that keeps the prior and N fixed would also help.

  3. Please state the source of each borrowed baseline row and rerun them in the same setup if possible. Does SparK still collapse on Qwen-2.5 with its default mean-only recovery?

  4. Please correct the significance statements to match Table 19. What loss is considered acceptable? An equivalence test with a stated margin and an example-level bootstrap would better support near-lossless claims.

4 comments

(1/3) We sincerely thank Reviewer KiiC for the precise and constructive review, and for recognizing that the paper separates channel sparsity from realized memory and provides useful controls. The review identified several issues that we have now corrected. Below, we provide detailed point-by-point responses; all revisions are highlighted in red in the manuscript.

W1. The novelty is narrower than claimed, and static masks are not compared. A group-shared, token-constant key mask already exists in NVIDIA kvpress ThinKPress [4] … This is Eq. 2 with uniform head weights and no norm prior. … The new parts are therefore the head weights, the norm prior, and the refinement. LeanK also finds K-channel importance largely static across inputs, and reports near Full-KV RULER scores at 70% pruning on LLaMA-3.1-8B, while GRIK loses 6.9 to 8.6 points at 40% pruning. A static mask has no prefill cost …

Q2. How does GRIK compare with kvpress ThinKPress [4], a static mask (w_k only, or energies averaged on calibration prompts), and LeanK [3]? A one-factor alpha_h ablation that keeps the prior and N fixed would also help.

We agree with this decomposition and ran every comparison requested. ThinKPress is Eq. 2 with uniform head weights, no prior and one step. We now cite kvpress and describe our ThinK-mean baseline this way (Section 6), and we claim only α_h, the prior and refinement as new. To measure what these add, we compared all variants at identical layout and bytes (LLaMA-3-8B, r=0.4, B=1024):

Mask (same layout and bytes)Training / dataLongBench-16
Static, ‖W_k‖ onlynone14.54
kvpress ThinKPress formnone39.33
GRIK without α_h only (prior and N=4 kept)none39.76
Static, calibration-averaged GRIK scorecalibration prompts39.81†
GRIK, N=4 (prompt-adaptive)none40.10 (39.95†)

† Same second GPU.

These results resolve the three questions:

  • Novelty. GRIK improves on ThinKPress by +0.77 at identical bytes, and the one-factor α_h ablation alone accounts for +0.34. The new components are therefore measurable gains, not relabeling. Across all LLaMA-3-8B cells GRIK leads ThinK-mean by up to +2.01, significant at B ≥ 512 (Table 6, Appendix I).
  • Static masks. A weight-only static mask fails (14.54). A calibration mask comes within 0.14 of GRIK, but only with calibration data. Prompt-adaptive GRIK obtains the best score without any training or data, which is its practical advantage (Section 5.4).
  • LeanK. In the same harness under LeanK's cache policy (LLaMA-3.1-8B, equal K bytes), GRIK's training-free channels with LeanK's widths score 45.95 vs 46.20 for LeanK, with a 95% CI of [−0.57, +0.08]. GRIK's channel choice thus matches a learned mask without training (Appendix M).

(W1, cont.) … reports near Full-KV RULER scores … while GRIK loses 6.9 to 8.6 points at 40% pruning.

This gap was an evaluation error on our side, now fixed. Comparing with LeanK's numbers led us to rerun Full KV in our runner. That run exposed a missing answer prefix (Shared Results, item 1). With the fix, GRIK r=0.4 is within 0.3 points of Full KV from 4K to 32K. Under LeanK's own policy at 4K, GRIK (choosing channels and widths within LeanK's per-layer budget) scores 93.95 vs 93.76 for LeanK. GRIK's RULER accuracy is therefore on par with LeanK's.

Finally, the "w/o α_h" row of Table 7 was mislabeled: it repeated the ThinK-mean run. Table 7 now contains the controlled α_h-only row above, together with N=1, query-energy and key-energy ablations.

(2/3)

W2. The cost of the recommended N=4 setting is not shown. … Table 5 does not state N, and its 4% TTFT increase at 2K input is below even the N=1 overhead in Table 13 … It is unclear whether mask selection is included. …

Q1. Which N does Table 5 use, and is mask selection included in TTFT? Please report N=4 and ThinK-mean prefill time on the same GPU, and compare quality and end-to-end latency against ThinK-mean on one workload at the same concurrency.

Table 5 did not include mask selection, so we measured the N=4 cost directly. In a paged cache shared across requests, a per-prompt mask cannot be kept consistent between cache writes and reads. The serving test therefore used one fixed mask per layer and KV head computed from the weights, and N does not apply. Mask contents do not affect speed, so the throughput numbers stand; we now state this in Section 5.3. To report the missing cost, we measured time to first token for N=4 and the ThinK-mean (ThinKPress) form on the same GPU and harness as Appendix G (GB10, LLaMA-3-8B, r=0.4, batch 1):

4K8K
Full KV1.15 s2.21 s
ThinKPress / ThinK-mean form (one-shot)†+18%+14%
GRIK N=1+15%+14%
GRIK N=4+27%+17%
GRIK N=32+127%+71%

† Second session on the same GPU, relative to that session's Full KV.

The recommended N=4 costs only about 0.15 s more prefill than the ThinK-mean form. In exchange it gains +0.77 LongBench points at identical cache size and decode speed (LLaMA-3-8B, B=1024), so the quality-vs-prefill trade-off the reviewer raises favors N=4. About half of the N=4 overhead comes from our research harness: its unpruned path already costs +13% / +10%. End-to-end latency at matched concurrency, including mask selection, is in Shared Results, item 7. With long outputs at r=0.5 the prefill cost is fully repaid.

W3. Several baselines are borrowed or use non-default settings. On LLaMA-3-8B, the Full KV and ThinK rows match Table 2 of [1] exactly … GRIK then beats Full KV in all four cells by 0.54 to 1.71 points, mostly through TREC (71.0 vs. 53.0 at B=2048, Table 14). The RULER Full KV row (Table 8) and ThinK r=0.7 row (Table 21) are identical to LeanK's table [3], without attribution. SparK [2] is run with Gaussian recovery …

Q3. Please state the source of each borrowed baseline row and rerun them in the same setup if possible. Does SparK still collapse on Qwen-2.5 with its default mean-only recovery?

The reviewer is right; we apologize for the missing attribution and reran the borrowed rows in our harness.

  • Sources. The LLaMA-3-8B Full KV and ThinK rows (Tables 2, 14) came from ThinK (Xu et al., 2025). The RULER Full KV (Table 8) and ThinK r=0.7 (Table 21) rows came from LeanK. All other rows are our runs.
  • Full KV reruns. LLaMA-3-8B Full KV is 38.41 / 40.06 / 40.95 at B=512 / 1024 / 2048 (cited: 37.23 / 38.70 / 39.59). GRIK remains above it, but by 0.06–0.35 rather than 0.54–1.71, so we no longer state that GRIK beats Full KV. TREC at B=2048 is 65.0 (cited 53.0).
  • RULER and labels. RULER Full KV is rerun (Shared Results, item 1). ThinK r=0.7 and LLaMA-3-8B B=128 are still cited and now labeled with their source.

SparK does not collapse with its default recovery; we reran it on all three backbones.

r=0.4, H2O: SparK default / GRIKB=128B=512B=1024B=2048
LLaMA-3-8B35.62 / 35.9238.30 / 38.6339.99 / 40.1240.93 / 41.30
LLaMA-3.1-8B39.49 / 39.6043.22 / 43.1744.54 / 44.6345.50 / 45.46
Qwen-2.5-7B38.88 / 38.8842.60 / 42.5344.26 / 43.9845.32 / 45.29

The two methods have equal quality: no cell's paired-bootstrap CI excludes zero. GRIK reaches this quality with a 0.60× K cache, while SparK keeps 1.00×, which is the advantage the paper now claims. We removed the collapse claim (Sections 3 and 5.2) and the "P(GRIK>SparK)=1.00" statement (Appendix I).

(3/3)

Q4. Please correct the significance statements to match Table 19. What loss is considered acceptable? An equivalence test with a stated margin and an example-level bootstrap would better support near-lossless claims.

We corrected the statements and added the equivalence test with a stated margin. The reviewer is right that our text contradicted Appendix I. Against Full KV on LLaMA-3.1-8B at r=0.4, GRIK's loss is small but significant at B=128, 1024 and 2048 (−0.3 points each). We removed "statistically indistinguishable" (Section 5.2, Appendix I).

We define the acceptable loss as a non-inferiority margin of δ = 1.0 LongBench point (about 2.5% relative): the lower end of the paired-bootstrap 95% interval must lie above −δ. Under the task-level test of Appendix I, non-inferiority holds in 7 of 8 Full-KV cells: all four LLaMA-3.1 budgets (lowest bound −0.6) and Qwen-2.5 at B ≥ 512. The exception is Qwen-2.5 at B=128 ([−1.6, +0.2]), which we state explicitly.

As requested, we also ran an example-level test (stratified resampling of examples within each task, 10,000 iterations) on the new LLaMA-3-8B runs, for which we have per-example predictions in one harness. The original LLaMA-3.1 / Qwen-2.5 per-example outputs were not retained.

Pair (LLaMA-3-8B, H2O)Δ95% CI (example level)non-inferior at δ=1
GRIK N=4 vs all channels, B=1024+0.04[−0.29, +0.37]yes
GRIK B=632 vs all channels B=512 (equal memory)+0.50[+0.13, +0.87]yes
GRIK B=1271 vs all channels B=1024 (equal memory)+0.53[+0.16, +0.91]yes

These intervals support both claims at the example level. At equal token budget, pruning 40% of K channels is non-inferior to keeping all channels. At equal memory, GRIK is significantly better (Appendix M).

Minor: Table 20 lists ThinK (r=0.6, B=1024) as 38.57, while Tables 2 and 14 and [1] give 37.58. App. I gives a gap range of [-0.27, -0.19], but Table 2 gives -0.21 to -0.27. App. E.3 cites thunlp/ThinK, while the official code is SalesforceAIResearch/ThinK. The N=1 row in Table 7 is empty.

All four are fixed in the revision.

  • Table 20, ThinK r=0.6, B=1024: corrected to 37.58.
  • Appendix I gap range: corrected to −0.27 to −0.21, matching Table 2.
  • Appendix E.3: the official ThinK repository is SalesforceAIResearch/ThinK.
  • Table 7, N=1 row: 39.85 (B=1024, measured against a same-GPU N=4 reference of 40.01).

Dear Reviewer KiiC,

Thank you again for the detailed review.

As the discussion period is coming to an end, we would like to ask whether our responses have addressed your concerns, in particular the comparison with kvpress ThinKPress, static masks and LeanK (Q2; GRIK +0.77 over ThinKPress at identical bytes and on par with LeanK's learned channels), the measured N=4 prefill cost (Q1), the borrowed baselines and SparK rerun with its default recovery in our harness (Q3), and the corrected significance statements with a stated non-inferiority margin (Q4). If so, we would appreciate it if you could consider updating your assessment.

Otherwise, we would be happy to clarify any remaining points.

Best regards, Authors

Reviewer_RWcSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

K-cache pruning along the channel dimension (hidden size) for grouped-query attention (GQA) LLMs is addressed. The proposed method, Grouped Relevance-Integrated Key Pruning (GRIK), constructs a prompt-adaptive mask satisfying conditions C1 and C2 by construction, enabling effective KV cache reduction under GQA. Extensive experiments demonstrate that GRIK achieves competitive performance with significant cache size reduction.

Critical strengths
  • Provide detailed experiments.
  • Writing is relatively clear.
Critical weaknesses
  • Although the experiment is rich, the proposed method is a minor update from previous methods.
  • Experiments:
    • H2O eviction is old (three years ago; and performance not good enough), SnapKV (two years ago) is relatively old. There are lots of newer evinced based methods, please include them based on your settings.
    • Limited comparison, only compare with ThinK and SparK.
  • Minor advice:
    • The term 'Full KV' is confusing throughout the manuscript. Some time it means the kv after token evince, while other it means the total kv at the same context length.
    • FP8, int4, bf16 ...; please keep the same style of naming, all upper cases or all lower cases.
Questions for authors
  • The Llama-3-8B-Instruct is trained with max context of 8K (as Table 11). The LongBench-v1 has inputs range from 4K to 32K (Apendix E.2). Table 14 is about LongBench on Llama-3-8B-instruct. What is the largest context length here? Although with kv evince, the original context length still important.
2 comments

We sincerely thank Reviewer RWcS for the thoughtful comments and constructive feedback, and for finding the experiments detailed and the writing clear. Below, we provide detailed point-by-point responses addressing each of your concerns.

W1. Although the experiment is rich, the proposed method is a minor update from previous methods.

We narrowed the claim to what is new, and measured that part separately. We agree that GRIK's ingredients are individually known. The contribution lies in the cache layout and in the channel choice within it. Under GQA, the released ThinK and SparK implementations do not realize the memory they prune: they store 3.20× and 1.00× of the baseline total KV. One mask per KV head, constant over tokens (C1/C2), is the layout that does. To show that the channel choice inside this layout also matters, we compared masks at identical bytes (LLaMA-3-8B, r=0.4, B=1024):

Mask (same layout and bytes)LongBench-16
Random shared mask14.95
Weight-norm only14.54
kvpress ThinKPress form (closest prior scorer)39.33
GRIK40.10

The layout alone loses about 25 points, and GRIK improves on the closest prior scorer by +0.77 at the same memory. GRIK is therefore a training-free method that both realizes the GQA memory saving and selects channels better within it, not a relabeling of prior work. We revised Sections 1 and 6 accordingly (highlighted in red).

W2. H2O eviction is old (three years ago; and performance not good enough), SnapKV (two years ago) is relatively old. There are lots of newer evinced based methods, please include them based on your settings.

We added PyramidKV, a more recent policy with layer-wise budgets. Every layer still keeps a rectangular cache, so GRIK's layout is unchanged. LLaMA-3-8B, r=0.4:

Evictionalone (all channels), B=1024+ GRIK, B=1024 (0.80× total KV)
H2O40.0640.10
SnapKV40.9741.16
PyramidKV41.0340.76

GRIK composes with every policy while cutting total KV to 0.80×. At equal memory it beats the policy alone: PyramidKV alone at B=832 (= 0.8125 × 1024) scores 40.63 vs 40.76 with GRIK, and for H2O the gain is +0.43 to +0.53 (Shared Results, item 3). GRIK's channel axis is therefore complementary to newer token-eviction policies (Section 5.3, Appendix M).

W3. Limited comparison, only compare with ThinK and SparK.

We added four channel-mask baselines at identical storage and reran SparK with its default setting. The four baselines are the kvpress ThinKPress form, LeanK's released learned mask in our harness, static masks (weight norm; calibration-averaged), and a random shared mask.

  • GRIK improves on ThinKPress by +0.77.
  • GRIK's channels match LeanK's learned channels without training (45.95 vs 46.20, n.s.).
  • GRIK outperforms every data-free static mask.
  • It matches SparK's quality at 0.60× instead of 1.00× K memory.

See Shared Results, items 2, 5 and 6, and Sections 5.2 and 5.4.

Minor. The term 'Full KV' is confusing throughout the manuscript. Some time it means the kv after token evince, while other it means the total kv at the same context length. FP8, int4, bf16 ...; please keep the same style of naming, all upper cases or all lower cases.

Both are fixed. In the budgeted tables, "Full KV" means all channels kept under the same token eviction. Section 5.1 now defines it explicitly, and the text writes "H2O (all channels)" wherever it must be distinguished from no compression. Precision names are unified to BF16 / FP8 / INT4 throughout the manuscript.

Q. The Llama-3-8B-Instruct is trained with max context of 8K (as Table 11). The LongBench-v1 has inputs range from 4K to 32K (Apendix E.2). Table 14 is about LongBench on Llama-3-8B-instruct. What is the largest context length here? …

The largest context is at most 7,950 + 512 tokens, inside the 8K window. Following the LongBench protocol used by ThinK and SnapKV:

  • Longer prompts are truncated in the middle, keeping the first and last halves, to 7,950 tokens (LLaMA-3/3.1) or 31,500 (Qwen-2.5).
  • Generation follows LongBench's per-task limits (≤512 tokens).
  • The budget B applies within this context.

We added this description to Appendix E.2 (highlighted in red).

Dear Reviewer RWcS,

Thank you again for your review.

As the discussion period is coming to an end, we would like to ask whether our responses have addressed your concerns, in particular the novelty of GRIK (the identical-bytes mask comparison, where GRIK improves on the closest prior scorer by +0.77 while a random or weight-only mask loses about 25 points), the newer eviction policy you requested (PyramidKV, where GRIK + PyramidKV outperforms PyramidKV alone at equal memory), and the additional channel-mask baselines (ThinKPress, LeanK and static masks). If so, we would appreciate it if you could consider updating your assessment.

Otherwise, we would be happy to clarify any remaining points.

Best regards, Authors

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We would like to thank the reviewers for their constructive and thorough evaluations.

We are encouraged that the reviewers highlighted the separation of channel sparsity from realized memory under grouped-query attention (KiiC), the simplicity of the structural conditions C1/C2, with byte accounting that matches measurement (KiiC), the relevance of the formulation and its practical storage benefit (ewjC), the training-free design that keeps a dense cache and composes with eviction and quantization (KiiC), and the detailed and extensive evaluation with useful controls (RWcS, ewjC, KiiC).

Below we summarize the major revisions, with detailed point-by-point responses provided for each reviewer. Results shared across reviewers are collected in the two comments that follow. All revisions are highlighted in red in the manuscript.

Summary of Key Revisions

  • Corrected RULER evaluation. RULER now runs both methods in one runner with the official answer prefix. GRIK at r=0.4 is within 0.3 points of Full KV from 4K to 32K (Section 5.5, Table 8, Appendix K).
  • SparK with its default recovery. All r=0.4 cells are rerun on three backbones. SparK matches GRIK's quality, and GRIK stores K at 0.60× instead of 1.00× (Sections 3 and 5.2, Table 2, Appendices H and I).
  • Baselines rerun in our harness. LLaMA-3-8B Full KV is rerun, the equal-memory comparison is recomputed (+0.43 to +0.53, example-level CI excluding zero), and every remaining borrowed number is labeled with its source (Tables 2 and 3, Section 5.2).
  • New channel-mask baselines at identical bytes. These are the kvpress ThinKPress form, LeanK's learned mask in the same harness, static masks (weight norm; calibration-averaged), and a random shared mask (Sections 5.4 and 6).
  • One-factor ablations. α_h, the static prior, query and key energy, and refinement depth are each ablated, and the claims are narrowed to what the ablations support (Table 7, Section 5.4).
  • Additional eviction policy. PyramidKV with and without GRIK, including an equal-memory comparison (Section 5.3).
  • GSM8K on the full test set. All 1,319 questions are evaluated, with exact McNemar tests (Section 5.5).
  • Serving and end-to-end cost. We built a shared-component engine and measured per-user decode at equal batch, throughput at equal memory, and request latency including mask selection; Table 5 is clarified (Section 5.3, Appendix G).
  • Statistical statements. These are corrected to match the bootstrap results, with a stated non-inferiority margin (Section 5.2, Appendix I).
  • Clarity. Full KV terminology, the truncation protocol, BF16/FP8/INT4 naming, and figure/table references are made consistent (Section 5.1, Appendix E).
AuthorsOpenReview ↗

Shared Results Referenced in the Individual Responses (1/2)

We collect here the results requested by more than one reviewer.

1. RULER gap (KiiC W1, ewjC W3/Q4). To find the source of the 6.9–8.6-point RULER gap, we ran Full KV through our own runner. Variable tracking scored 0 for Full KV too, which exposed an error: our runner omitted RULER's task-specific answer_prefix, while the submitted Full KV row came from the official harness. We fixed the runner and reran both methods in it (LLaMA-3.1-8B, r=0.4, no eviction):

RULER-134K8K16K32K
Full KV94.7494.0392.4186.16
GRIK r=0.494.8594.0192.1386.08

The gap was an artifact of our evaluation, not of GRIK. In one runner, GRIK stays within 0.3 points of Full KV at every length, including the aggregation tasks. The largest per-task gap is common-word extraction at 16K (70.8 vs 73.95). At 32K this task is near floor for Full KV itself (7.2), a limit of the base model. Higher ratios do hurt aggregation (r=0.7: 62.20 / 58.53 at 4K / 8K), so we recommend r ≤ 0.4 and state this as a limitation. The fixed runner is in the anonymous repository.

2. SparK with its default recovery (KiiC W3/Q3). To test whether SparK's collapse came from our choice of Gaussian recovery, we reran SparK with its released default (mean) recovery in every r=0.4 cell on all three backbones (submitted value in parentheses):

H2O, r=0.4, B=1024SparK, defaultGRIKFull KV
LLaMA-3-8B39.99 (34.68)40.1240.06
LLaMA-3.1-8B44.54 (39.94)44.6344.88
Qwen-2.5-7B44.26 (11.49)43.9843.79

With its default recovery, SparK matches GRIK's quality: GRIK − SparK ranges from −0.28 to +0.37 across all 12 cells, and no paired-bootstrap CI excludes zero. SparK, however, stores a full-width 1.00× K cache, while GRIK stores 0.60×. GRIK therefore reaches the same accuracy with 40% less K memory, which is the advantage the paper now claims. Some notes on the runs:

  • All budgets are in Table 2.
  • On Qwen-2.5, prefill attention ran through SDPA to fit memory, with cache selection unchanged (GRIK 44.01 vs 43.98 eager).
  • SparK at r=0.6 was not rerun and is labeled as such.

3. Equal total KV memory (KiiC W3, ewjC S2). Rerunning H2O with all channels in our harness raised it by 0.9–1.4 points over the earlier run, so we recomputed the equal-memory comparison. GRIK gets budget B/0.8125, where 0.8125 is its stored total-KV ratio with K padded to 80 channels (LLaMA-3-8B, H2O, same harness):

H2O, all channelsGRIK, same bytesΔ (example-level 95% CI)
B=512: 38.41B=632: 38.91+0.50 [+0.13, +0.87]
B=1024: 40.06B=1271: 40.59+0.53 [+0.16, +0.91]
B=2048: 40.95B=2520: 41.38+0.43

At a fixed memory budget, spending the saved channel bytes on more tokens is consistently better than keeping all channels. The gain is +0.43 to +0.53 rather than the +1.55 to +1.62 we reported, and it is significant at the example level. The same holds with PyramidKV: GRIK + PyramidKV at B=1024 scores 40.76, vs 40.63 for PyramidKV alone at the equal-memory B=832.

AuthorsOpenReview ↗

Shared Results Referenced in the Individual Responses (2/2)

4. GSM8K drop (ewjC W3). To test whether the 77.8 → 73.5 drop on the 234-question subset holds up, we evaluated all 1,319 GSM8K test questions (LLaMA-3.1-8B, 8-shot CoT, no eviction) with exact McNemar tests:

Full KVGRIK N=4GRIK N=32
Accuracy76.6575.7475.97
Exact McNemar vs Full KV—p=0.38p=0.51

On the full test set the difference is within 1 point and not statistically significant. Multi-step reasoning is therefore preserved, and the subset overstated the gap. Full KV run through our patched harness scores 76.72, so the harness itself costs nothing.

5. What the score adds beyond the layout (KiiC W1/Q2, ewjC W2/Q1, RWcS W3). To separate the contribution of the channel choice from that of the layout, we compared masks that all store the identical H_kv×T×k cache (LLaMA-3-8B, r=0.4, B=1024, H2O):

Mask (same layout and bytes)LongBench-16
Random shared mask14.95
Static, ‖W_k‖ only14.54
kvpress ThinKPress form (group-mean q-energy × k-energy, one-shot)39.33
Static, calibration-averaged GRIK score39.81
GRIK without α_h only (prior and N=4 kept)39.76
GRIK, N=440.10

The layout alone does not preserve quality: a random or weight-only mask loses about 25 points. GRIK's score improves on the closest existing scorer (ThinKPress) by +0.77, and each added component contributes (α_h +0.34). This shows that the channel choice inside the realizable layout is a genuine part of the contribution. On a second GPU (same-GPU reference 40.01), removing the query energy costs 0.81, while removing the key energy does not hurt (+0.25). We therefore no longer claim that the key term is needed.

6. LeanK in the same harness (KiiC W1/Q2, ewjC W1/Q1). To compare directly with a learned static mask, we ran LeanK's released LLaMA-3.1-8B mask under its own cache policy and gave GRIK the same K bytes:

No eviction, LeanK's cache policyTrainingLongBench-16
Full KV—46.07
LeanK (released mask)yes46.20
GRIK channels + LeanK's per-head widthsnone for channels45.95 (CI [−0.57, +0.08])
GRIK, own widths (equal per-layer budget)none45.64

GRIK's training-free channel choice is not significantly different from LeanK's learned one. The remaining 0.56-point gap of fully training-free GRIK comes from LeanK's learned per-head widths, not from the channel choice. On RULER-13 at 4K, LeanK scores 93.76, and GRIK scores 93.95 when it chooses channels and widths within LeanK's per-layer budget.

7. Serving at matched conditions (ewjC W4/Q3, KiiC W2/Q1). To compare at matched concurrency and include every cost, we built one engine in which Full KV and GRIK share the paged cache, attention kernel and model code. Masks are computed from each prompt inside the engine. Results on LLaMA-3.1-8B (GB10):

  • Equal batch, r=0.5: per-user decode is +12% (4K×32), +18% (16K×16) and +18% (32K×8).
  • Equal KV memory: GRIK holds 1.34× more sequences, with +27–32% aggregate throughput and −3% to −5% per-user speed.
  • End to end: including prefill, mask selection and packing, 16-token requests are 7–12% slower, while 512-token requests at r=0.5 are 1–4% faster.

When KV reads dominate, GRIK is faster per user at the same batch, and under a memory limit it serves more users. The one-time prefill cost matters only for short outputs, which we state as a limitation. At r=0.4 (K padded to 80 channels) decode is 1–3% slower on this kernel, so r=0.5 is the latency setting.

AuthorsOpenReview ↗

Summary Comments to Area Chair

Dear Area Chair,

We sincerely thank all reviewers for their time and careful evaluations. We are encouraged that the reviewers found the separation of channel sparsity from realized memory and the structural conditions C1/C2 valuable (KiiC), the formulation relevant with a practical storage benefit (ewjC), and the evaluation detailed and well-controlled (RWcS, ewjC, KiiC). During the discussion, we addressed every raised concern with new experiments, and we corrected the manuscript wherever the new evidence changed a number. After reading our responses, Reviewer ewjC raised their score. Reviewers RWcS and KiiC have not yet had the opportunity to respond to our rebuttal, so we hope the summary below helps in assessing whether their concerns have been addressed.

Novelty of GRIK

  • A problem prior channel pruning did not solve. Under grouped-query attention, the released ThinK and SparK implementations do not realize the memory they prune: they store 3.20× and 1.00× of the baseline total KV. GRIK identifies why, and states two conditions (C1/C2: one mask per KV head, constant over tokens) under which channel pruning actually shrinks a dense cache.
  • A better channel choice within that layout. At identical bytes, a random or weight-only mask loses about 25 points, while GRIK's training-free score improves on the closest prior scorer (kvpress ThinKPress) by +0.77 and matches LeanK's learned channels without training (45.95 vs 46.20, n.s.).
  • A new axis that composes with existing methods. GRIK prunes channels, not tokens, so it stacks on H2O, SnapKV, PyramidKV and quantization. At equal total memory it outperforms H2O alone by +0.43 to +0.53 (example-level 95% CI excludes zero).

How each major concern was resolved

  1. RWcS (novelty, newer eviction, limited baselines). We added the identical-bytes mask comparison above, PyramidKV (GRIK + PyramidKV beats PyramidKV alone at equal memory), and four channel-mask baselines (Sections 5.3, 5.4, 6, Appendix M).
  2. KiiC (ThinKPress and LeanK, N=4 cost, borrowed baselines, significance). We cite ThinKPress as the uniform special case of our score, measured N=4 prefill (+27% / +17% at 4K / 8K), reran all borrowed rows, and reran SparK with its default recovery: it matches GRIK's quality but keeps a 1.00× K cache vs GRIK's 0.60×. Significance statements now follow the bootstrap with a stated non-inferiority margin (Sections 5.2–5.4, Appendices G and I).
  3. ewjC (component isolation, RULER and GSM8K, serving cost). We added one-factor ablations (Table 7). The RULER gap came from a missing answer prefix in our runner; fixed, GRIK is within 0.3 points of Full KV from 4K to 32K. On all 1,319 GSM8K questions the difference is not significant (p = 0.38 / 0.51). A shared-component engine reports end-to-end cost including mask selection (Sections 5.3 and 5.5).
RULER-13, LLaMA-3.1-8B, r=0.4 (same runner)4K8K16K32K
Full KV94.7494.0392.4186.16
GRIK94.8594.0192.1386.08

Why we believe GRIK merits acceptance Beyond accuracy, GRIK delivers memory savings that a serving system can actually use. Because its cache stays dense and rectangular, it fits an existing paged-attention engine with no training and no custom sparse format. In a shared engine at r=0.5, it holds 1.34× more sequences at equal KV memory, with +27–32% aggregate throughput, and per-user decode is 12–18% faster at equal batch. We also took the reviewers' critiques seriously: where they found weak points, we reran the baselines in one harness and report the corrected, more conservative numbers. We believe the revised paper is a sound and practical contribution to KV-cache compression for GQA models.

We respectfully ask you to consider these revisions in your final assessment. Thank you very much for your careful evaluation.

Sincerely, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.