acceptodds
Under review as a conference paper at ICLR 2027

Rotation-Aligned Key Channel Pruning for Efficient Vision-Language Model Inference

Abstract

Vision-Language Models suffer severe KV cache pressure at inference, as a single image often encodes into thousands of tokens. To reduce this cost, most prior works exploit token sparsity through token pruning, but permanently discarding visual content substantially degrades fine-grained perception. This motivates a complementary axis, feature sparsity, which compresses the channel dimension to preserve more visual tokens at the same KV cache budget. Existing Key channel pruning methods, however, face a structural trade-off: token-wise channel pruning is accurate but unstructured and slow, while head-wise pruning is hardware-friendly but degrades at high compression. We attribute this degradation to informative Key directions that change across tokens and inputs, so no fixed channel subset serves them all. We propose RotateK, a training-free method that builds a query-weighted Key basis for each input at prefill, allowing a head-wise mask to retain attention-relevant information. To make this practical, we implement a graph-capturable top-k eigensolver, a fused sparse-channel decode kernel, and a paged KV layout in vLLM. Across six VLMs, up to 32B parameters, and four token pruning methods, RotateK consistently improves over token pruning alone at matched KV budgets and is comparable in accuracy to token-wise channel pruning, while increasing KV capacity and generation throughput.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficient inference, KV cache compression, vision-language models, channel pruning, token pruning, PCA rotation, GPU kernel

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_Ua6LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a training-free method that rotates visual key representations into an input-adaptive basis before pruning their channels. RotateK integrates with existing vision token pruning methods, allowing vision language models to retain more visual tokens at the same visual KV-cache budget. The custom kernels accelerate decoding relative to full-channel baselines.

Critical strengths
  1. The authors provide an analysis of the basis mismatch, which is particularly relevant for fine-grained visual understanding tasks where aggressive token pruning can lead to substantial performance degradation.
  2. The proposed method provides an interesting solution to the limitations of structured channel pruning. The query weighted PCA formulation is well motivated by the objective of minimizing attention-score perturbations.
  3. Beyond algorithmic contributions, the paper provides a efficient implementation compatible with modern frameworks. The reported improvements in decoding throughput and KV cache capacity demonstrate practical benefits. Although RotateK adds a layer of additional latency to the prefill stage, compared to the vision token pruning methods, I believe the improvements in benchmark results make the trade off a worthy one. I encourage the authors to highlight the latency measurements in the main tables (which are currently in appendix tables 9, 10).
  4. The paper reports extensive experiments across a diverse combination of settings. RotateK consistently improves performance over token-only pruning under matched KV cache budgets and achieves accuracy comparable to token-wise channel pruning. Furthermore The paper includes extensive ablation studies, providing insights into the method.
Critical weaknesses
  1. Sec. 2 motivates RotateK through the observation that informative K channels vary across tokens. However, Fig. 2 visualizes pruning decisions rather than directly measuring channel importance, while the all-token intersection metric in App. E.4 may exaggerate disagreement even when individual masks largely overlap. Although Fig. 6, Tables 4 and 6 provide supporting evidence for rotation and compact Key subspaces, the connection between mask disagreement and attention-relevant information loss remains indirect. This limits how strongly Sec 2 supports basis mismatch as an explanation for the degradation of head-wise pruning.
  2. RotateK introduces an additional degree of freedom for KV-cache compression. However, the paper provides limited guidance on how to allocate a fixed memory budget between these dimensions. For a given VLM, token-pruning method, and task, how should users choose the retention ratios? While the paper evaluates several configurations, it is unclear how these configurations were selected and whether they are near-optimal. Moreover, how sensitive is downstream performance to the budget allocation, and can a configuration selected for one model or workload transfer effectively to another? The authors should provide a systematic analysis of this trade-off, together with practical guidelines or an inexpensive selection heuristic.
  3. Sec. 3.2 compares configurations using the visual KV budget (ρ=ρt(1+ρc)/2\rho=\rho_t(1+\rho_c)/2). This does not include intermediate matrices, such as the cached rotation matrices, mean-correction vectors, or uncompressed prompt and generated-text KV states. Although the paper demonstrates practical memory savings, it remains unclear whether budget matching consistently accounts for these additional states and their storage precision. Their relative cost may become substantial when few visual tokens are retained. The authors should provide complete memory accounting across representative input lengths by distinguishing visual KV payload from total allocated memory.
  4. Sec. 3.2 states that matched-budget configurations differ only in prefill, but Table 10 shows that joint pruning remains approximately 11–16% slower e2e than token-only pruning at batches 8–32. The authors should qualify this statement and evaluate different generation lengths, since decode parity does not establish equality at the request level.
  5. Minor: Table 3 shows that using full eigendecomposition leads to better results, with differences up to 1.14. The authors should qualify the claims on “purely latency” (App B.1) and ridge’s “no measurable effect” (App. F), and report uncertainty across different random initializations.
Questions for authors

See weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Ua6L (1)

We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.


Strength 3 Beyond algorithmic contributions, the paper provides a efficient implementation compatible with modern frameworks. The reported improvements in decoding throughput and KV cache capacity demonstrate practical benefits. Although RotateK adds a layer of additional latency to the prefill stage, compared to the vision token pruning methods, I believe the improvements in benchmark results make the trade off a worthy one. I encourage the authors to highlight the latency measurements in the main tables (which are currently in appendix tables 9, 10).

TL;DR. Agreed; both latency tables are now in the main text as Table 3 (Sec. 4.3), with the matched-budget accuracy shown alongside. Revision: Sec. 4.3 and Table 3(a,b); full results in Tables 21–22 (App. D.6).

Answer. We thank the reviewer for this assessment and agree that the latency results belong in the main text. In the revision, we have moved both latency results into Sec. 4.3 as Table 3: (a) basis construction cost and (b) end-to-end latency at matched KV budgets, with the matched-budget accuracy alongside. Full results are in Tables 21–22 (App. D.6).


W1 Sec. 2 motivates RotateK through the observation that informative K channels vary across tokens. However, Fig. 2 visualizes pruning decisions rather than directly measuring channel importance, while the all-token intersection metric in App. E.4 may exaggerate disagreement even when individual masks largely overlap. Although Fig. 6, Tables 4 and 6 provide supporting evidence for rotation and compact Key subspaces, the connection between mask disagreement and attention-relevant information loss remains indirect. This limits how strongly Sec 2 supports basis mismatch as an explanation for the degradation of head-wise pruning.

TL;DR. Pairwise overlap (only 17–19 of 32 channels shared) and the attention error of head-wise vs. per-token masks support basis mismatch beyond mask disagreement. Revision: Sec. 1 (first contribution, L093–097); Sec. 2 (L146–149, L152–157); Tables 30–31 (App. E.7–E.8).

Answer. We agree with both points: the all-token intersection of App. E.7 may overstate the disagreement, and Fig. 2 and App. E.7 show that selections differ across tokens, but do not explicitly show that this causes attention-relevant loss. We address each below.

1) Pairwise overlap instead of the all-token intersection. When each visual token keeps its own top-32 Key channels (as in SparK), only 3 (LLaVA-NeXT) and 5 (Qwen2.5-VL) channels are shared by all tokens, which overstates disagreement. Over 512 random token pairs per layer and head, two tokens still share only 17 and 19 of their 32 channels on average: the disagreement is smaller, but still substantial.

2) Attention error of head-wise and per-token masks. To test whether this disagreement causes attention-relevant loss, we restrict the visual Keys to 32 of 128 channels per KV head in three ways: (i) one mask per head (ThinK), (ii) one mask per token (SparK), and (iii) one subspace per head in RotateK's rotated basis. All three restore the mean of the discarded part, so (i) vs. (ii) isolates the selection and (i) vs. (iii) isolates the basis.

Table R-1. Error against uncompressed visual Keys (ref) on decode queries (FastV and VisionZip × TextVQA, ChartQA, InfoVQA × 50 samples). Attention-weight error: 12∥p−pref∥1\frac{1}{2}\lVert p-p_{ref}\rVert_1, where pp is the attention distribution of a decode query over all cached tokens; output error: ∥o−oref∥/∥oref∥\lVert o-o_{ref}\rVert / \lVert o_{ref}\rVert, where oo is the attention output. The ordering (i) > (ii) > (iii) holds in all 12 settings for both errors.

Visual Keys (32 of 128)SelectionBasisLLaVA-NeXT-8B weight / outputQwen2.5-VL-7B weight / output
(i) Head-wise maskper headchannel0.063 / 0.1870.066 / 0.143
(ii) Per-token masksper tokenchannel0.053 / 0.1520.052 / 0.107
(iii) Shared subspace (RotateK)per headrotated0.043 / 0.1260.045 / 0.098

A head-wise mask raises both errors over per-token masks, so the disagreement in 1) does distort attention. A shared subspace in the rotated basis falls even below per-token masks: the informative directions are shared across tokens but not aligned with the channel axes. This is the basis mismatch that degrades head-wise pruning.

AuthorsOpenReview ↗

Response to Reviewer Ua6L (2)

W2 RotateK introduces an additional degree of freedom for KV-cache compression. However, the paper provides limited guidance on how to allocate a fixed memory budget between these dimensions. For a given VLM, token-pruning method, and task, how should users choose the retention ratios? While the paper evaluates several configurations, it is unclear how these configurations were selected and whether they are near-optimal. Moreover, how sensitive is downstream performance to the budget allocation, and can a configuration selected for one model or workload transfer effectively to another? The authors should provide a systematic analysis of this trade-off, together with practical guidelines or an inexpensive selection heuristic.

TL;DR. Keep 25% of the Key channels (best on average, or within 1.1 points of the best) and spend the saved memory on more visual tokens. Revision: Tables 13–14 and guideline (App. B.4); Figure 5 caption.

Answer. Our guideline is simple: keep 25% of the Key channels, and choose the token ratio as for token-only pruning, then raise it to use the KV memory that the channel pruning saves. The allocation is simpler than the two continuous ratios suggest, as we show below.

(1) The channel ratio has only a few practical values. Decode kernels use power-of-two tile sizes, so the retained Key width is efficient only at 16, 32 or 64 of 128 channels (12.5%, 25% or 50%); other widths, such as 48, are padded to the next power of two and are slower in practice. The allocation therefore reduces to choosing one of three channel ratios, and the token ratio then follows from the budget.

(2) Sensitivity, near-optimality and transfer. Table R-2(a) varies the allocation at a fixed visual KV budget (that of Table 1), and Table R-2(b) breaks the 25% default down by task. We include 37.5% for completeness, although it is slower in practice.

Table R-2(a). Mean accuracy (%) over TextVQA, InfoVQA, ChartQA, DocVQA and VizWiz at a fixed visual KV budget. Columns: retained Key channels. Token ratios: FastV 0.25 / 0.33 / 0.36 / 0.40 / 0.44, VisionZip 0.28 / 0.37 / 0.41 / 0.45 / 0.50.

Setting100% (token-only)50%37.5%25%12.5%
LLaVA-NeXT + FastV (0.25×)51.2453.6554.1053.4448.16
LLaVA-NeXT + VisionZip (0.28×)53.4455.5754.9254.4347.98
Qwen2.5-VL + FastV (0.25×)66.7771.4072.6273.9568.82
Qwen2.5-VL + VisionZip (0.28×)68.9172.8473.4674.3970.36

Table R-2(b). Accuracy (%) with 25% of the Key channels per task; in parentheses, the difference to the best of the five allocations for that task, and that allocation when it is not 25%.

SettingTVQAIVQACQADVQAVizWizMean
LLaVA-NeXT + FastV65.6 (−1.3, 50%)29.4 (0.0)58.2 (−1.4, 37.5%)62.9 (−1.4, 37.5%)51.1 (0.0)53.4 (−0.7, 37.5%)
LLaVA-NeXT + VisionZip65.3 (−2.9, 50%)30.2 (0.0)59.0 (−0.6, 50%)66.3 (−2.4, 50%)51.3 (−0.2, 37.5%)54.4 (−1.1, 50%)
Qwen2.5-VL + FastV85.2 (−0.6, 37.5%)66.7 (0.0)70.8 (−0.2, 12.5%)86.5 (0.0)60.5 (0.0)73.9 (0.0)
Qwen2.5-VL + VisionZip81.4 (−0.2, 50%)69.2 (0.0)72.0 (−1.0, 12.5%)90.3 (0.0)59.0 (0.0)74.4 (0.0)

Every allocation between 50% and 25% of the Key channels outperforms token-only pruning on average at the same budget, and token-only is never the best allocation on any task. A single default of 25% transfers well across models, pruners and tasks: it is the best allocation on average on Qwen2.5-VL, and on LLaVA-NeXT it is within 1.1 points of the best. Pruning further to 12.5% is too aggressive and should be avoided. Since we used 25% throughout the paper without tuning, the reported gains are conservative on LLaVA-NeXT.

AuthorsOpenReview ↗

Response to Reviewer Ua6L (3)

W3 Sec. 3.2 compares configurations using the visual KV budget. This does not include intermediate matrices, such as the cached rotation matrices, mean-correction vectors, or uncompressed prompt and generated-text KV states. Although the paper demonstrates practical memory savings, it remains unclear whether budget matching consistently accounts for these additional states and their storage precision. Their relative cost may become substantial when few visual tokens are retained. The authors should provide complete memory accounting across representative input lengths by distinguishing visual KV payload from total allocated memory.

TL;DR. RkR_k and δμ\delta\mu add a fixed ≈2.1 MiB per image (0.4–6.6% of the total stored memory), which is now reported explicitly. Revision: Sec. 3.2; Table 26 and the per-benchmark overhead (App. E.4).

Answer. We thank the reviewer and agree that the visual KV budget omits these states, so we measured every tensor held after generation.

Rotation and mean correction. Per layer and KV head, RotateK stores RR (128 × 32) and δμ\delta\mu (128) in bf16, i.e., 8.25 KiB, the size of 16.5 full-KV visual tokens. This totals 2.06 MiB on LLaVA-NeXT-8B and 0.90 MiB on Qwen2.5-VL-7B per image, independent of sequence and generation length.

Complete accounting. Table R-3 compares token-only pruning and RotateK at the same visual KV budget (30-token prompt and question, 128 generated tokens). Prompt and text KV are uncompressed and identical in both, so RR and δμ\delta\mu are RotateK's only extra state.

Table R-3. Memory (MiB) stored for one request on LLaVA-NeXT-8B at a matched visual KV budget (token-only / RotateK). N: visual tokens retained by token-only pruning; RotateK retains 1.6N. Qwen2.5-VL-7B gives the same percentages.

NVisual KVPrompt + text KVRR + δμ\delta\muTotal stored
648.0 / 8.023.4 / 23.4– / 2.131.4 / 33.4 (+6.6%)
25632.0 / 32.023.4 / 23.4– / 2.155.4 / 57.5 (+3.8%)
1024128.0 / 128.023.4 / 23.4– / 2.1151.4 / 153.4 (+1.4%)
4096512.0 / 512.023.4 / 23.4– / 2.1535.4 / 537.5 (+0.4%)

Their relative cost grows as fewer visual tokens are kept but stays small: +6.6% of the total at N = 64, +1.4% at 1024, and +0.4% at 4096. On our benchmarks, it is 1.7–2.5% on LLaVA-NeXT and 2.4–7.3% on Qwen2.5-VL, the upper end being ChartQA with its small images.

Reviewer_ryZbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method for compressing keys by using a rotation matrix to identify their most important channels.

Critical strengths

The method proposes a per-token channel compression scheme designed to facilitate fused kernel implementation.

Critical weaknesses
  1. The proposed method aims to address the limitations of SPARK. However, Tables 1 and 2 do not include comparisons with SPARK, while Figure 6 shows that SPARK performs better than the proposed method.

  2. The method follows an approach similar to SPARK by selecting important channels for each token. Although it introduces a rotation matrix for this purpose, the technical contribution is incremental. Moreover, compressing every token to the same number of channels removes the flexibility of adaptive channel allocation, and the resulting gains rely on kernel customization.

Questions for authors
  1. What is the computational cost of dynamically computing RR?

  2. How representative are the recent queries used by the method of future queries?

  3. Why does applying ThinK to LLaMA3-LLaVA-NeXT-8B lead to a performance drop?

  4. What is the overhead of projecting queries and keys using RR at every decoding step?

  5. Since RR is also cached, is its memory footprint included in the reported KV cache size?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ryZb (1)

We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.


W2 The method follows an approach similar to SPARK by selecting important channels for each token. Although it introduces a rotation matrix for this purpose, the technical contribution is incremental. Moreover, compressing every token to the same number of channels removes the flexibility of adaptive channel allocation, and the resulting gains rely on kernel customization.

TL;DR. RotateK prunes head-wise, not token-wise; the contribution is showing that the head-/token-wise gap stems from the channel basis, and the fixed per-token width is a deliberate choice for paged serving.

Answer. First of all, we would like to emphasize that RotateK does not select channels for each token. It builds one basis per (input, head), and all tokens of that head share the same k retained directions, so the pruning is head-wise and structured, not token-wise (L091, L099–100, L209–210).

Hence, the contribution is not adding a rotation to SparK. Instead, we show that the accuracy gap between head-wise and token-wise pruning comes mainly from the channel basis rather than from mask granularity, and we build a method that exploits this finding. For example, ThinK and RotateK apply the same kind of head-wise mask and differ only in the basis, yet ThinK loses up to 9.7 points at 75% sparsity while RotateK stays within 0.7 points of token-wise SparK (L418–422).

On adaptive channel allocation: keeping the same number of channels per token is a deliberate design choice. A fixed per-token width lets the compressed cache fit paged KV management without changes to the block manager or scheduler (L502–504). Adaptive per-token allocation, by contrast, leads to irregular memory layouts that are difficult to accelerate in practice.

On kernel customization: any Key channel pruning method needs kernel support to realize speedups. RotateK's fixed-width layout keeps this support simple, unlike token-wise masks, which cause irregular memory access under any kernel (L480–481). In vLLM, it reuses the existing decode path for multi-head latent attention (L506).


W1 The proposed method aims to address the limitations of SPARK. However, Tables 1 and 2 do not include comparisons with SPARK, while Figure 6 shows that SPARK performs better than the proposed method.

TL;DR. RotateK aims to close the gap to token-wise SparK while staying head-wise; at fixed budgets it is at most 0.4 points below SparK yet decodes much faster. Revision: Table 23 (App. E.2).

Answer. RotateK is not designed to beat SparK on accuracy. Its goal is to close the accuracy gap between head-wise and token-wise pruning while keeping the head-wise structure (Sec. 4.2; L1552–1553).

Token-wise masks keep different channels for each token, which prevents a fixed-width paged cache, adds an O(Ld)O(Ld) mask, and makes memory access irregular (L479–481). We left SparK out of Table 1 because its per-token masks take extra memory that the KV budget does not count (L423–425, Figure 7(e)).

Table R-1. SparK added to the Table 1 settings (25% of Key channels). *SparK's KV budget excludes the memory for its per-token masks.

MethodTokenChannelKV CacheTVQAIVQACQADVQAVizWizMean
LLaVA-NeXT-8B
Baseline1.00×1.00×1.00×65.4032.2769.0872.4457.5059.34
FastV + SparK0.40×0.25×0.25×*63.2328.4864.9663.9558.4955.82
FastV + RotateK0.40×0.25×0.25×62.6728.3765.2562.4658.3455.42
VisionZip + SparK0.45×0.25×0.28×*63.3229.2063.2067.7258.5856.40
VisionZip + RotateK0.45×0.25×0.28×62.6228.9762.8866.9958.9456.08
Qwen2.5-VL-7B
Baseline1.00×1.00×1.00×82.9280.1283.0494.4070.5882.21
FastV + SparK0.40×0.25×0.25×*82.4266.8976.8888.1269.9976.86
FastV + RotateK0.40×0.25×0.25×82.3865.7977.4087.2870.3276.63
VisionZip + SparK0.45×0.25×0.28×*80.8871.5080.0091.8769.9678.84
VisionZip + RotateK0.45×0.25×0.28×80.8470.5179.5691.2970.1378.47

RotateK's mean accuracy is slightly lower than SparK's, by at most 0.4 points in every setting, as expected since SparK chooses channels per token. However, RotateK is much faster at decode (Sec. 4.3). SparK can even become slower than full-channel attention, because it must reconstruct full-channel Keys at every step (L468–471). RotateK attends directly over the pruned Keys in a single fused kernel and runs up to 1.6× faster than full-channel attention (L473–475).

AuthorsOpenReview ↗

Response to Reviewer ryZb (2)

Q1 What is the computational cost of dynamically computing (R)?

TL;DR. RR is computed once per input at prefill (≈0.60 ms per layer, independent of N) and reused for every decode step. Revision: Sec. 4.3 and Table 3(a); Tables 20–21 (App. D.6).

Answer. The cost of computing RR is already reported in Sec. 3.2, Sec. 4.3 ("Prefill latency"), and Tables 20–21 (Appendix D.6). RR is computed once per input at prefill and reused for all decode steps. Its cost is ≈0.60 ms per layer, mostly a top-k subspace iteration whose cost does not depend on the number of visual tokens. This amounts to 2.9% of prefill at 8K visual tokens (10.6% at 2K, 5.3% at 4K).


Q2 How representative are the recent queries used by the method of future queries?

TL;DR. Recent prefill queries are as useful as future decode queries: an oracle basis built from the actual decode queries is not consistently better (at most 2.0 points). Revision: Table 6 (App. B.2).

Answer. Since pruned channels cannot be recovered, the basis must be fixed at prefill, so RotateK necessarily uses prefill queries as a proxy for future decode queries (L246–247). To measure how representative they are, we compare RotateK with an oracle that knows the future queries.

Oracle baseline. The full, uncompressed model first generates the answer. The oracle then computes the per-channel query norms σW\sigma_W in Eq. 5 from all of its decode queries instead of the last 32 prefill queries; everything else is unchanged (25% of Key channels; FastV 0.30× and VisionZip 0.35× tokens, as in Table 2(a)). The oracle is non-causal and cannot be deployed.

We evaluate on LLaVA-in-the-Wild because its answers span hundreds of tokens (≈245 on average), so each sample produces many decode queries.

Table R-2. LLaVA-in-the-Wild (GPT judge). Both columns are scored in the same evaluation run.

ModelToken prunerRotateK (last 32 prefill queries)Oracle (future decode queries)
LLaVA-NeXT-8BFastV85.485.1
LLaVA-NeXT-8BVisionZip83.081.0
Qwen2.5-VL-7BFastV104.2104.9
Qwen2.5-VL-7BVisionZip106.4108.2

RotateK and the oracle perform comparably: each is ahead in two of the four settings, by at most 2.0 points, so neither is consistently better. The recent prefill queries are therefore as useful as the future decode queries for constructing the basis, even in long generation.


Q3 Why does applying ThinK to LLaMA3-LLaVA-NeXT-8B lead to a performance drop?

TL;DR. In the original basis, LLaVA-NeXT's informative channels are poorly shared across tokens, so ThinK's channel loss outweighs the extra tokens; RotateK's rotated basis avoids this. Revision: App. E.7 (Table 30).

Answer. We understand the question as follows. In Table 1, at a matched KV cache budget, adding ThinK to token pruning lowers accuracy below token-only pruning on LLaMA3-LLaVA-NeXT-8B (FastV + ThinK 49.91% vs. FastV 52.64%; VisionZip + ThinK 50.43% vs. VisionZip 54.28%), whereas on Qwen2.5-VL-7B it improves over token-only pruning (73.79% vs. 69.17% with FastV; 75.45% vs. 71.86% with VisionZip). Why does ThinK fail specifically on LLaVA-NeXT-8B?

The answer is in Table 30 (Appendix E.7). Suppose each token chose its own most important channels. On LLaVA-NeXT-8B, only 72.8% of the channels that ThinK keeps per head are commonly important, i.e., important for more than half of the tokens, versus 77.7% on Qwen2.5-VL-7B ("channel agreement" in Table 30). In other words, ThinK's head-wise mask captures commonly important channels better on Qwen2.5-VL-7B than on LLaVA-NeXT-8B.

This explains why, on LLaVA-NeXT-8B, the accuracy lost from pruning channels is larger than the accuracy gained from retaining more visual tokens, so ThinK falls below token-only pruning. RotateK keeps the same head-wise structure but applies it in a per-input rotated basis, where important directions are shared across tokens, and it improves over token-only pruning on both models (Table 1).

AuthorsOpenReview ↗

Response to Reviewer ryZb (3)

Q4 What is the overhead of projecting queries and keys using (R) at every decoding step?

TL;DR. Keys are never re-projected at decode; only the incoming query is rotated inside the fused kernel (≈0.05% of per-token FLOPs), already included in all reported latencies. Revision: App. E.4.

Answer. As stated in the paper, Keys are not projected at decode. RkR_k is computed once at prefill, the visual Keys are cached as K~=KRk\tilde{K} = KR_k, and only the incoming query is rotated, inside the decode kernel (L207–208, Algorithm 2). This cost is already included in all reported decode latencies (Sec. 4.3, Fig. 7(b,d)).

The remaining per-step operation, q~t=qtRk\tilde{q}_t = q_t R_k, is a single (1×d)·(d×k) vector–matrix product per head (d=128, k=32). On LLaVA-NeXT-8B, it amounts to ≈8 MFLOPs per generated token, about 0.05% of the model's ≈16 GFLOPs per token. RkR_k is only 8 KB per (layer, KV head), negligible next to the KV cache and weights read at every step (see Q5).


Q5 Since (R) is also cached, is its memory footprint included in the reported KV cache size?

TL;DR. The KV budget counts only visual Keys and Values; RkR_k and δμ\delta\mu add a fixed ≈2.1 MiB per image (0.4–6.6% of stored memory), far below the budget differences behind RotateK's gains. Revision: Sec. 3.2 (L296–298); Table 26 (App. E.4).

Answer. The KV budgets in Tables 1–2 (Sec. 3.2, L291–292) count only the cached Keys and Values. We excluded RkR_k because it is a small, fixed per-request cost that does not scale with the number of visual tokens N.

With head dimension d = 128 and k = 32 retained Key channels, each (layer, KV head) stores RkR_k, a d-to-k dimension projection (d×k values), and the mean correction δμ\delta\mu (d values), 4,224 values in total. This is about the size of 26 compressed visual tokens, or ≈2.2 MB per request on LLaVA-NeXT-8B. Its relative overhead therefore shrinks as N grows: it adds 0.008×, 0.004×, 0.002×, and 0.001× to the KV ratio at 2K, 4K, 8K, and 16K visual tokens, respectively.

Including it does not change our conclusions. RkR_k amounts to under 1% of the KV budget (about 26 visual tokens per request), whereas the gains of RotateK over token-only pruning in Tables 34–35 correspond to much larger budget differences. Measured peak memory (Fig. 7(f)) already includes RkR_k, since it is taken from actual GPU allocations.


Flag for ethics review. We noticed the review flags the submission for ethics review but does not specify a concern. To our understanding, the work raises no ethical issues: it proposes a training-free inference-efficiency method, uses only publicly released models and standard public benchmarks under their respective licenses, and involves no human subjects, personal data, or new data collection. Our use of LLM assistance in writing is disclosed in the paper in accordance with the ICLR policy. We would be grateful if the reviewer could clarify the specific concern, and we will gladly address it.

Reviewer_5H7MOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

To address the visual KV cache memory expansion in vision language models and the loss of fine grained details caused by pure token pruning, this paper proposes RotateK, a training free structured Key channel pruning framework. By constructing a query weighted orthogonal rotation basis online for each input during the prefill stage, it aligns and concentrates scattered informative features across different tokens into a few shared channels, enabling hardware friendly per head pruning to retain more visual tokens with fewer channels. Under the same memory budget, RotateK not only approaches the accuracy of unstructured pruning and substantially outperforms pure token pruning, but also achieves a 1.60x increase in KV capacity and up to a 1.29x decoding speedup in vLLM.

Critical strengths
  1. The paper offers a new perspective on why structured Key pruning fails, showing that the issue is not merely the lack of flexibility in per head masks, but rather that the original channel basis is not well suited to represent the informative directions of the current input.
  2. Beyond proposing the core algorithm, this work delivers practical system solutions that reduce computational complexity and accelerate inference in real world deployment scenarios, achieving strong empirical results.
Critical weaknesses
  1. Eq. 3 explicitly characterizes the attention score error introduced by projection, but eigendecomposition of the query-weighted covariance does not generally minimize this error. Although the appendix acknowledges this limitation and provides empirical validation, the main text should distinguish more clearly between the stated objective and the heuristic design. The writing around Eq. 3 could be better organized to motivate and transition into the query weighted PCA module.
  2. Prefill queries are used as a proxy for subsequent decoding queries, with a fixed window size of W = 32. However, the paper does not analyze different window sizes, query sampling positions, or stages of generation. In addition, although mean correction has a clear mathematical motivation, its contribution to attention fidelity and downstream accuracy has not been quantified through an isolated ablation.
  3. The experiments cover four token pruning strategies, but it remains unclear whether retaining additional tokens would provide comparable benefits when combined with diversity-based token pruning methods such as DivPrune[1] and CDPruner[2]. These methods may also change the distribution of retained Keys and consequently affect channel compressibility. Further evaluation would strengthen the claims about the general applicability of the combination.

[1] https://arxiv.org/abs/2503.02175 [2] https://arxiv.org/abs/2506.10967

Questions for authors
  1. Could fixing W to 32 be insufficient for OCR tasks such as OCRBench[3], given that varying document information might require adjusting this value flexibly, and could comprehension oriented VQA tasks also face a similar issue?
  2. Could the authors provide an ablation that enables and disables mean correction while keeping all other settings unchanged, and report attention output error and task accuracy across different channel retention ratios? Does this correction primarily affect the relative allocation of attention between visual and text tokens?
  3. Could the authors evaluate RotateK in combination with diversity-oriented token pruning methods such as DivPrune or CDPruner? At matched visual KV cache budgets, reporting accuracy, prefill latency, decoding latency, and end-to-end latency would help determine whether the additional accuracy gains justify the increased prefill cost.

If the authors adequately address the concerns above, I will raise my rating.

[3] https://arxiv.org/abs/2305.07895

7 comments
AuthorsOpenReview ↗

Response to Reviewer 5H7M (1)

We sincerely thank the reviewer for the insightful comments. We hope we have addressed the concerns and questions below, and that the reviewer will reconsider the rating. Unless marked as "original", all table, section, equation, and line references follow the revised PDF.


W1 Eq. 3 explicitly characterizes the attention score error introduced by projection, but eigendecomposition of the query-weighted covariance does not generally minimize this error. Although the appendix acknowledges this limitation and provides empirical validation, the main text should distinguish more clearly between the stated objective and the heuristic design. The writing around Eq. 3 could be better organized to motivate and transition into the query weighted PCA module.

TL;DR. Query-weighted PCA is a score-aware surrogate of the averaged Eq. 3 objective rather than its exact minimizer. Revision: Sec. 3.1 restructured around the averaged objective (Eq. 4), pointing to App. B.1.

Answer. We agree that Eq. 3 is the error we would ideally minimize, but query-weighted PCA does not minimize it exactly. We have restructured Sec. 3.1 to separate this objective from our design, as follows.

Objective. Averaged over decode queries, the error in Eq. 3 becomes

Eq∥q(I−Pk)Kc⊤∥2=tr((I−Pk)C(I−Pk)ΣQ)\mathbb{E}_q \lVert q(I-P_k)K_c^\top \rVert^2 = \mathrm{tr}\big((I-P_k) C (I-P_k) \Sigma_Q\big) \quad (4)

where KcK_c are the mean-centered Keys (the mean is restored at decode, see W2-Part2), C=Kc⊤KcC = K_c^\top K_c, and ΣQ=E[q⊤q]\Sigma_Q = \mathbb{E}[q^\top q]. Eq. (4) has no closed-form minimizer unless ΣQ∝I\Sigma_Q \propto I, in which case it reduces to Key PCA.

Surrogate. To obtain a closed-form solution, we heuristically relax Eq. (4) while keeping its main term. We keep only the diagonal of ΣQ\Sigma_Q, Dq2=diag(ΣQ)D_q^2 = \mathrm{diag}(\Sigma_Q), which carries the per-channel query energy in Eq. (4), and drop its cross-channel terms. Weighting the Keys by DqD_q before projection gives the weighted Key PCA ∥KcDq(I−Pk)∥2\Vert K_c D_q (I-P_k)\Vert ^2, solved by the top-kk eigenvectors of DqCDqD_q C D_q (Eq. 5). The diagonal is also what we can reliably estimate at prefill from the last WW queries, which gives Dq∝diag(σW)D_q \propto \mathrm{diag}(\sigma_W). This relaxation generalizes Key PCA (Dq∝ID_q \propto I) and retains the per-channel query magnitudes in Eq. (4).

The revised Sec. 3.1 now presents query-weighted PCA as this surrogate rather than the exact minimizer of Eq. (4), and refers to App. B.1 for an empirical evaluation of the score error.

AuthorsOpenReview ↗

Response to Reviewer 5H7M (2)

W2-Part1 Prefill queries are used as a proxy for subsequent decoding queries, with a fixed window size of W = 32. However, the paper does not analyze different window sizes, query sampling positions, or stages of generation.

TL;DR. The score error saturates at W = 32 and plateaus over generation instead of accumulating, and query positions barely change accuracy. Revision: Tables R-1–R-3 added to App. B.2 as Tables 7–9 (Table 9 also includes Table R-5).

Answer. We measure the score error (25% Key channels) on LLaVA-Bench (In-the-Wild) with up to 512 generated tokens. As a reference, we compute the query weighting from the decode queries that the sample actually generates, and use this basis to measure the score error over the whole generation ("decode-query reference").

Table R-1. Attention-score error on decode queries (lower is better), mean over FastV and VisionZip.

Query proxyLLaVA-NeXT-8BQwen2.5-VL-7B
Last W = 80.3850.274
Last W = 32 (default)0.3790.262
Last W = 1280.3740.261
32 uniformly spaced positions0.3490.263
First 32 (system prompt)0.4220.277
Decode-query reference0.3330.262
  • Window size. The error saturates at W = 32. W = 128 changes it by at most 0.005, while W = 8 is slightly worse on Qwen2.5-VL (0.274 vs. 0.262).
  • Sampling positions. Our default places the 32 queries at the end of the prefill. Spreading them uniformly lowers the error on LLaVA-NeXT (0.349 vs. 0.379) and matches the default on Qwen2.5-VL. Placing them on the system prompt ("First 32"), which precedes the question, gives the highest error on both models.
  • Stages of generation. Each basis is built once at prefill and kept for the whole generation. We measure its score error separately on the decode queries of each stage (Table R-2), to test whether it grows as generation moves away from the prefill.

Table R-2. Attention-score error by decode step, mean over FastV and VisionZip.

Decode stepsLLaVA: last 32LLaVA: uniform 32LLaVA: referenceQwen: last 32Qwen: uniform 32Qwen: reference
00.3370.3280.3160.2490.2510.254
8–160.3550.3360.3200.2550.2560.257
64–1280.3890.3540.3360.2670.2670.265
256–5120.3830.3500.3360.2600.2610.259

The error rises over the first ~100 steps and then plateaus rather than accumulating. On Qwen2.5-VL, the prefill proxy matches the reference at every stage. On LLaVA-NeXT, the default proxy ends 14% above the reference (0.383 vs. 0.336), partly because the reference itself drifts (0.316 → 0.336) as decode queries move away from the prompt.

Table R-3. LLaVA-W accuracy with the last vs. uniformly spaced 32 prefill queries (Table 2(a) setting, GPT-4o-mini judge).

Model / prunerLast 32 (default)Uniform 32
LLaVA-NeXT + FastV85.485.2
LLaVA-NeXT + VisionZip83.081.8
Qwen2.5-VL + FastV104.2106.7
Qwen2.5-VL + VisionZip106.4106.0

Accuracy. These differences in score error do not always translate into accuracy: on LLaVA-W, our default (last 32) and uniform placements achieve comparable accuracy, within 2.5 points with no consistent direction (Table R-3).

AuthorsOpenReview ↗

Response to Reviewer 5H7M (3)

Q1 Could fixing W to 32 be insufficient for OCR tasks such as OCRBench, given that varying document information might require adjusting this value flexibly, and could comprehension oriented VQA tasks also face a similar issue?

TL;DR. W = 32 suffices for both OCR and comprehension-oriented VQA: varying W from 8 to 128 changes OCRBench accuracy by at most 1.9 points and the LLaVA-in-the-Wild score by at most 1.7 points. Revision: Tables R-4 and R-5 added to App. B.2 as Tables 10 and 9.

Answer.

OCR tasks. No, fixing W to 32 is sufficient for OCRBench. We vary the window on the full OCRBench (1,000 questions).

Table R-4. OCRBench accuracy (%) at a matched visual KV budget (0.30×). Token-only keeps 0.30 of the tokens; ThinK and RotateK keep 0.48 tokens + 25% Key channels. RotateK uses the last W prefill queries (W = 32 is our default).

SettingFullToken-only+ ThinK+ RotateK W=8W=32W=128
LLaVA-NeXT + FastV53.348.835.949.349.549.7
LLaVA-NeXT + VisionZip53.351.636.050.951.351.7
Qwen2.5-VL + FastV83.964.267.974.475.175.1
Qwen2.5-VL + VisionZip83.970.969.677.678.779.5

With the default W = 32, RotateK outperforms ThinK in every setting and token-only in three of four. Varying the window from 8 to 128 changes accuracy by at most 1.9 points, so a fixed W = 32 is not a bottleneck on OCR. Since accuracy is flat across windows, an input-adaptive window would bring little benefit.

Comprehension-oriented VQA. These tasks are similarly insensitive to the window. On LLaVA-in-the-Wild, varying W from 8 to 128 changes the score by at most 1.7 points, and the default W = 32 stays within 1.2 points of the best window in every setting (Table R-5). Spreading the 32 queries uniformly also changes the score by at most 2.5 points, with no consistent direction. With W = 32, RotateK outperforms both token-only pruning and ThinK in all eight settings on LLaVA-in-the-Wild and MM-Vet (Table 2(a)).

Table R-5. LLaVA-in-the-Wild score (GPT-4o-mini judge) with the last W prefill queries (W = 32 is our default) or 32 uniformly spaced prefill queries, in the setting of Table 2(a).

SettingW=8Last 32W=128Uniform 32
LLaVA-NeXT + FastV84.885.483.785.2
LLaVA-NeXT + VisionZip83.983.082.881.8
Qwen2.5-VL + FastV104.4104.2103.5106.7
Qwen2.5-VL + VisionZip107.6106.4106.5106.0
Reviewer_VbLnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VLMs face severe KV-cache pressure, as a single image can encode into thousands of tokens. Rather than pruning tokens, the paper proposes feature sparsity: compressing the channel dimension so more visual tokens fit the same KV budget. It identifies a trade-off in existing Key channel pruning: token-wise is accurate but unstructured and slow, head-wise is hardware-friendly but degrades at high compression, and attributes the latter to informative Key directions shifting across tokens and inputs, so no fixed channel subset serves all. Their method, RotateK, builds a query-weighted Key basis per input at prefill, letting a head-wise mask retain attention-relevant channels, backed by a graph-capturable top-k eigensolver, a fused sparse-channel decode kernel, and a paged vLLM KV layout. Across six VLMs up to 32B and four token pruners, it beats token-only pruning at matched budgets and matches token-wise accuracy, while raising KV capacity and throughput.

Critical strengths
  1. Engineering is complete for this literature: a fused kernel, a graph-capturable eigensolver, and a paged vLLM backend is real work.
  2. Extensive experiments across 6 VLMs up to 32B, 4 baselines, 5 VQA + 2 open-ended benchmarks, plus MLVU-needle and V*Bench.
  3. The paper is well-written and easy-to-follow.
Critical weaknesses
  1. The core contribution reduces to prefill-time Key PCA. Table 15 says per-input construction drives most of the gain while Table 3 shows query weighting losing in 5 of 12 cells. Yet ShadowKV and Loki, which already build prompt-dependent Key bases at prefill, are cited but never run as baselines.

  2. Tensor parallelism is never mentioned and all experiments run on a single A100-80GB, so the vLLM deployment claim is untested at the scale where KV capacity actually binds.

Questions for authors

none

4 comments
AuthorsOpenReview ↗

Response to Reviewer VbLn (1)

We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.


W1 The core contribution reduces to prefill-time Key PCA. Table 15 says per-input construction drives most of the gain while Table 3 shows query weighting losing in 5 of 12 cells. Yet ShadowKV and Loki, which already build prompt-dependent Key bases at prefill, are cited but never run as baselines.

TL;DR. Loki in fact corresponds to a static PCA basis, which RotateK exceeds by up to 17.2 points; RotateK is on par with ShadowKV's Key projection, whose head-shared basis stores Hkv×H_{kv}\times more basis values and, under tensor parallelism, must be replicated on every GPU or exchanged at each decode step. Revision: Table 17 (App. B.5); Table 32 (App. E.9).

Answer. We agree that per-input projection is central to RotateK, and that query weighting is a secondary design on top of it (Sec. 3.1). However, neither Loki nor ShadowKV allows a direct, apple-to-apple comparison with RotateK.

1) Loki: it computes its Key projection offline on calibration data and reuses it for every input, so its projection is not prompt-dependent; it also uses the projection to select tokens rather than to compress the KV cache. Its offline basis corresponds to the static PCA of Table 32 (Table 15 in the original submission), which RotateK exceeds by up to 17.2 points at 12.5% Key channels.

2) ShadowKV: it does build a prompt-dependent low-rank Key projection, but much of its contribution also lies in sparse attention and Value offloading to the CPU. A direct comparison is therefore difficult, and we compare only its Key projection mechanism with RotateK, at the same number of stored values per token (Table R-1).

Comparison of Key projections. At 25% Key channels, RotateK and ShadowKV's Key projection are on par on Qwen2.5-VL (74.2 vs. 73.9 with FastV; 74.2 vs. 74.9 with VisionZip), while ShadowKV is slightly ahead on LLaVA-NeXT (+0.9 and +1.9). The gap comes from sharing the basis across heads.

Table R-1: Per-input Key projections at 25% Key channels. Accuracy (%) on TextVQA / ChartQA / InfoVQA (mean), with the same number of stored values per token. ShadowKV denotes only its Key projection (one pre-RoPE basis shared across KV heads). Basis overhead: values stored per input per layer.

SettingUncompressedRotateKShadowKV
LLaVA-NeXT + FastV67.7 / 59.2 / 28.4 (51.8)65.6 / 58.2 / 29.4 (51.1)67.5 / 59.4 / 29.0 (52.0)
LLaVA-NeXT + VisionZip68.5 / 60.6 / 30.3 (53.1)65.3 / 59.0 / 30.2 (51.5)69.2 / 60.4 / 30.7 (53.4)
Qwen2.5-VL + FastV85.3 / 69.6 / 67.0 (74.0)85.2 / 70.8 / 66.7 (74.2)85.0 / 69.8 / 67.0 (73.9)
Qwen2.5-VL + VisionZip83.7 / 72.0 / 69.9 (75.2)81.4 / 72.0 / 69.2 (74.2)83.2 / 71.6 / 69.9 (74.9)
Basis (LLaVA-NeXT / Qwen2.5-VL)–33K / 16K262K / 66K

Sharing across heads is a design choice, not a free gain. It stores Hkv×H_{kv}\times more basis per input (HkvH_{kv}: number of KV heads), and it ties all heads to the same per-token coefficients. This matters for tensor parallelism, which the reviewer raises in W2: under head-sharded TP, these shared coefficients must be replicated on every GPU or exchanged at each decode step, whereas our per-head basis keeps each head independent. Whether to share thus depends on model size and parallelism.

RotateK's contribution relative to ShadowKV. Beyond the Key projection itself, RotateK uses channel sparsity to keep more visual tokens at the same KV budget (Sec. 4.1), identifies basis mismatch as the cause of head-wise degradation (Sec. 2), and, because the mask stays head-wise, turns this into real speedups in vLLM (1.60× KV capacity, up to 1.29× decode; Sec. 4.3).

AuthorsOpenReview ↗

Response to Reviewer VbLn (2)

W2 Tensor parallelism is never mentioned and all experiments run on a single A100-80GB, so the vLLM deployment claim is untested at the scale where KV capacity actually binds.

TL;DR. Each head's basis uses only that head's Keys, so TP adds no communication; the 1.60× capacity gain and 1.22–1.25× decode speedup hold at TP = 2 and 4. Revision: Sec. 4.3; Table 19 (App. D.5).

Answer. RotateK is seamlessly compatible with tensor parallelism (TP). TP shards attention by head, and RotateK builds each head's basis from that head's own Keys, so each GPU handles only its own heads and RotateK adds no communication. We measured RotateK in vLLM at TP = 1, 2, and 4 with the protocol of Table 18 (A100-80GB GPUs, 16K-token contexts; Table R-2).

Table R-2: RotateK in vLLM under tensor parallelism. KV capacity is the number of tokens the KV cache holds at a fixed memory budget. Speedup is full-Key decode time divided by RotateK decode time per step. "–": the full-Key cache cannot hold the batch (RotateK can).

ModelTPKV capacity (tokens)Max concurrent 16K requestsDecode speedup, B = 16 / 32 / 64
Qwen2.5-VL-7B10.90M → 1.45M (1.60×)55 → 881.15 / 1.22 / –
22.07M → 3.32M (1.60×)126 → 2021.13 / 1.22 / 1.23
44.42M → 7.08M (1.60×)269 → 4311.01 / 1.08 / 1.24
LLaVA-NeXT-8B10.40M → 0.64M (1.60×)25 → 401.28 / – / –
20.92M → 1.48M (1.60×)57 → 921.18 / 1.25 / –
41.98M → 3.16M (1.60×)123 → 1971.02 / 1.25 / 1.24

1) The capacity gain holds under TP. RotateK stores 1.60× more KV tokens at every TP size, so the same GPUs serve 1.6× more concurrent 16K-token requests (e.g., 123 → 197 for LLaVA-NeXT at TP = 4).

2) The decode speedup holds under TP. The per-step decode speedup reaches 1.22–1.25× at TP = 2 and 4, as on a single GPU (1.22–1.28×). With more GPUs, each GPU holds less KV per request, so decoding becomes KV-bound only at larger batches, and the gain appears there (B = 64 at TP = 4). These are the batch sizes that the additional memory from TP makes possible.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer VbLn,

We would like to sincerely thank you for your positive assessment of our work. We were truly encouraged by your kind remarks on the system implementation and the breadth of our experiments, and we greatly appreciate the time and care you devoted to reviewing our paper.

Your insightful comments have helped us strengthen the work, and we have incorporated the corresponding additions, including the comparison with ShadowKV and Loki and the tensor-parallel evaluation, into the revised manuscript.

Best regards,

Authors

Dear Reviewer VbLn,

Thank you again for your positive assessment of our work and for your encouraging comments. As the author-reviewer discussion period closes in about 12 hours, we wanted to briefly share a summary of how we addressed your comments, in case it is helpful:

  • Comparison with Loki and ShadowKV (W1): Loki corresponds to a static PCA basis, which RotateK exceeds by up to 17.2 points; RotateK is on par with ShadowKV's Key projection, whose head-shared basis stores more basis values and, under tensor parallelism, must be replicated on every GPU or exchanged at each decode step (Table 17, App. B.5; Table 32, App. E.9).
  • Tensor parallelism (W2): each head's basis uses only that head's Keys, so TP adds no communication; the 1.60× capacity gain and 1.22–1.25× decode speedup hold at TP = 2 and 4 (Sec. 4.3; Table 19, App. D.5).

If anything remains unclear, we would be happy to clarify.

Best regards,
Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and the AC

We thank all reviewers for their constructive feedback. We summarize the key contributions and major revisions below.

Strengths recognized by reviewers.

  • Channel-basis mismatch: a new explanation of why head-wise Key pruning degrades (5H7M, Ua6L).
  • Practical system implementation: graph-capturable eigensolver, fused sparse-channel decode kernel, and paged vLLM backend (VbLn, 5H7M, Ua6L).
  • Extensive evaluation: six VLMs up to 32B and multiple token pruners (VbLn, Ua6L).

Core contribution. RotateK uses head-wise, not token-wise, pruning. Our key insight is that the accuracy gap between head-wise and token-wise pruning stems substantially from the channel basis, rather than mask granularity alone. By constructing an input-adaptive basis shared across each KV head, RotateK approaches token-wise SparK accuracy (within 0.4 points; Table 23) while enabling structured KV storage, 1.60× KV capacity, and up to 1.29× faster decoding in vLLM.

Major revisions:

  • Additional baselines and token pruners (VbLn, ryZb, 5H7M): comparisons with SparK, ShadowKV, and Loki (Tables 17, 23, 32), and combination with DivPrune and CDPruner (Tables 24–25).
  • Scalability and efficiency (VbLn, ryZb, Ua6L): tensor parallelism (TP = 1/2/4; Table 19), basis-construction cost (Tables 20–21), complete memory accounting (Table 26), and end-to-end latency (Tables 3, 22).
  • Algorithmic validation (5H7M, ryZb, Ua6L): query proxy and oracle analysis (Tables 6–10), mean correction (Tables 11–12), direct basis-mismatch evidence (Tables 30–31), budget allocation (Tables 13–14), and eigensolver variance (Table 4).
  • Writing clarification (5H7M): Sec. 3.1 now distinguishes the attention-score error objective from the query-weighted PCA surrogate.

We hope these revisions address the reviewers' concerns, and we thank the reviewers and the AC for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.