- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a training-free method that rotates visual key representations into an input-adaptive basis before pruning their channels. RotateK integrates with existing vision token pruning methods, allowing vision language models to retain more visual tokens at the same visual KV-cache budget. The custom kernels accelerate decoding relative to full-channel baselines.
Critical strengths
- The authors provide an analysis of the basis mismatch, which is particularly relevant for fine-grained visual understanding tasks where aggressive token pruning can lead to substantial performance degradation.
- The proposed method provides an interesting solution to the limitations of structured channel pruning. The query weighted PCA formulation is well motivated by the objective of minimizing attention-score perturbations.
- Beyond algorithmic contributions, the paper provides a efficient implementation compatible with modern frameworks. The reported improvements in decoding throughput and KV cache capacity demonstrate practical benefits. Although RotateK adds a layer of additional latency to the prefill stage, compared to the vision token pruning methods, I believe the improvements in benchmark results make the trade off a worthy one. I encourage the authors to highlight the latency measurements in the main tables (which are currently in appendix tables 9, 10).
- The paper reports extensive experiments across a diverse combination of settings. RotateK consistently improves performance over token-only pruning under matched KV cache budgets and achieves accuracy comparable to token-wise channel pruning. Furthermore The paper includes extensive ablation studies, providing insights into the method.
Critical weaknesses
- Sec. 2 motivates RotateK through the observation that informative K channels vary across tokens. However, Fig. 2 visualizes pruning decisions rather than directly measuring channel importance, while the all-token intersection metric in App. E.4 may exaggerate disagreement even when individual masks largely overlap. Although Fig. 6, Tables 4 and 6 provide supporting evidence for rotation and compact Key subspaces, the connection between mask disagreement and attention-relevant information loss remains indirect. This limits how strongly Sec 2 supports basis mismatch as an explanation for the degradation of head-wise pruning.
- RotateK introduces an additional degree of freedom for KV-cache compression. However, the paper provides limited guidance on how to allocate a fixed memory budget between these dimensions. For a given VLM, token-pruning method, and task, how should users choose the retention ratios? While the paper evaluates several configurations, it is unclear how these configurations were selected and whether they are near-optimal. Moreover, how sensitive is downstream performance to the budget allocation, and can a configuration selected for one model or workload transfer effectively to another? The authors should provide a systematic analysis of this trade-off, together with practical guidelines or an inexpensive selection heuristic.
- Sec. 3.2 compares configurations using the visual KV budget (). This does not include intermediate matrices, such as the cached rotation matrices, mean-correction vectors, or uncompressed prompt and generated-text KV states. Although the paper demonstrates practical memory savings, it remains unclear whether budget matching consistently accounts for these additional states and their storage precision. Their relative cost may become substantial when few visual tokens are retained. The authors should provide complete memory accounting across representative input lengths by distinguishing visual KV payload from total allocated memory.
- Sec. 3.2 states that matched-budget configurations differ only in prefill, but Table 10 shows that joint pruning remains approximately 11–16% slower e2e than token-only pruning at batches 8–32. The authors should qualify this statement and evaluate different generation lengths, since decode parity does not establish equality at the request level.
- Minor: Table 3 shows that using full eigendecomposition leads to better results, with differences up to 1.14. The authors should qualify the claims on “purely latency” (App B.1) and ridge’s “no measurable effect” (App. F), and report uncertainty across different random initializations.
Questions for authors
See weaknesses.
7 comments
Response to Reviewer Ua6L (2)
W2 RotateK introduces an additional degree of freedom for KV-cache compression. However, the paper provides limited guidance on how to allocate a fixed memory budget between these dimensions. For a given VLM, token-pruning method, and task, how should users choose the retention ratios? While the paper evaluates several configurations, it is unclear how these configurations were selected and whether they are near-optimal. Moreover, how sensitive is downstream performance to the budget allocation, and can a configuration selected for one model or workload transfer effectively to another? The authors should provide a systematic analysis of this trade-off, together with practical guidelines or an inexpensive selection heuristic.
TL;DR. Keep 25% of the Key channels (best on average, or within 1.1 points of the best) and spend the saved memory on more visual tokens. Revision: Tables 13–14 and guideline (App. B.4); Figure 5 caption.
Answer. Our guideline is simple: keep 25% of the Key channels, and choose the token ratio as for token-only pruning, then raise it to use the KV memory that the channel pruning saves. The allocation is simpler than the two continuous ratios suggest, as we show below.
(1) The channel ratio has only a few practical values. Decode kernels use power-of-two tile sizes, so the retained Key width is efficient only at 16, 32 or 64 of 128 channels (12.5%, 25% or 50%); other widths, such as 48, are padded to the next power of two and are slower in practice. The allocation therefore reduces to choosing one of three channel ratios, and the token ratio then follows from the budget.
(2) Sensitivity, near-optimality and transfer. Table R-2(a) varies the allocation at a fixed visual KV budget (that of Table 1), and Table R-2(b) breaks the 25% default down by task. We include 37.5% for completeness, although it is slower in practice.
Table R-2(a). Mean accuracy (%) over TextVQA, InfoVQA, ChartQA, DocVQA and VizWiz at a fixed visual KV budget. Columns: retained Key channels. Token ratios: FastV 0.25 / 0.33 / 0.36 / 0.40 / 0.44, VisionZip 0.28 / 0.37 / 0.41 / 0.45 / 0.50.
| Setting | 100% (token-only) | 50% | 37.5% | 25% | 12.5% |
|---|---|---|---|---|---|
| LLaVA-NeXT + FastV (0.25×) | 51.24 | 53.65 | 54.10 | 53.44 | 48.16 |
| LLaVA-NeXT + VisionZip (0.28×) | 53.44 | 55.57 | 54.92 | 54.43 | 47.98 |
| Qwen2.5-VL + FastV (0.25×) | 66.77 | 71.40 | 72.62 | 73.95 | 68.82 |
| Qwen2.5-VL + VisionZip (0.28×) | 68.91 | 72.84 | 73.46 | 74.39 | 70.36 |
Table R-2(b). Accuracy (%) with 25% of the Key channels per task; in parentheses, the difference to the best of the five allocations for that task, and that allocation when it is not 25%.
| Setting | TVQA | IVQA | CQA | DVQA | VizWiz | Mean |
|---|---|---|---|---|---|---|
| LLaVA-NeXT + FastV | 65.6 (−1.3, 50%) | 29.4 (0.0) | 58.2 (−1.4, 37.5%) | 62.9 (−1.4, 37.5%) | 51.1 (0.0) | 53.4 (−0.7, 37.5%) |
| LLaVA-NeXT + VisionZip | 65.3 (−2.9, 50%) | 30.2 (0.0) | 59.0 (−0.6, 50%) | 66.3 (−2.4, 50%) | 51.3 (−0.2, 37.5%) | 54.4 (−1.1, 50%) |
| Qwen2.5-VL + FastV | 85.2 (−0.6, 37.5%) | 66.7 (0.0) | 70.8 (−0.2, 12.5%) | 86.5 (0.0) | 60.5 (0.0) | 73.9 (0.0) |
| Qwen2.5-VL + VisionZip | 81.4 (−0.2, 50%) | 69.2 (0.0) | 72.0 (−1.0, 12.5%) | 90.3 (0.0) | 59.0 (0.0) | 74.4 (0.0) |
Every allocation between 50% and 25% of the Key channels outperforms token-only pruning on average at the same budget, and token-only is never the best allocation on any task. A single default of 25% transfers well across models, pruners and tasks: it is the best allocation on average on Qwen2.5-VL, and on LLaVA-NeXT it is within 1.1 points of the best. Pruning further to 12.5% is too aggressive and should be avoided. Since we used 25% throughout the paper without tuning, the reported gains are conservative on LLaVA-NeXT.
Response to Reviewer Ua6L (3)
W3 Sec. 3.2 compares configurations using the visual KV budget. This does not include intermediate matrices, such as the cached rotation matrices, mean-correction vectors, or uncompressed prompt and generated-text KV states. Although the paper demonstrates practical memory savings, it remains unclear whether budget matching consistently accounts for these additional states and their storage precision. Their relative cost may become substantial when few visual tokens are retained. The authors should provide complete memory accounting across representative input lengths by distinguishing visual KV payload from total allocated memory.
TL;DR. and add a fixed ≈2.1 MiB per image (0.4–6.6% of the total stored memory), which is now reported explicitly. Revision: Sec. 3.2; Table 26 and the per-benchmark overhead (App. E.4).
Answer. We thank the reviewer and agree that the visual KV budget omits these states, so we measured every tensor held after generation.
Rotation and mean correction. Per layer and KV head, RotateK stores (128 × 32) and (128) in bf16, i.e., 8.25 KiB, the size of 16.5 full-KV visual tokens. This totals 2.06 MiB on LLaVA-NeXT-8B and 0.90 MiB on Qwen2.5-VL-7B per image, independent of sequence and generation length.
Complete accounting. Table R-3 compares token-only pruning and RotateK at the same visual KV budget (30-token prompt and question, 128 generated tokens). Prompt and text KV are uncompressed and identical in both, so and are RotateK's only extra state.
Table R-3. Memory (MiB) stored for one request on LLaVA-NeXT-8B at a matched visual KV budget (token-only / RotateK). N: visual tokens retained by token-only pruning; RotateK retains 1.6N. Qwen2.5-VL-7B gives the same percentages.
| N | Visual KV | Prompt + text KV | + | Total stored |
|---|---|---|---|---|
| 64 | 8.0 / 8.0 | 23.4 / 23.4 | – / 2.1 | 31.4 / 33.4 (+6.6%) |
| 256 | 32.0 / 32.0 | 23.4 / 23.4 | – / 2.1 | 55.4 / 57.5 (+3.8%) |
| 1024 | 128.0 / 128.0 | 23.4 / 23.4 | – / 2.1 | 151.4 / 153.4 (+1.4%) |
| 4096 | 512.0 / 512.0 | 23.4 / 23.4 | – / 2.1 | 535.4 / 537.5 (+0.4%) |
Their relative cost grows as fewer visual tokens are kept but stays small: +6.6% of the total at N = 64, +1.4% at 1024, and +0.4% at 4096. On our benchmarks, it is 1.7–2.5% on LLaVA-NeXT and 2.4–7.3% on Qwen2.5-VL, the upper end being ChartQA with its small images.
Response to Reviewer Ua6L (1)
We sincerely thank the reviewer for their insightful comments on our manuscript. We hope we have addressed their concerns and questions regarding the paper and hope they will reconsider their rating. Unless marked as "original", all table, section, equation, and line references below follow the revised PDF.
TL;DR. Agreed; both latency tables are now in the main text as Table 3 (Sec. 4.3), with the matched-budget accuracy shown alongside. Revision: Sec. 4.3 and Table 3(a,b); full results in Tables 21–22 (App. D.6).
Answer. We thank the reviewer for this assessment and agree that the latency results belong in the main text. In the revision, we have moved both latency results into Sec. 4.3 as Table 3: (a) basis construction cost and (b) end-to-end latency at matched KV budgets, with the matched-budget accuracy alongside. Full results are in Tables 21–22 (App. D.6).
TL;DR. Pairwise overlap (only 17–19 of 32 channels shared) and the attention error of head-wise vs. per-token masks support basis mismatch beyond mask disagreement. Revision: Sec. 1 (first contribution, L093–097); Sec. 2 (L146–149, L152–157); Tables 30–31 (App. E.7–E.8).
Answer. We agree with both points: the all-token intersection of App. E.7 may overstate the disagreement, and Fig. 2 and App. E.7 show that selections differ across tokens, but do not explicitly show that this causes attention-relevant loss. We address each below.
1) Pairwise overlap instead of the all-token intersection. When each visual token keeps its own top-32 Key channels (as in SparK), only 3 (LLaVA-NeXT) and 5 (Qwen2.5-VL) channels are shared by all tokens, which overstates disagreement. Over 512 random token pairs per layer and head, two tokens still share only 17 and 19 of their 32 channels on average: the disagreement is smaller, but still substantial.
2) Attention error of head-wise and per-token masks. To test whether this disagreement causes attention-relevant loss, we restrict the visual Keys to 32 of 128 channels per KV head in three ways: (i) one mask per head (ThinK), (ii) one mask per token (SparK), and (iii) one subspace per head in RotateK's rotated basis. All three restore the mean of the discarded part, so (i) vs. (ii) isolates the selection and (i) vs. (iii) isolates the basis.
Table R-1. Error against uncompressed visual Keys (ref) on decode queries (FastV and VisionZip × TextVQA, ChartQA, InfoVQA × 50 samples). Attention-weight error: 21∥p−pref∥1, where p is the attention distribution of a decode query over all cached tokens; output error: ∥o−oref∥/∥oref∥, where o is the attention output. The ordering (i) > (ii) > (iii) holds in all 12 settings for both errors.
A head-wise mask raises both errors over per-token masks, so the disagreement in 1) does distort attention. A shared subspace in the rotated basis falls even below per-token masks: the informative directions are shared across tokens but not aligned with the channel axes. This is the basis mismatch that degrades head-wise pruning.