- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes R2Pruner, a training-free method for pruning vision tokens inside the LLM. The authors argue that a token is redundant if it remains highly similar even to its K-th most similar token, and formulate this as a worst-case local dissimilarity prior. Based on this prior, they define a token-level score () and a neighborhood-level score (), and multiply them with an existing token importance score and question-conditioned relevance at the first of three progressive pruning stages. The authors evaluate R2Pruner on LLaVA-1.5-7B and Qwen2-VL across multiple benchmarks and analyze its efficiency.
Critical strengths
1. Strong empirical results under aggressive pruning. R2Pruner achieves the best average performance among the compared training-free methods on both LLaVA-1.5-7B and Qwen2-VL, with larger margins at lower token budgets.
2. Training-free and lightweight. The proposed scores require no additional training or parameters and are computed from existing hidden states. Computing them adds little overhead relative to LLM prefill.
Critical weaknesses
1. Novelty over prior work seems limited. The idea that token importance alone is insufficient and that redundancy should also be considered has been explored in prior work [A, B, C]. In particular, [A] already prunes tokens by their redundancy inside the LLM. The proposed method differs from these works mainly in how redundancy is measured: it derives local dissimilarity scores from the K-th-neighbor similarity of LLM hidden states and aggregates them across neighboring tokens. However, the paper neither cites nor compares with these methods, so it remains unclear whether this approach offers an advantage. The contribution therefore appears incremental.
2. The contribution of the proposed components is not established.
-
Effect of and . In Table 2, the "w/o , " variant uses only the CLSE score and the question-conditioned relevance . At 64 tokens, it already matches or outperforms CLSE on all four benchmarks. On POPE and SEEDB, most of the gain over CLSE thus appears to come from or the pruning setting rather than from and . Furthermore, adding and improves the results by only about 1% (relative) on average, so their benefit seems marginal.
-
Ablation. Table 2 does not include ablations that remove only or only from the full model; the ablations are reported only for variants without . The contribution of each to the full model is therefore unclear. Even in the reported ablations, removing gives nearly identical results (rows 2–3). In addition, some statements in the text are inconsistent with Table 2:
- The text states that removing causes a larger drop (L468), but MME rises from 1792 to 1801.
- The text states that alone yields the weakest results (L469), but using only and scores lower on POPE (74.8 vs. 75.5) and MME (1700 vs. 1764).
-
Generality. The authors claim that the proposed prior can be combined effectively with existing token importance scores (L22–23, L267–268). However, they evaluate it only with CLSE. Since the benefit of and likely depends on how redundant the tokens favored by already are, results with other scores (e.g., attention-based ones) are needed to support this claim.
3. The efficiency claims are not well supported.
-
KV-cache. The authors claim that R2Pruner achieves the smallest KV-cache because its final token count is small (L460). However, KV-cache size depends on the token count summed over all layers, not on the final count alone. Based on the schedule in B.3, the visual KV-cache of R2Pruner at 64 tokens is estimated to be about 56 MiB, which is similar to or larger than that of FastV-style pruning (about 48–56 MiB) and larger than that of pre-LLM pruning (about 32 MiB). The measurement protocol and per-method values should be reported.
-
Output length. The measurement prompt (L448–449) asks for a single word or phrase, which yields very short outputs. Decoding-side gains from the reduced KV-cache are therefore not captured.
-
Attention implementation. All methods are measured with eager attention (L450). In practice, however, FlashAttention is widely used, and methods that do not require attention weights, such as [A] and [B], can benefit from it. R2Pruner needs attention weights to compute at every stage and thus appears incompatible with FlashAttention. Its practical efficiency relative to such methods may therefore be overestimated under the current setup. This limitation should be discussed.
References
[A] Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More, in EMNLP 2025.
[B] PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models, in CVPR 2025.
[C] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs, in NeurIPS 2025.
Questions for authors
-
Was the CLSE baseline evaluated using the same pruning layers and retention schedule as R2Pruner? Does it also use question-conditioned relevance at the first pruning stage?
-
The authors argue that raw dot-product similarity provides a "richer and more differentiated signal" than cosine similarity (L176–179). However, raw dot products are affected by feature magnitudes, so it is unclear whether captures local redundancy or is partly driven by token norms. Could the authors provide a comparison with cosine similarity?
-
R2Pruner selects the top-N tokens based on per-token scores (Eq. (9)). Since all tokens in a redundant region receive similarly low scores, they may be removed together, discarding the region entirely. Prior methods avoid this by keeping a representative token for each group of similar tokens or by selecting a diverse subset [A, B, C]. R2Pruner does not appear to include such a mechanism. How do the authors ensure that such regions are not discarded entirely?
-
VisPruner is cited as Zhang et al. (2025) in the text (L126, L353–354) but as Fan et al. (2025) in Table 1. Which method was actually compared?
-
How is the token importance score (the spectral evolution score of CLSE) defined?