acceptodds
Withdrawn from ICLR 2027

Beyond Token-wise Importance: Vision Token Pruning with Local Relational Redundancy

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Large Language Models (MLLMs) process hundreds or even thousands of visual tokens, resulting in substantial computational and memory overhead. Since these tokens contribute unequally to response generation, *vision token pruning* aims to reduce this overhead by retaining only the most informative tokens. Existing methods typically estimate token importance from individual attention scores, visual features, or token-level redundancy. In this paper, we propose worst-case local dissimilarity as a simple prior for token selection, where high similarity even to a token’s -th most similar visual token indicates strong redundancy. Based on this prior, we introduce Pruner, a training-free vision token pruning method that uses local representation structure to guide progressive token selection. Pruner applies the prior at both the token and neighborhood levels, using local dissimilarity as a multiplicative weight on token importance. We find that this prior is effective both on its own and in combination with existing token-importance signals. When combined with an existing spectral evolution score, Pruner retains 96.4% of the unpruned model's performance on average across eight multimodal benchmarks at an average visual-token budget of only 64 out of the original 576 tokens. This exceeds the previous state-of-the-art by 1.7 percentage points at the same token budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Vision Token Pruning, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bendOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes R2Pruner, a training-free method for pruning vision tokens inside the LLM. The authors argue that a token is redundant if it remains highly similar even to its K-th most similar token, and formulate this as a worst-case local dissimilarity prior. Based on this prior, they define a token-level score (DiD_i) and a neighborhood-level score (AiA_i), and multiply them with an existing token importance score and question-conditioned relevance at the first of three progressive pruning stages. The authors evaluate R2Pruner on LLaVA-1.5-7B and Qwen2-VL across multiple benchmarks and analyze its efficiency.

Critical strengths

1. Strong empirical results under aggressive pruning. R2Pruner achieves the best average performance among the compared training-free methods on both LLaVA-1.5-7B and Qwen2-VL, with larger margins at lower token budgets.

2. Training-free and lightweight. The proposed scores require no additional training or parameters and are computed from existing hidden states. Computing them adds little overhead relative to LLM prefill.

Critical weaknesses

1. Novelty over prior work seems limited. The idea that token importance alone is insufficient and that redundancy should also be considered has been explored in prior work [A, B, C]. In particular, [A] already prunes tokens by their redundancy inside the LLM. The proposed method differs from these works mainly in how redundancy is measured: it derives local dissimilarity scores from the K-th-neighbor similarity of LLM hidden states and aggregates them across neighboring tokens. However, the paper neither cites nor compares with these methods, so it remains unclear whether this approach offers an advantage. The contribution therefore appears incremental.

2. The contribution of the proposed components is not established.

  • Effect of DiD_i and AiA_i. In Table 2, the "w/o DiD_i, AiA_i" variant uses only the CLSE score EiE_i and the question-conditioned relevance RiR_i. At 64 tokens, it already matches or outperforms CLSE on all four benchmarks. On POPE and SEEDB, most of the gain over CLSE thus appears to come from RiR_i or the pruning setting rather than from DiD_i and AiA_i. Furthermore, adding DiD_i and AiA_i improves the results by only about 1% (relative) on average, so their benefit seems marginal.

  • Ablation. Table 2 does not include ablations that remove only DiD_i or only AiA_i from the full model; the ablations are reported only for variants without EiE_i. The contribution of each to the full model is therefore unclear. Even in the reported ablations, removing DiD_i gives nearly identical results (rows 2–3). In addition, some statements in the text are inconsistent with Table 2:

    • The text states that removing AiA_i causes a larger drop (L468), but MME rises from 1792 to 1801.
    • The text states that RiR_i alone yields the weakest results (L469), but using only DiD_i and AiA_i scores lower on POPE (74.8 vs. 75.5) and MME (1700 vs. 1764).
  • Generality. The authors claim that the proposed prior can be combined effectively with existing token importance scores (L22–23, L267–268). However, they evaluate it only with CLSE. Since the benefit of DiD_i and AiA_i likely depends on how redundant the tokens favored by EiE_i already are, results with other scores (e.g., attention-based ones) are needed to support this claim.

3. The efficiency claims are not well supported.

  • KV-cache. The authors claim that R2Pruner achieves the smallest KV-cache because its final token count is small (L460). However, KV-cache size depends on the token count summed over all layers, not on the final count alone. Based on the schedule in B.3, the visual KV-cache of R2Pruner at 64 tokens is estimated to be about 56 MiB, which is similar to or larger than that of FastV-style pruning (about 48–56 MiB) and larger than that of pre-LLM pruning (about 32 MiB). The measurement protocol and per-method values should be reported.

  • Output length. The measurement prompt (L448–449) asks for a single word or phrase, which yields very short outputs. Decoding-side gains from the reduced KV-cache are therefore not captured.

  • Attention implementation. All methods are measured with eager attention (L450). In practice, however, FlashAttention is widely used, and methods that do not require attention weights, such as [A] and [B], can benefit from it. R2Pruner needs attention weights to compute RiR_i at every stage and thus appears incompatible with FlashAttention. Its practical efficiency relative to such methods may therefore be overestimated under the current setup. This limitation should be discussed.

References

[A] Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More, in EMNLP 2025.
[B] PACT: Pruning and Clustering-Based Token Reduction for Faster Visual Language Models, in CVPR 2025.
[C] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs, in NeurIPS 2025.

Questions for authors
  • Was the CLSE baseline evaluated using the same pruning layers and retention schedule as R2Pruner? Does it also use question-conditioned relevance RiR_i at the first pruning stage?

  • The authors argue that raw dot-product similarity provides a "richer and more differentiated signal" than cosine similarity (L176–179). However, raw dot products are affected by feature magnitudes, so it is unclear whether DiD_i captures local redundancy or is partly driven by token norms. Could the authors provide a comparison with cosine similarity?

  • R2Pruner selects the top-N tokens based on per-token scores (Eq. (9)). Since all tokens in a redundant region receive similarly low scores, they may be removed together, discarding the region entirely. Prior methods avoid this by keeping a representative token for each group of similar tokens or by selecting a diverse subset [A, B, C]. R2Pruner does not appear to include such a mechanism. How do the authors ensure that such regions are not discarded entirely?

  • VisPruner is cited as Zhang et al. (2025) in the text (L126, L353–354) but as Fan et al. (2025) in Table 1. Which method was actually compared?

  • How is the token importance score (the spectral evolution score of CLSE) defined?

Reviewer_iJc2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes R2Pruner, a training-free vision-token pruning method that adds a redundancy signal to existing token-importance criteria. The signal is a genuine, cheap and well-motivated complement to importance-based pruning.

Critical strengths
  1. The observation that a high-importance token may be locally replaceable is the right criticism of attention/importance-only selection, and the multiplicative instantiation is a clean way to express it. Besides, this is a training-free and drop-in weight on an existing score.
  2. The paper makes sufficient evaluations. The gains are consistent across both backbones and across the token-budget axis.
Critical weaknesses
  1. The method is a K-th-nearest-neighbour min-similarity statistic multiplied onto an existing score, on top of an existing progressive-pruning pipeline and an existing question-relevance signal. The “worst-case local dissimilarity prior” is largely a re-description of the maximum-similarity redundancy criterion already used by merging/diversity methods, restricted to a K-NN set.
  2. The ablation itself is internally consistent, but it also shows that the redundancy factors contribute modestly on top of the importance criterion.
  3. The paper does not report the average retained token counts per method, so the reader cannot check that the baselines were run at a matched final budget rather than a matched nominal one.
Questions for authors
  1. Both D_i and A_i are used only at the initial pruning stage. Given the layer-wise evolution you show in Fig. 4, have you tested recomputing local redundancy at later stages, and does it help?
  2. Your efficiency protocol fixes the prompt to “Describe this image. Answer with a single word or phrase.” for all methods. Does the ranking of methods change under question-conditioned prompts, given that R_i is question-dependent?
Reviewer_R6B2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To reduce the computational cost of multimodal large language models, this manuscript proposes R2Pruner, a training-free vision token pruning method that considers local relational redundancy in addition to token importance. The method constructs token-level and neighborhood-level dissimilarity scores and combines them with an existing importance criterion and question-conditioned relevance for progressive token selection. The manuscript is clearly written, and the perspective is interesting. However, the empirical observations do not sufficiently establish the proposed interpretation of redundancy, and several key methodological choices require stronger motivation and validation.

Critical strengths

(1) The manuscript is well-written and easy to follow.

(2) Considering neighborhood-level redundancy alongside token importance is an interesting perspective for vision token pruning.

(3) The reported results show promising performance-efficiency trade-offs. On LLaVA-1.5-7B, the method retains 96.4% of the unpruned model’s average normalized performance at an average visual-token budget of 64 out of 576 tokens.

Critical weaknesses

(1) The empirical observations supporting the central hypothesis are not sufficiently convincing. Figure 3 shows two examples where the selected tokens cover more foreground regions, but better foreground coverage does not itself establish that the discarded information is replaceable. Similarly, Figure 4 illustrates question-dependent attention rather than directly validating local redundancy. It would be beneficial to quantitatively examine whether tokens assigned higher redundancy actually have a smaller effect on predictions when removed, while controlling for their importance and question relevance.

(2) The connection between local similarity and information preservation after pruning is not clearly established. The neighborhoods used to compute DiDi​ and AiAi​ are defined before the first pruning stage, whereas selection at that stage uses fixed per-token scores. A token may therefore be downweighted because it has similar neighbors even if those neighbors are also discarded. How does the proposed selection preserve representative information from such neighborhoods? An analysis of neighborhood coverage after pruning would help clarify whether the scores measure actual replaceability within the retained token set.

(3) Several design choices are insufficiently motivated. In Section 3.1, raw dot-product similarity is preferred because it preserves feature magnitude, but it is unclear why magnitude should indicate redundancy rather than confound its estimation. The advantages of using the KK-th largest similarity, averaging neighboring dissimilarity scores, and multiplying all four signals in Eq. (8) are also not convincingly established. Controlled comparisons with cosine similarity, simpler neighborhood statistics, and additive score fusion would help distinguish the proposed rationale from a useful but largely heuristic combination.

Questions for authors

same as weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.