acceptodds
Withdrawn from ICLR 2027

When Text Matters: Design Principles for Visual Token Pruning in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual token pruning has been widely studied as a practical approach to reducing the computational cost of large vision-language models. However, it struggles to preserve essential visual information, which can lead to substantial performance degradation. In particular, image-based token selection can overlook task-relevant details, while text-guided token selection may fail to capture the text–visual relationships needed for complex reasoning. We find that applying textual guidance too early can limit its ability to identify answer-relevant visual regions, whereas text-to-visual attention becomes more informative at intermediate decoder depths. This finding motivates our training-free method, which separates early vision-guided pruning from deferred text-guided reselection. We first prune visual tokens using vision-encoder attention, retain additional candidates until the decoder midpoint, and then use text-to-visual attention to determine the final visual-token set. Across eight benchmarks and three models, our method outperforms the best-performing baselines by an average of 11.10 and 16.84 percentage points in performance recovery at 80% and 90% pruning, respectively, with comparable or lower LLM-prefill latency than most baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Compression, Token Pruning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WSQbOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of existing visual token pruning methods in VLMs particularly their tendency to discard task-relevant visual information. The authors identify two major limitations: image-based pruning can overlook visually insignificant but task-relevant details, while early text-guided pruning may fail to capture complex relationships between textual instructions and visual evidence. Through a layer-wise analysis of text-to-visual attention, the authors demonstrate that its ability to identify question-relevant visual regions improves substantially at intermediate decoder layers.

Critical strengths
  1. This work provides a comprehensive analysis of which layer provides the most useful attention map for pruning guidance using dense-relative performance retention, which is a valuable contribution.
  2. The authors evaluate three VLM backbones, eight benchmarks, and three pruning ratios, comparing against five established baselines. The improvements are particularly impressive at high pruning ratios.
Critical weaknesses
  1. Although observing that the middle layer provides the best guidance for pruning is a valuable observation, the number of test samples is unclear, and the randomness of the answers is not addressed in this analysis.
  2. The most important contribution of this work is to evaluate which layer of the LLM provides the best attention map. However, this work uses one metric for analysis; more embedding or hidden-state analysis [1] would be helpful to support the conclusion.
  3. Although the motivation is interesting, the proposed method primarily combines two existing token-selection mechanisms: vision-encoder attention for early pruning and text-to-visual attention for subsequent pruning. The main contribution lies in the timing and combination of these strategies rather than a fundamentally new token-importance estimation mechanism.

[1] Skean, Oscar, et al. "Layer by layer: Uncovering hidden representations in language models." arXiv preprint arXiv:2502.02013 (2025).

Questions for authors
  1. The authors use a fixed reserve fraction of alpha=0.2. How sensitive is the optimal reserve fraction to image resolution, prompt complexity, or the number of relevant visual regions? Could an adaptive reserve size further reduce computational overhead?
Reviewer_uqgqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes the ability of text-to-visual attention to distinguish answer-relevant regions across decode depths and finds that the attention signal becomes much stronger at middle layers. Based on this observation, this paper introduce a two-stage training-free pruning methods that combines early image-based pruning with deferred text-guided selection. Experiments with multiple models and a variety of datasets show the effectiveness of the proposed approach.

Critical strengths
  1. The paper provides a detailed quantitative analysis of the reliability of text-to-vision attention, making the motivation for the proposed method convincing.

  2. The proposed method substantially outperforms the evaluated baselines, providing strong empirical evidence of its effectiveness.

Critical weaknesses
  1. This paper is motivated by the observation that the reliability of text-to-visual attention changes across LLM layers. However, this phenomenon has already been widely studied in prior work, including VScan (TMLR 2026) and LearnPruner (ICLR 2026). The proposed two-stage pruning framework is also highly similar to these earlier approaches. Therefore, it is unclear what new insight or improvement this paper offers beyond prior work.

  2. The second stage requires text-to-vision attention result, which may not be directly compatible with efficient implementations such as FlashAttention. Although the paper recomputes these weights from queries and keys, the additional computation and memory overhead should be reported separately to clarify their impact on practical inference efficiency.

Questions for authors

Please refer to Weaknesses 1 and 2.

Reviewer_zFc1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a training-free two-stage visual token pruning method for Vision-Language Models (VLMs). The key insight is that textual guidance applied too early in the decoder limits its ability to identify answer-relevant visual regions, while text-to-visual attention becomes more informative at intermediate decoder depths. Based on this observation, the method first prunes visual tokens using vision-encoder attention (preserving an expanded candidate set via a reserve fraction parameter), then at the decoder midpoint, reselects the final token set using text-to-visual attention. The approach is evaluated on Qwen3-VL-4B, Qwen3-VL-8B, and LLaVA-OV-1.5-8B across eight benchmarks, reporting competitive accuracy retention with reduced token counts and favorable latency compared to FastV, SparseVLM, VisPruner, ZOO-Prune, RESTORE, PyramidDrop, and FlowCut.

Critical strengths

The empirical analysis in Section 3.1 is genuinely insightful: the finding that text-to-visual attention's ability to rank answer-relevant regions peaks at intermediate decoder depths (rather than near the input) is a non-trivial observation that directly motivates the two-stage design, and the ablation in Table 6 quantifies the concrete gain from deferred reselection (e.g., +18.37pp on InfoVQA for Qwen3-VL-4B). The method is entirely training-free and requires no additional model optimization, making it broadly applicable across different VLM backbones with minimal engineering effort -- a practical advantage over training-based pruning methods.

Critical weaknesses
  1. My primary concern is the limited technical novelty of the core mechanism. The first stage relies on standard vision-encoder attention scores for token selection, and the second stage simply re-ranks candidates using text-to-visual attention at the midpoint -- both are straightforward attention-based signals without any novel architectural component, learnable module, or theoretical guarantee. The paper's contribution thus amounts to a specific scheduling strategy (when to apply text guidance) rather than a new pruning principle, which may be insufficient for a venue like ICLR that expects methodological innovation.

  2. Beyond this novelty concern, the method's performance gain over strong baselines is modest in absolute terms. As shown in Tables 2-4, while Ours achieves the best Avg. Rel. in most settings, the margin over ZOO-Prune and SparseVLM is often narrow (e.g., +1.57pp Avg. Rel. over ZOO-Prune at 70% pruning on Qwen3-VL-4B in Table 2), yet the two-stage design introduces additional computation (candidate reserve management and midpoint reselection) that these single-stage methods do not incur. The latency results in Table 5 show Ours is faster than most baselines at 80% pruning but slower than FastV, and at 90% pruning the gap narrows further -- raising questions about whether the accuracy benefit justifies the architectural complexity.

  3. Compounding these issues, the paper lacks comparison with training-free token merging approaches such as Token Merging (ToMe) and its VLM adaptations. Given that merging operates under similar no-training assumptions, it remains unclear whether the proposed two-stage scheduling is genuinely superior to well-tuned single-stage attention pruning or to merging-based alternatives.

Questions for authors
  1. How does the proposed method compare with training-free token merging approaches such as Token Merging (ToMe) and its VLM adaptations? Given that merging operates under similar no-training assumptions, could a fair comparison clarify whether attention-based reselection provides a meaningful advantage over merging-based redundancy removal?

  2. The reserve fraction parameter alpha controls the number of additional candidates preserved until midpoint reselection. How sensitive is the final performance to this hyperparameter across different pruning ratios (70%, 80%, 90%) and different VLM architectures? Does the optimal alpha vary significantly across models, and is there a principled way to set it without per-model tuning?

  3. The paper analyzes text-to-visual attention correlation with masking-based region importance, but does not ablate alternative text-guided signals (e.g., gradient-based importance, entropy-based uncertainty, or knowledge-distillation scores). Could such signals complement or substitute attention-based reselection, and at what computational cost?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.