- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of existing visual token pruning methods in VLMs particularly their tendency to discard task-relevant visual information. The authors identify two major limitations: image-based pruning can overlook visually insignificant but task-relevant details, while early text-guided pruning may fail to capture complex relationships between textual instructions and visual evidence. Through a layer-wise analysis of text-to-visual attention, the authors demonstrate that its ability to identify question-relevant visual regions improves substantially at intermediate decoder layers.
Critical strengths
- This work provides a comprehensive analysis of which layer provides the most useful attention map for pruning guidance using dense-relative performance retention, which is a valuable contribution.
- The authors evaluate three VLM backbones, eight benchmarks, and three pruning ratios, comparing against five established baselines. The improvements are particularly impressive at high pruning ratios.
Critical weaknesses
- Although observing that the middle layer provides the best guidance for pruning is a valuable observation, the number of test samples is unclear, and the randomness of the answers is not addressed in this analysis.
- The most important contribution of this work is to evaluate which layer of the LLM provides the best attention map. However, this work uses one metric for analysis; more embedding or hidden-state analysis [1] would be helpful to support the conclusion.
- Although the motivation is interesting, the proposed method primarily combines two existing token-selection mechanisms: vision-encoder attention for early pruning and text-to-visual attention for subsequent pruning. The main contribution lies in the timing and combination of these strategies rather than a fundamentally new token-importance estimation mechanism.
[1] Skean, Oscar, et al. "Layer by layer: Uncovering hidden representations in language models." arXiv preprint arXiv:2502.02013 (2025).
Questions for authors
- The authors use a fixed reserve fraction of alpha=0.2. How sensitive is the optimal reserve fraction to image resolution, prompt complexity, or the number of relevant visual regions? Could an adaptive reserve size further reduce computational overhead?