acceptodds
Under review as a conference paper at ICLR 2027

RecastPrune: Rethinking Token Selection and Information Retention in Visual Token Pruning

Abstract

Training-free visual token pruning reduces the number of visual tokens processed by multimodal large language models without updating model parameters. However, existing methods often rely on image-only importance signals that may be misaligned with the user query, greedily select tokens without revisiting earlier decisions, and permanently discard the information carried by unselected tokens. We propose RecastPrune, which aligns token importance with the user query, revises the initial token selection, and selectively fuses information from unselected tokens. RecastPrune first applies query-aware relevance calibration by combining CLS-to-patch attention with patch–query similarity from the frozen CLIP encoders. It then performs coverage-aware initialization followed by restricted local exchange. Finally, saliency-aware selective absorption fuses relevant, compatible information from unselected tokens into retained representatives. Across LLaVA-1.5 and LLaVA-NeXT, RecastPrune achieves the highest average accuracy relative to the full-token upper bound at every evaluated token budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language model, visual token pruning, training-free acceleration, token aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3pi3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RecastPrune, a training-free visual token compression method for vision-language models (VLMs). The method combines query-aware relevance calibration, coverage-aware token selection, restricted local exchange, and saliency-aware selective absorption to preserve task-relevant visual information under a fixed token budget. Experiments on three VLMs across a range of benchmarks show that RecastPrune achieves competitive performance under different token budgets.

Critical strengths
  1. The paper studies training-free visual token compression, an important direction for reducing the computational cost of VLM inference.
  2. The method is evaluated on multiple VLMs, token budgets, and diverse benchmarks. The paper also reports compression overhead in addition to downstream task performance.
Critical weaknesses
  1. Limited methodological novelty. The proposed method appears to combine and extend existing ideas, including query-aware token relevance, set-coverage-based selection, local token swapping, and token merging. The methodological advances over existing approaches, particularly SCOPE, are not sufficiently clear.
  2. Modest and inconsistent improvements over strong baselines. The reported performance gains of the proposed method are generally small. For example, on LLaVA-NeXT, RecastPrune improves average retention over SCOPE by only 0.2–0.4 percentage points, while on Qwen2-VL it does not consistently outperform.
  3. Insufficient evidence for the individual contributions of the proposed components. In Table 5, query-aware calibration improves NES from 0.1667 to 0.1800, whereas adding Restricted Local Exchange slightly reduces both OCRBench score and NES. After incorporating SASA, the full method achieves the highest OCRBench score, but NES decreases. The results do not clearly substantiate the claimed benefits of each component, particularly the effectiveness of Exchange and SASA.
Questions for authors
  1. The ablation experiments are conducted only on OCRBench with 64 visual tokens. Could the authors provide ablation results on more benchmarks and token budgets to demonstrate the robustness and generality of the proposed components?
  2. The proposed method achieves only modest performance gains over existing baselines while introducing additional compression overhead. How do the authors justify this trade-off, and under what circumstances does the proposed method provide sufficient benefits to outweigh the additional overhead?
Reviewer_h3vSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RecastPrune, a training-free method for visual token compression. Building on SCOPE’s coverage-based selection, it combines CLS attention with CLIP patch–query similarity to estimate relevance, selects representative tokens, refines the selected set through local exchange, and selectively merges relevant and similar unselected tokens into retained representatives. The output contains a fixed number of visual tokens. Experiments evaluate quality and efficiency on LLaVA-1.5, LLaVA-NeXT, and Qwen2-VL.

Critical strengths
  1. The method jointly considers query relevance, representative coverage, and information recovery from unselected tokens. Its motivation is clear, and it requires no parameter training.
  2. The evaluation includes multiple models and text-related tasks, and reports compression overhead, time to first token (TTFT), and peak GPU memory, helping assess the overall computational cost.
Critical weaknesses

Improvements over SCOPE are small in some standard settings. However, in the 64-token setting in Table 6, TTFT increases from 54.2 ms for SCOPE to 75.7 ms, while peak GPU memory increases from 14.35 GB to 14.82 GB. The TTFT speedup over the full model is only approximately 1.15×. It remains unclear whether RecastPrune retains an advantage under matched latency or memory budgets.

The main component ablation focuses on a single token budget on OCRBench. Adding local exchange does not improve the official benchmark score, and the final combination increases the score from SCOPE’s 295 to 297. Improvements in the coverage surrogate do not necessarily translate into better task performance, and the current results do not establish consistent benefits from each added operation.

The default formulation relies on CLS-to-patch attention and CLIP text–patch alignment, yet the paper reports experiments on Qwen2-VL. For visual encoders without the same CLS signal or CLIP feature space, the paper does not clearly explain how alternative saliency scores are obtained, how features are aligned, or whether an additional encoder is used. This limits the assessment of the method’s applicability and reproducibility.

Questions for authors
  1. Under matched actual latency or memory budgets, rather than matched token counts alone, does RecastPrune still offer a quality advantage over SCOPE?
  2. Could you evaluate the individual benefits of query calibration, local exchange, and fusion across more tasks and token budgets?
  3. How are CLS-based saliency and CLIP patch–query similarity obtained for Qwen2-VL? Are additional encoders, alternative scoring methods, or feature mappings used, and are their costs included in the reported results?
Reviewer_qgrwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RecastPrune addresses visual token pruning through a combination of query-aware selection, subset refinement, and information recovery. Building on saliency-coverage selection, it combines visual saliency with patch-query similarity to calibrate token relevance, refines the initial subset through restricted local exchanges, and merges selected compatible features from discarded tokens into retained representatives. This final step preserves additional information without increasing the retained token count. The training-free pipeline is evaluated on two LLaVA models and Qwen2-VL using general visual question-answering and text-rich benchmarks. The experiments report performance across token budgets, component ablations, and measurements of time to first token and memory usage.

Critical strengths
  1. The method addresses three relevant failure modes within one pipeline: query-irrelevant saliency, irreversible greedy choices, and complete loss of discarded information. The manuscript explicitly acknowledges that a local swap improves the refinement surrogate rather than the initializer's different criterion, which makes the optimization claim appropriately scoped.

  2. The results contain useful gains beyond the original LLaVA setting. At the tightest Qwen2-VL budget, average retention improves from 95.0% for VLM-Pruner to 95.8%. On text-rich LLaVA benchmarks at 64 tokens, average retention increases from 89.8% for SCOPE to 92.2%, including a DocVQA improvement from 23.2 to 24.9. The paper also reports the additional latency and memory costs instead of claiming that fewer tokens automatically reduce both.

Critical weaknesses
  1. The incremental benefit of the added mechanisms is not convincingly established. On LLaVA-NeXT, the gains over SCOPE are only 0.2-0.4 average-retention points. The only component ablation is on OCRBench at one budget: SCOPE scores 295, calibration 296, calibration plus exchange 295, and the full method 297; normalized edit similarity is highest before exchange and absorption (Table 5). This supports a small net improvement but does not show that local exchange, or the full combination, is consistently beneficial. With several interacting hyperparameters and an incompletely specified adapted-parameter variant, broader controlled ablations and uncertainty estimates are important to substantiate the additional methodological complexity.

  2. A useful accuracy-efficiency advantage over the direct baseline has not been demonstrated. At 64 tokens, Table 6 reports 75.7 ms TTFT for RecastPrune versus 54.2 ms for SCOPE, with memory increasing from 14.35 to 14.82 GB. The full model takes 86.7 ms, so the measured end-to-end saving is limited despite the large token reduction. This is acknowledged in the paper, but a fixed-token comparison is insufficient to show that the additional operations are preferable to simply allowing a cheaper selector more tokens. The lack of documented precision, batch size, software versions, and a Qwen-specific relevance implementation further limits reproducibility of this trade-off.

Questions for authors
  1. At representative budgets on both a LLaVA model and Qwen2-VL, what are the separate effects of calibration, exchange, and absorption, including the full method without exchange? Please provide paired uncertainty estimates for the small accuracy gains, rather than only a sequential ablation on OCRBench.

  2. At the same TTFT or memory budget, how does RecastPrune compare with SCOPE or VLM-Pruner allowed a larger retained-token count? Please include all relevance computation, selection, projection, and absorption costs, with a fully specified timing setup.

  3. How are CLS saliency and CLIP patch-query relevance constructed and aligned to Qwen2-VL tokens? Please also specify the adapted parameters marked with a dagger in Table 1 and the data used to choose them. These details determine whether the architecture-transfer and best-result claims are reproducible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.