acceptodds
Withdrawn from ICLR 2027

ALIGN AND BRIDGE: VISUAL TOKEN PRUNING IN THE PRESENCE OF A MODALITY GAP

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Vision-language models (VLMs) achieve strong performance across multimodal tasks, but their numerous visual tokens incur substantial computational and memory costs. Pruning visual tokens before the LLM decoder reduces the input sequence length, yet identifying query-relevant tokens remains challenging in the presence of a modality gap and limited cross-modal interaction. We propose Align and Bridge, a framework that learns query-conditioned relevance before the decoder and propagates it throughout progressive token pruning. Its nonlinear Vision-Text Aligner (VTA) is trained solely on pre-decoder visual and textual embeddings without executing the LLM decoder. Score-dispersion regularization encourages clearer separation between query-relevant regions and background. Within the decoder, Prior-aware Diverse Relevance (PDR) progressively prunes visual tokens by combining text-to-vision attention with pre-decoder importance priors and token diversity. This design connects early token selection with subsequent pruning to preserve relevant visual information while suppressing redundancy. Experiments across multiple VLMs and multimodal benchmarks show improved performance retention on most evaluated backbones, with substantial reductions in visual tokens and FLOPs; on Qwen2.5-VL, a learned baseline remains stronger. On Qwen3-VL-8B, at a nominal budget of 22.2% of visual tokens averaged across decoder layers, our method preserves 93.4% of the original QA performance and improves relative visual grounding performance by about 12.0 percentage points over the strongest grounding baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Visual Token Pruning, Cross-Modal Alignment, Query-Aware Token Selection, Efficient Multimodal Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ho9ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Align and Bridge, a visual-token pruning framework for VLMs. It consists of two components: VTA, which learns query-conditioned relevance from pre-decoder visual and textual embeddings without running the LLM decoder, and PDR, which progressively prunes visual tokens inside the decoder by combining the VTA prior, text-to-vision attention, and diversity-aware selection. The method is evaluated across multiple VLM backbones and multimodal benchmarks. It shows particularly strong grounding retention under aggressive pruning on LLaVA-1.5, LLaVA-NeXT, and Qwen3-VL-8B, while being less competitive on Qwen2.5-VL. The paper also provides extensive ablations and efficiency measurements.

Critical strengths
  1. Strong preservation of query-relevant visual evidence. The main empirical strength is the substantial improvement on grounding-sensitive tasks. For example, on Qwen3-VL-8B at the nominal 22.2% budget, the method preserves 87.78% relative grounding performance, compared with 71.04% for LearnPruner, while maintaining similar relative QA performance. This suggests that the method is effective at avoiding the loss of localized visual evidence under aggressive pruning.
  2. Broad and transparent evaluation. The paper evaluates multiple backbones, budgets, task types, and efficiency metrics, and includes extensive ablations of the aligner, prior fusion, diversity term, and scoring strategy. It also reports negative results, including weaker performance on Qwen2.5-VL and mixed contributions from several VTA components. This makes the empirical analysis relatively convincing and transparent.
Critical weaknesses
  1. The full-system gains are not cleanly attributable to the proposed “Align and Bridge” mechanisms. At matched per-sample token counts on Qwen3-VL-8B, VTA does not significantly improve RefCOCOg [email protected] over saliency or raw cosine, with gains mainly appearing in mean IoU. Moreover, encoder-only VTA is weaker than saliency on Qwen2-VL and Qwen2.5-VL, and on Qwen2.5-VL, diversity-only pruning outperforms the full PDR configuration across all reported metrics.These results make it difficult to determine whether the large full-pipeline gains primarily come from VTA and prior propagation, or from adaptive budgeting, progressive pruning, diversity selection, and backbone-specific schedules.
  2. The practical efficiency advantage is limited. Despite substantial theoretical FLOP reductions, wall-clock acceleration is modest. On Qwen3-VL-8B, the reported end-to-end speedup is 0.95$\times$, i.e., slightly slower than dense inference. In addition, selector overhead is excluded from the reported FLOP ratios, and the main tables compare nominal rather than strictly matched compute budgets. Therefore, the method demonstrates a favorable accuracy/grounding trade-off on several backbones, but its practical inference-efficiency advantage is less clear.
Questions for authors
  1. Contribution attribution. Could the authors isolate the effect of adaptive encoder budgeting from VTA and PDR, for example by comparing fixed-budget and adaptive-budget variants under the same full pipeline?
  2. Robustness of prior propagation. Why does the encoder prior help on LLaVA backbones but degrade performance relative to diversity-only pruning on Qwen2.5-VL? Clarifying whether this stems from the fixed γℓ\gamma_\ell schedule, backbone architecture, or prior quality would materially affect the assessment of the proposed “Bridge” mechanism.
  3. Matched-compute comparison. Could the authors report comparisons against the strongest relevant baselines at matched measured token retention or decoder FLOPs rather than nominal budgets? This would make the practical accuracy-efficiency trade-off substantially clearer.
Reviewer_NaTLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses visual token pruning in VLMs, where identifying the visual tokens relevant to the query before the LLM decoder is challenging due to the modality gap. To this end, it proposes a Vision-Text Aligner (VTA), a residual nonlinear aligner trained on the features before the decoder with a bidirectional contrastive loss and a score dispersion term. It also proposes Prior-aware Diverse Relevance (PDR) that prunes visual tokens progressively within the decoder, combining the attention from text to visual tokens with both the VTA prior and token diversity.

Critical strengths
  • S1. To avoid backpropagation through the LLM decoder, VTA is trained only on the visual and text features before the decoder, reducing the training cost compared with LearnPruner.

  • S2. PDR then reuses the relevance learned by VTA as a prior for pruning within the decoder, connecting the two pruning stages.

Critical weaknesses

W1. Limited comparison with existing methods.

  • Training-free methods already select the visual tokens relevant to the query before the decoder, such as CDPruner [A], MMTok [B], and VisionTrim [C]. This paper provides no comparison with them. It is thus unclear whether VTA and PDR have an advantage over these methods, since VTA requires additional training without clearly better performance.
  • The method improves the performance on visual grounding considerably, while it is not compared with Nüwa [D], which addresses visual grounding under token pruning. Nüwa shows that the grounding performance depends heavily on the position IDs of the retained tokens. This paper assigns the position IDs differently for LLaVA and Qwen, but provides no analysis of this.

[A] Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs, NeurIPS 2025
[B] MMTok: Multimodal Coverage Maximization for Efficient Inference of VLMs, ICLR 2026
[C] VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration, ICLR 2026
[D] Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning, ICLR 2026


W2. It is unclear how training on image–query pairs within a batch yields the token relevance concentrated on the queried region.

  • Apart from the dispersion term, VTA is trained only to match each image with its query against the other pairs in the batch. The inference, however, ranks individual visual tokens by Eq. 9. This objective may be satisfied by any region that distinguishes the image from the others in the batch, not necessarily by the queried region.
  • In Eq. 6, every query token, including function words and punctuation such as “a”, “the”, and “,”, classifies the matching image with equal weight. Since the text embeddings are taken before the decoder and the aligner acts on each token independently, such a token has the same embedding in every query, making its targets contradictory within a batch. The same tokens also enter the max over the query tokens in Eq. 9, adding a component to the relevance that does not depend on the query.
Questions for authors

Please see weaknesses.

Reviewer_soe5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper trains an aligner to pick query-relevant visual tokens before the decoder, then uses attention and diversity to keep useful tokens through later pruning; it tests the method on several VLMs and vision-language benchmarks.

Critical strengths

At the nominal 22.2% budget on Qwen3-VL-8B, RefCOCOg accuracy is 77.46%, versus 63.89% for LearnPruner. That is a meaningful gain for grounding.

VTA finds more ground-truth regions than raw cosine similarity on LLaVA-1.5, which gives some support to the selection idea.

Critical weaknesses
  1. When token counts are matched on Qwen3-VL-8B, VTA does not significantly improve final grounding accuracy over simpler scores.

  2. LearnPruner beats the full method at all three tested budgets on Qwen2.5-VL.

  3. On Qwen3-VL-8B, end-to-end inference is slightly slower than running the dense model.

  4. The token score takes the best match to any single query word. That is a simple way to rank patches, but it does not directly handle relationships such as “the person to the left of the car.”

  5. The pruning recipe needs backbone-specific settings. The layer schedules and mixing weights are tuned separately, so it is unclear how well the method would transfer to a new VLM without more tuning.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.