acceptodds
Withdrawn from ICLR 2027

Output-Aligned Visual Token Routing for Efficient Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual token pruning accelerates Vision-Language Models (VLMs), yet existing methods predominantly rely on intermediate proxies such as attention weights or feature diversity to rank tokens. Visual elements typically exhibit strong spatial and semantic interdependencies. Consequently, ranking tokens in isolation fails to preserve the joint context required for generation, a challenge we term proxy metric misalignment. We propose Output-Aligned Routing (OAR), which supervises visual token selection directly via the inherent generation behavior of a frozen base VLM. In the offline stage, the base model uses unlabeled calibration data to evaluate retained visual subsets through the output KL divergence between unpruned and pruned generation distributions. We distill these divergence signals into soft inclusion targets to train a 1.91M-parameter Router. At inference time, the Router selects and gathers the top- tokens in a single, query-aware pass prior to language-model prefill, requiring zero online intervention from the base model. Across demanding benchmarks spanning charts, documents, infographics, and scene text, OAR preserves 92–95% of full-model accuracy on ChartQA and TextVQA while retaining only 80 visual tokens, an approximately 85% reduction in the visual sequence. Crucially, it achieves up to a 2.94 end-to-end speedup and a 4.8 reduction in KV cache memory, with an online selection overhead of under 1 ms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, visual token pruning, multimodal model compression, inference acceleration, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vkB6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Output-Aligned Routing (OAR), a visual token pruning method for efficient vision-language models (VLMs). The authors identify proxy metric misalignment, arguing that existing attention- or diversity-based pruning methods may fail to preserve visual information that is jointly necessary for generation. To address this, OAR evaluates candidate visual token subsets using the KL divergence between the output distributions of a frozen VLM with full and pruned visual contexts. These offline evaluations are distilled into soft inclusion targets to train a lightweight, query- and position-aware router with 1.91M parameters. At inference time, the router selects the top-B visual tokens before LLM prefill with minimal overhead. The method is primarily evaluated on Qwen2.5-VL-7B, with in-domain router calibration using unlabeled image-question pairs for several benchmarks, alongside an additional evaluation on InternVL3-8B. Across CharQA, DocVQA, AI2D, and TextVQA, the authors report improved accuracy-efficiency trade-offs over the evaluated pruning baselines, achieving up to 2.94x end-to-end speedup and a 4.84x reduction in KV-cache memory. The paper also presents ablation studies, cross-domain transfer experiments, and evaluations of hallucination and visual dependency.

Critical strengths
  • Clear motivation and a reasonable approach to token selection. The paper addresses a meaningful limitation of existing visual token pruning methods, which overlook important relationships between visual tokens. Using changes in the model's output to guide token selection is an interesting direction, and the ablation results provide some evidence that this supervision is more effective than conventional proxy-based targets.

  • Efficient inference design. The separation of offline supervision generation and online token selection is a practical design choice. By using a lightweight router to select visual tokens before LLM prefill, the method reduces inference cost without modifying the base VLM. The reported results suggest that this approach can achieve a reasonable balance between efficiency and performance.

Critical weaknesses

1. Dependence on benchmark-specific training and limited cross-domain generalization. OAR trains routers using image-question pairs from the training splits of several target benchmarks. While this may be reasonable for domain-specific applications, it assumes access to representative target-domain data before deployment, which may not be practical for general-purpose VLMs. Figure 6 shows considerable performance drops across several related datasets. For example, retained quality on DocVQA drops from 75.7% to 56.5%, and on InfographicVQA from 66.7% to 49.3%, when using a TextVQA-trained router. Although some transferred routers still outperform training-free baselines, these results suggest a considerable dependence on the training distribution. Since the main results mostly rely on benchmark-specific calibration, the broader applicability of OAR remains insufficiently demonstrated.

2. Missing learning-based comparisons and unclear evaluation settings. The paper mainly compares OAR with training-free methods, despite requiring additional router training. Learning-based methods such as LearnPruner [1] are not included, making is difficult to assess the advantages of OAR over existing learned pruning methods. The router checkpoints used for AI2D, POPE, CHAIR, and MMStar are not clearly specified. In addition, the main ChartQA evaluation uses ANLS rather that the standard relaxed accuracy metric, making direct comparisons with prior work difficult.


[1] Takezoe, Rinyoichi, et al. "Learnpruner: Rethinking attention-based token pruning in vision language models." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors

1. The paper describes offline target generation for ChartQA, DocVQA, InfographicVQA, and TextVQA, but the router checkpoints used for AI2D, POPE, CHAIR, and MMStar are not clearly specified. Could the authors clarify which routers were used for these benchmarks and whether any additional calibration was performed?

2. Why is ANLS used for ChartQA instead of the standard relaxed accuracy metric? Could the authors also report relaxed accuracy? Also, could the authors clarify whether the main results in Table 1 were evaluated on the 540 examples reported in Appendix A or on the full official test set of 2,500 examples? If the 540-example set was used, how was it constructed and why was it selected?

3. Figure 6 shows considerable performance degradation in several cross-domain settings, but the transfer experiments cover only three related datasets. Could the authors extend this evaluation to include ChartQA, both as a training source and an evaluation target? More importantly, could the authors report results using a single fixed Router across different benchmarks without training on the target benchmark or performing additional calibration? This would help clarify whether the reported improvements can be maintained beyond benchmark-specific training.

4. The comparisons mainly focus on training-free pruning methods, although OAR requires additional Router training. A comparison of learning-based methods such as LearnPruner using the same backbone and comparable inference budgets would help assess the advantages of OAR over existing learning-based pruning methods.

5. Most experiments are conducted on Qwen2.5-VL-7B, while the InternVL3-8B evaluation is limited to ChartQA at a single token budget. Could the authors provide additional results on InternVL3-8B using another benchmark? It would also be helpful to evaluate OAR on a more general visual understanding benchmark, such as GQA or MMBench, preferably without target-specific calibration. This would help assess the applicability of the method across different backbones and tasks.

Reviewer_i446OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that existing visual token pruning methods rank tokens by proxies such as attention or feature diversity, which ignore the fact that some tokens are only useful together. It proposes OAR, which learns token selection directly from the model's output. Offline, a frozen VLM scores candidate token subsets by how much pruning changes its output distribution relative to the full input, and these scores are distilled into a small router (1.91M parameters) trained on unlabeled data from each dataset. Online, the router selects the tokens to keep in a single pass, with under 1 ms of overhead. On Qwen2.5-VL-7B and InternVL3-8B with 80 retained tokens, OAR remains strongest, and OAR reduces prefill allocation by 4.9x and KV cache memory by 4.8x on ChartQA.

Critical strengths

Controlled evidence that output-based supervision beats proxy-based supervision. With the router, data, budget and inference path fixed and only the training targets changed, output-based targets reach 0.846 ANLS on ChartQA at 80 tokens, versus 0.776 when imitating HAWK and 0.514 when imitating CDPruner (Table 3), and the advantage holds from 64 to 192 tokens (Table 12).

Large gains on structured tasks at negligible online cost. selection takes under 1 ms, needs no extra LLM pass and no attention maps.

Critical weaknesses

The comparison is unfair. OAR is trained on each benchmark's own training split, while all baselines are training-free, and no trainable selector is compared. Part of the gain may come from in-domain training alone.

Questions for authors

How to seperate the benefit of output-based targets from that of in-domain training ?

Reviewer_NitPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OAR, a learned visual token selection method for efficient VLM inference. It argues that existing pruning methods rely on proxy signals such as attention weights or feature diversity, which may not preserve the visual context the model actually needs for generation. In an offline stage, the frozen base VLM scores candidate token subsets by the KL divergence between the full-token and pruned-token next-token distributions along the full model's greedy trajectory, without using labels. These divergence scores are converted into soft per-token inclusion targets, which are distilled into a lightweight router. At inference, the router scores all visual tokens in a single pass and keeps the top-B before LLM prefill. The method is evaluated mainly on Qwen2.5-VL-7B on a set of text-heavy vision-language benchmarks.

Critical strengths
  • The idea is interesting and intuitive. Selecting visual tokens based on their effect on the model's output distribution, rather than on token diversity or attention, addresses the root problem of proxy-based pruning. The offline supervision is also label-free.
  • The gains are strong on the open-ended benchmarks, especially for chart and document tasks. The controlled target ablation, where the router is trained to imitate other methods' selections under the same setting, nicely shows that the output-divergence supervision itself is what drives the improvement.
  • Pruning happens before the LLM, so the savings are solid and deployment is straightforward, with sub-millisecond router overhead.
Critical weaknesses
  • The comparison against training-free baselines is a bit unfair. OAR's router is calibrated separately on each dataset's own training split, while all compared baselines are training-free and task-agnostic. Even though no labels are used, fitting the router in-domain is a significant advantage. The zero-shot transfer experiment in the appendix actually shows that the router tuned on one task does not generalize well to other tasks. A fairer setting would train a single router on a general data mixture and evaluate it across all benchmarks.

  • The motivation is inconsistent with the proposed method, which sets readers off-guard. Section 3.1 and Proposition 1 argue that ranking tokens independently fails because visual evidence is non-additive. However, at inference time the router does exactly this: it scores each token independently and keeps the top-B. The "joint subset evaluation" in the teacher is also quite coarse, consisting of a small grid of regions (e.g., 4×4), each evaluated with fixed anchors, plus only three complete proposals. Two of these proposals come from CDPruner and HAWK, which are the proxy-based methods the paper criticizes. Either the motivation should be reframed, or the paper should show evidence that the router actually preserves non-additive dependencies better than independent proxy scoring.

  • The evaluation benchmarks are limited. The main results cover only four chart and document benchmarks, plus POPE, CHAIR and MMStar. There are no video or multi-image evaluation. Also the main experiments use only Qwen2.5-VL-7B. InternVL3-8B is evaluated only on ChartQA against a small set of baseline. No newer models like the Qwen3-VL series or later, or larger models are evaluated. It's also unclear which router (trained on which dataset) is used for POPE, CHAIR and MMStar.

Questions for authors
  1. Can you report results with a single router trained on a general mixed dataset and evaluated across all benchmarks, including general VQA benchmarks? How does OAR compare to trained pruning methods such as LearnPruner or ATP-LLaVA?
  2. Which router is used for POPE, CHAIR and MMStar, and what data is it trained on?
  3. What image resolution and maximum visual-token settings are used for all methods? Can you explain why CDPruner's TextVQA result at 128 tokens is higher than in its original paper but collapses at 80 tokens, and why DART performs so poorly on DocVQA?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.