acceptodds
Withdrawn from ICLR 2027

Reroute, Don’t Remove: Recoverable Visual Token Routing for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Many decoder-side visual-token pruning methods follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. We show that this irreversible action can be fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding- sensitive queries. We propose Reroute, a training-free, scorer-preserving plug-in that replaces removal with recoverable routing. At each routing stage, selected visual tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. By preserving the underlying ranking signal and modifying only the post-selection action, Reroute enables a controlled study of irreversible visual-token pruning. Reroute inherits existing attention-score ranking rules within a stage-wise routing framework, largely preserving the TFLOPs and KV-cache efficiency of the pruning method it augments while retaining most of the corresponding speedup. Across LLaVA-1.5 and Qwen backbones, Reroute improves matched-budget grounding performance across decoder-side visual-token pruning settings while maintaining general VQA performance under aggressive token reduction. These results suggest that VLM token reduction should not be viewed only as irreversible pruning, but also as recoverable routing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models (VLMs), Visual Token Pruning, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2TSkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reroute, a training-free, recoverable visual token routing method that addresses the loss of important visual information caused by irreversible token removal in existing visual token pruning methods. Reroute retains the importance-scoring mechanisms of existing pruning methods but replaces permanent token removal with stage-wise routing: unselected visual tokens bypass the current computation stage and re-enter the candidate pool at later stages. Experiments on multiple VLM backbones, including LLaVA-1.5, Qwen2.5-VL, and Qwen3.5, show that Reroute improves general VQA and visual grounding performance under aggressive token reduction in most evaluated settings, while preserving most of the computational efficiency gains.

Critical strengths

S1. This paper presents a simple and practically useful approach by addressing the irreversibility of existing visual token pruning methods. Reroute requires no additional training or redesign of the importance scorer and can be directly integrated with existing pruning methods, making it easy to implement.

S2. Reroute can substantially improve visual grounding performance under certain aggressive token reduction settings while retaining most of the computational efficiency benefits.

Critical weaknesses

W1. Limited methodological novelty. Although the authors discuss the relationship between Reroute and MoD in Section 4.4, their core computational mechanisms are largely similar. Reroute mainly replaces the learnable router with importance scores from existing pruning methods and adopts or modifies their stage-wise schedules. However, these modifications appear to be straightforward combinations and adaptations of existing techniques rather than genuinely novel algorithmic designs.

W2. The central hypothesis of this paper is that visual token importance changes across decoder depth, causing early pruning decisions to mistakenly remove tokens that may become important in later layers. However, Figure 2 mainly presents a few qualitative examples, without systematic quantitative analysis of cross-layer importance changes.

W3. We observe that FastV and PDrop exhibit nearly collapsed visual grounding performance under aggressive token reduction, while Reroute is primarily evaluated against these two methods and demonstrates substantial improvements. However, for Nüwa, which already achieves relatively strong grounding performance, the additional gains from Reroute in Table 1(a) are very limited, with slight performance degradation in some settings. Furthermore, although Table 12 includes results from other methods, it only reports the integration of Reroute with Nüwa, lacking systematic evaluation across a broader range of token reduction strategies.

W4. To improve the clarity and comparability of the experimental results, we suggest using consistent RefCOCO evaluation splits throughout the paper.

W5. Appendix Table 18 shows that Reroute degrades visual grounding performance on Qwen3-VL-8B at both 66.7% and 77.8% token reduction, with substantial drops in some cases. The explanation in Appendix C.3 remains largely speculative without supporting experimental evidence, leaving the applicability of Reroute across different model architectures and token reduction levels unclear.

Questions for authors

Please also address the concerns raised in W1–W5.

Q1. Existing visual token pruning methods handle low-importance tokens through either token dropping or token merging. In merging-based methods, information from compressed tokens is already partially preserved in the remaining tokens. Is Reroute still applicable to such methods? From this perspective, the rank-and-remove paradigm does not fully characterize existing pruning methods. Since both Reroute and token merging aim to preserve information from compressed tokens, what advantages does Reroute offer over token merging? We encourage the authors to provide further discussion or experimental comparisons.

Reviewer_wLcNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Reroute introduces a training-free visual-token routing mechanism that replaces irreversible pruning with recoverable token bypass and re-entry, while preserving existing attention-based scoring methods. Experiments across LLaVA-1.5 and Qwen-family models demonstrate improved visual grounding under matched token budgets, while largely maintaining general VQA performance and computational efficiency.

Critical strengths

Reroute can be integrated into existing decoder-side pruning methods without additional training or modifications to their scoring mechanisms, making it a practical and flexible approach.

Critical weaknesses

W1.

Reroute largely reuses existing conditional computation and token reactivation ideas, without direct comparisons to closely related approaches (e.g., Ma et al., 2026a), making its distinct contribution unclear.

  • Jie Ma, Zhike Qiu, Jiayi Ji, Xiaoshuai Sun, and Rongrong Ji. Look less, reason more: Block-wise attention skipping for efficient multimodal llms. arXiv preprint arXiv:2606.08511, 2026b.

W2.

The evaluation lacks comparisons with random re-entry and quantitative analysis of whether re-admitted tokens recover task relevant visual evidence. it remains unclear whether the improvements stem from meaningful token recovery or simply from periodic token reselection.

Questions for authors

see weakness

Reviewer_q9XKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Reroute, a training-free framework for visual token reduction in vision-language models. Different from permanently removing low-scored visual tokens, Reroute preserves them through a bypass path and allows them to be selected again at subsequent routing stages. The method reuses existing attention-based scoring rules and is evaluated across different backbones. Extensive experiments verify the effectiveness of Reroute.

Critical strengths
  1. Replacing irreversible deletion with recoverable deferral is interesting and the motivation is clear.
  2. Experiments verify that recoverable routing can improve performance across multiple token budgets. The PDrop comparisons are particularly informative because the routing stages can be aligned between pruning and rerouting.
  3. Efficiency comparisons are conducted with the on-device latency, which is important for practical application.
Critical weaknesses
  1. It would be better to conduct more comparisons with other methods, e.g., HoloV (NeurIPS 2025), to fully verify the effectiveness of the proposed method.
  2. The method relies on the attention scores, which is incompatible with efficient attention implementation, e.g., FlashAttention. The resulting overhead may offset the savings from token reduction.
  3. The improvements in general VQA over the PDrop baseline are relatively minor. For example, at 128 tokens, the aggregate Avg. Ratio improves only from 96.3% to 96.8%, and at 192 tokens, the performance degrades from 98.9% to 98.6%.
Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.