- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Output-Aligned Routing (OAR), a visual token pruning method for efficient vision-language models (VLMs). The authors identify proxy metric misalignment, arguing that existing attention- or diversity-based pruning methods may fail to preserve visual information that is jointly necessary for generation. To address this, OAR evaluates candidate visual token subsets using the KL divergence between the output distributions of a frozen VLM with full and pruned visual contexts. These offline evaluations are distilled into soft inclusion targets to train a lightweight, query- and position-aware router with 1.91M parameters. At inference time, the router selects the top-B visual tokens before LLM prefill with minimal overhead. The method is primarily evaluated on Qwen2.5-VL-7B, with in-domain router calibration using unlabeled image-question pairs for several benchmarks, alongside an additional evaluation on InternVL3-8B. Across CharQA, DocVQA, AI2D, and TextVQA, the authors report improved accuracy-efficiency trade-offs over the evaluated pruning baselines, achieving up to 2.94x end-to-end speedup and a 4.84x reduction in KV-cache memory. The paper also presents ablation studies, cross-domain transfer experiments, and evaluations of hallucination and visual dependency.
Critical strengths
-
Clear motivation and a reasonable approach to token selection. The paper addresses a meaningful limitation of existing visual token pruning methods, which overlook important relationships between visual tokens. Using changes in the model's output to guide token selection is an interesting direction, and the ablation results provide some evidence that this supervision is more effective than conventional proxy-based targets.
-
Efficient inference design. The separation of offline supervision generation and online token selection is a practical design choice. By using a lightweight router to select visual tokens before LLM prefill, the method reduces inference cost without modifying the base VLM. The reported results suggest that this approach can achieve a reasonable balance between efficiency and performance.
Critical weaknesses
1. Dependence on benchmark-specific training and limited cross-domain generalization. OAR trains routers using image-question pairs from the training splits of several target benchmarks. While this may be reasonable for domain-specific applications, it assumes access to representative target-domain data before deployment, which may not be practical for general-purpose VLMs. Figure 6 shows considerable performance drops across several related datasets. For example, retained quality on DocVQA drops from 75.7% to 56.5%, and on InfographicVQA from 66.7% to 49.3%, when using a TextVQA-trained router. Although some transferred routers still outperform training-free baselines, these results suggest a considerable dependence on the training distribution. Since the main results mostly rely on benchmark-specific calibration, the broader applicability of OAR remains insufficiently demonstrated.
2. Missing learning-based comparisons and unclear evaluation settings. The paper mainly compares OAR with training-free methods, despite requiring additional router training. Learning-based methods such as LearnPruner [1] are not included, making is difficult to assess the advantages of OAR over existing learned pruning methods. The router checkpoints used for AI2D, POPE, CHAIR, and MMStar are not clearly specified. In addition, the main ChartQA evaluation uses ANLS rather that the standard relaxed accuracy metric, making direct comparisons with prior work difficult.
[1] Takezoe, Rinyoichi, et al. "Learnpruner: Rethinking attention-based token pruning in vision language models." International Conference on Learning Representations. Vol. 2026. 2026.
Questions for authors
1. The paper describes offline target generation for ChartQA, DocVQA, InfographicVQA, and TextVQA, but the router checkpoints used for AI2D, POPE, CHAIR, and MMStar are not clearly specified. Could the authors clarify which routers were used for these benchmarks and whether any additional calibration was performed?
2. Why is ANLS used for ChartQA instead of the standard relaxed accuracy metric? Could the authors also report relaxed accuracy? Also, could the authors clarify whether the main results in Table 1 were evaluated on the 540 examples reported in Appendix A or on the full official test set of 2,500 examples? If the 540-example set was used, how was it constructed and why was it selected?
3. Figure 6 shows considerable performance degradation in several cross-domain settings, but the transfer experiments cover only three related datasets. Could the authors extend this evaluation to include ChartQA, both as a training source and an evaluation target? More importantly, could the authors report results using a single fixed Router across different benchmarks without training on the target benchmark or performing additional calibration? This would help clarify whether the reported improvements can be maintained beyond benchmark-specific training.
4. The comparisons mainly focus on training-free pruning methods, although OAR requires additional Router training. A comparison of learning-based methods such as LearnPruner using the same backbone and comparable inference budgets would help assess the advantages of OAR over existing learning-based pruning methods.
5. Most experiments are conducted on Qwen2.5-VL-7B, while the InternVL3-8B evaluation is limited to ChartQA at a single token budget. Could the authors provide additional results on InternVL3-8B using another benchmark? It would also be helpful to evaluate OAR on a more general visual understanding benchmark, such as GQA or MMBench, preferably without target-specific calibration. This would help assess the applicability of the method across different backbones and tasks.