- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the gap between vision-language models' (VLMs) ability to recognize whether a question is answerable and their tendency to spontaneously abstain from answering unanswerable questions. To study and address this gap, the authors introduce a new benchmark, VAD-R, and a visual question answering (VQA) framework, Rep2Act.
VAD-R is a VQA benchmark comprising both answerable and unanswerable questions, accompanied by reasoning annotations and six categories of missing visual evidence. Experimental results show that VLMs rarely abstain from answering unanswerable questions, despite achieving high recall when explicitly assessing the answerability of image-question pairs.
To address this discrepancy, the authors propose Rep2Act, which employs a classification head to predict answerability from the model's hidden states. The framework is trained using two objectives: (1) a signed margin loss that encourages separation between the vocabulary logits for answerable and unanswerable decisions, and (2) a route-contrastive loss that encourages consistency between the selected routing decision and the model's final responses.
Experiments with Qwen2.5-VL-3B and Qwen2.5-VL-7B demonstrate substantial improvements in accuracy on VAD-R, from 56.67% to 86.33% and from 59.33% to 88.67%, respectively. The proposed method also shows generalization to other benchmarks while largely preserving performance on standard VQA tasks.
Critical strengths
-
A benchmark for diagnosing the representation–answer gap. The authors introduce VAD-R, a benchmark designed to systematically evaluate the discrepancy between VLMs' ability to recognize answerability and their actual abstention behavior.
-
Substantial empirical improvements. The proposed method achieves approximately 30 percentage-point improvements in accuracy on VAD-R, supported by experiments on two model scales.
Critical weaknesses
-
Limited methodological novelty. The central motivation and technical approach of this work substantially overlap with existing research. For example, Zhao et al. [1] demonstrate that token logit distributions contain information about VLMs' uncertainty and can be used to identify unanswerable visual questions. Similarly, Yao et al. [2] investigate the use of latent representation probes for abstention. Given these closely related findings, the novelty of Rep2Act is not sufficiently established, and its methodological contributions relative to prior work remain unclear.
-
Insufficient justification for the route-contrastive loss. The paper does not adequately establish why the route-contrastive loss is necessary. Rather than training the VLM backbone to align its responses with the classification head's predictions, a simpler alternative would be to use the classification head as a hard router that directly determines whether the model should answer or abstain, or to train the VLM to explicitly predict answerability. The results in Table 3(a) raise questions about the benefit of the proposed routing mechanism. For the 3B model, VLM-only routing achieves 86.33% accuracy; adding the classification head provides only a 1% improvement. For the 7B model, both approaches achieve the same accuracy of 88.33%.
Questions for authors
-
How does Rep2Act differ from existing abstention methods? Prior studies, including Zhao et al. [1] and Yao et al. [2], have already demonstrated that internal model representations or token distributions contain useful signals for detecting uncertainty and supporting abstention. Could the authors more clearly state the methodological novelty of Rep2Act and explain how it advances beyond these existing approaches?
-
Does training the VLM to align with the router improve abstention performance? Could the authors provide an ablation in which the classification head is used solely as a hard abstention router, without training the VLM to align its responses with the router's predictions? Such a comparison would help establish whether the route-contrastive loss provides meaningful benefits beyond those achievable through direct routing.
-
How were the chain-of-thought (CoT) annotations and training data generated and validated? Could the authors provide the exact prompts used with the teacher model to generate the CoT annotations and training data? Additionally, how did the authors verify the correctness and reliability of the generated reasoning steps?
[1] Zhao, Q., Xu, M., Gupta, K., Asthana, A., Zheng, L., and Gould, S. "The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?" European Conference on Computer Vision (ECCV), 2024. https://arxiv.org/abs/2403.09037
[2] Yao, J., Kulshrestha, A., Rauschmayr, N., Roberts, R., Zhu, B., Tsvetkov, Y., and Tombari, F. "Reading Between the Lines: Abstaining from VLM-Generated OCR Errors via Latent Representation Probes." arXiv preprint arXiv:2511.19806, 2025. https://arxiv.org/abs/2511.19806