acceptodds
Withdrawn from ICLR 2027

From Knowing to Abstaining: Bridging the Representation–Action Gap in Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The ability of VLMs to correctly abstain from answering unanswerable questions is as important as their ability to generate accurate answers to answerable ones. Recently, several benchmarks have emerged to evaluate and improve VLM abstention. However, they suffer from substantial limitations. First, their samples often contain unintended shortcut cues in either the images or questions that inadvertently reveal answerability; moreover, they typically provide an explicit “unanswerable” option, preventing an accurate assessment of whether VLMs can abstain spontaneously. Second, when used as training data, they generally provide only binary labels without fine-grained explanations to support deeper supervision. To address these limitations, we introduce Visual Answerability Diagnosis with Rationales (VAD-R), a new benchmark constructed through a two-stage pipeline of shortcut filtering and quality verification to prevent answerability leakage. Each example is annotated with step-by-step rationales and causal evidence-gap labels. Evaluation of state-of-the-art open- and closed-source VLMs on VAD-R reveals strikingly limited spontaneous abstention capabilities, with the two model groups achieving average recall rates of only 11.4% and 16.3%, respectively. We then conduct probing analyses, revealing that hidden-state representations in certain layers can effectively distinguish answerability, yet this internal distinction fails to manifest in VLMs’ final responses. Motivated by this observation, we introduce Rep2Act, a representation-to-action alignment method that translates latent answerability awareness into explicit abstention decisions. Rep2Act substantially improves action accuracy on VAD-R from 56.67% to 86.33% for Qwen2.5-VL-3B and from 59.33% to 88.67% for Qwen2.5-VL-7B. On the out-of-distribution TUBench, Rep2Act achieves an average F1 score of 53.3% with only a 3B model, surpassing the closed-source GPT-4 Turbo and GPT-4o by 16.2% and 1.1%, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM, abstain, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LEwQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the gap between vision-language models' (VLMs) ability to recognize whether a question is answerable and their tendency to spontaneously abstain from answering unanswerable questions. To study and address this gap, the authors introduce a new benchmark, VAD-R, and a visual question answering (VQA) framework, Rep2Act.

VAD-R is a VQA benchmark comprising both answerable and unanswerable questions, accompanied by reasoning annotations and six categories of missing visual evidence. Experimental results show that VLMs rarely abstain from answering unanswerable questions, despite achieving high recall when explicitly assessing the answerability of image-question pairs.

To address this discrepancy, the authors propose Rep2Act, which employs a classification head to predict answerability from the model's hidden states. The framework is trained using two objectives: (1) a signed margin loss that encourages separation between the vocabulary logits for answerable and unanswerable decisions, and (2) a route-contrastive loss that encourages consistency between the selected routing decision and the model's final responses.

Experiments with Qwen2.5-VL-3B and Qwen2.5-VL-7B demonstrate substantial improvements in accuracy on VAD-R, from 56.67% to 86.33% and from 59.33% to 88.67%, respectively. The proposed method also shows generalization to other benchmarks while largely preserving performance on standard VQA tasks.

Critical strengths
  1. A benchmark for diagnosing the representation–answer gap. The authors introduce VAD-R, a benchmark designed to systematically evaluate the discrepancy between VLMs' ability to recognize answerability and their actual abstention behavior.

  2. Substantial empirical improvements. The proposed method achieves approximately 30 percentage-point improvements in accuracy on VAD-R, supported by experiments on two model scales.

Critical weaknesses
  1. Limited methodological novelty. The central motivation and technical approach of this work substantially overlap with existing research. For example, Zhao et al. [1] demonstrate that token logit distributions contain information about VLMs' uncertainty and can be used to identify unanswerable visual questions. Similarly, Yao et al. [2] investigate the use of latent representation probes for abstention. Given these closely related findings, the novelty of Rep2Act is not sufficiently established, and its methodological contributions relative to prior work remain unclear.

  2. Insufficient justification for the route-contrastive loss. The paper does not adequately establish why the route-contrastive loss is necessary. Rather than training the VLM backbone to align its responses with the classification head's predictions, a simpler alternative would be to use the classification head as a hard router that directly determines whether the model should answer or abstain, or to train the VLM to explicitly predict answerability. The results in Table 3(a) raise questions about the benefit of the proposed routing mechanism. For the 3B model, VLM-only routing achieves 86.33% accuracy; adding the classification head provides only a 1% improvement. For the 7B model, both approaches achieve the same accuracy of 88.33%.

Questions for authors
  1. How does Rep2Act differ from existing abstention methods? Prior studies, including Zhao et al. [1] and Yao et al. [2], have already demonstrated that internal model representations or token distributions contain useful signals for detecting uncertainty and supporting abstention. Could the authors more clearly state the methodological novelty of Rep2Act and explain how it advances beyond these existing approaches?

  2. Does training the VLM to align with the router improve abstention performance? Could the authors provide an ablation in which the classification head is used solely as a hard abstention router, without training the VLM to align its responses with the router's predictions? Such a comparison would help establish whether the route-contrastive loss provides meaningful benefits beyond those achievable through direct routing.

  3. How were the chain-of-thought (CoT) annotations and training data generated and validated? Could the authors provide the exact prompts used with the teacher model to generate the CoT annotations and training data? Additionally, how did the authors verify the correctness and reliability of the generated reasoning steps?

[1] Zhao, Q., Xu, M., Gupta, K., Asthana, A., Zheng, L., and Gould, S. "The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models?" European Conference on Computer Vision (ECCV), 2024. https://arxiv.org/abs/2403.09037

[2] Yao, J., Kulshrestha, A., Rauschmayr, N., Roberts, R., Zhu, B., Tsvetkov, Y., and Tombari, F. "Reading Between the Lines: Abstaining from VLM-Generated OCR Errors via Latent Representation Probes." arXiv preprint arXiv:2511.19806, 2025. https://arxiv.org/abs/2511.19806

Reviewer_t1UHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work examines a task where VLM fails to provide answers to questions without visual grounding (evidence), instead inducing hallucinations. To address this challenge, the authors propose the VAD-R benchmark, which eliminates shortcut hints, and the Rep2Act framework, which converts answerability information within the model into explicit abstention decisions.

Critical strengths
  1. The authors introduce VAD-R, a benchmark designed to decrease answerability from linguistic shortcuts.
  2. The authors examine the gap between VLM's internal representations of answerability and the fact that this signal does not translate into actual rejection behavior.
Critical weaknesses
  1. The authors mainly use the old model in their experiments. Why not report the performance of the GPT-4 series using the more recent, stronger models instead?
  2. The authors have included related work in Appendix A. However, Appendix A is not referenced anywhere in the main text. I believe that related work is a crucial part of the paper, but I'm worried that this behavior is considered page cheating.
  3. I think a more comprehensive evaluation of the VAD-R's quality is needed. A majority vote through multiple annotations might be helpful. It seems that a more detailed explanation of human evaluation is needed.
  4. To verify generalizability, I think it would be good to add additional seed image datasets (now the authors use only DOCCI).
  5. I think the authors need to mention and compare similar previous works (benchmarks). To my best knowledge, there are several works related to answer refusal in VQA, etc.
  6. The authors use LLM-as-a-judge (line 368), but there is no report about human alignment (also, I am curious about whether Qwen3.6-27B can serve as a judge).
Questions for authors
  1. I think it would be better to improve the visibility of Figure 1 and Figure 2.
  2. In Figure 10, there are some blue billboard signs behind the car, so I think that is not a good example.
  3. The current version is difficult to follow. It would be better to improve readability.
Reviewer_RbHyOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies whether vision-language models (VLMs) spontaneously abstain from visually unanswerable questions. It introduces VAD-R, a 1,852-item benchmark built on DOCCI images, with rationales and six evidence-gap labels; the items are LLM-generated, filtered for text shortcuts, and LLM-audited. On VAD-R, eleven VLMs rarely abstain when answering directly (11.4%/16.3% abstention recall for open/closed models) but do so readily when explicitly asked (78.0%/94.9%). Meanwhile, later-layer hidden states linearly separate answerability. To close this "representation-to-action" gap, the proposed Rep2Act emits an explicit A/U routing token before a rationale and answer. It is trained with a routing-head loss, a signed margin loss, and a route-contrastive loss. Rep2Act raises VAD-R action accuracy from 56.67% to 86.33% (3B) and from 59.33% to 88.67% (7B), gives modest out-of-distribution gains on TUBench and UNK-VQA, and preserves general VQA performance.

Critical strengths
  • Existing abstention benchmarks either offer an explicit "unanswerable" option or leak the label through text or image cues. In Table 9, the TUBench UTabMWP image-only probe and the MM-UPD MMAAD text-only probe both reach 100% AUROC. VAD-R removes the hint and attaches causal rationales to every item, making it a careful step toward measuring abstention when the model is not told it may abstain. If the labels prove reliable, this reusable resource is the paper's most lasting contribution.

  • Table 11 shows a consistent gap between spontaneous and prompted abstention across 11 models from several families, including frontier closed models. Abstention recall rises by 49 to 87 points under explicit judgment. This finding is practically important and gives a compelling motivation for training models to act on their own uncertainty, not just to report it.

Critical weaknesses

W1. The data pipeline and evaluation rely on one model family at every stage, which risks circularity, and both the benchmark's validity and the method's gains rest on it. The leakage check in Table 1 uses the same probe type and backbone (Qwen2.5-VL-7B) as the shortcut filter, so near-chance accuracy is expected by construction, and with only 300 items the check has little power. Unanswerable labels are effectively defined by Qwen3.6-27B, which passed only 22.8% of candidates. That same model then achieves the top explicit-judgment accuracy (91.3%) and also serves as the action judge. C-Acc is graded by Qwen3.8-27B against Qwen-generated references written in the same style Rep2Act is trained on. Rep2Act’s C-Acc lead over LVLM-LP at 7B (60.67 vs. 47.33) may therefore partly reflect style matching, and a single-annotator audit of 100 items cannot rule these effects out.

W2. Gains over the strongest baselines are small and mixed, and some claims go beyond the evidence. At 7B, Rep2Act exceeds LVLM-LP by only 3 of 300 test items in classification accuracy, LVLM-LP has the higher AUROC (95.82 vs. 93.82), and no seeds or confidence intervals are reported. The vocabulary-only variant matches the full method at 7B, but it is still trained with L_route, so the contribution of the representation component is untested. Out of distribution, Rep2Act trails baselines on TUBench response M-F1 (50.71 vs. 52.86) and UNK-VQA classification M-F1 (43.58 vs. 48.73). The abstract's GPT-4o comparison omits GPT-4o mini, which scores higher (53.5 vs. 53.3 F1). Appendix C.2's claim of "leading" UTabMWP performance is not supported by Table 13.

Questions for authors
  1. How were the 926 answerable items selected from the roughly 4,291 answerable candidates? Were they audited for genuine answerability, and is the selection matched across visual skills?
  2. Please report means and standard deviations (or bootstrap confidence intervals) over at least 3 training seeds for Tables 3, 7, and 8. Please include a significance test against LVLM-LP and P_JA at 7B.
  3. Please measure over-refusal directly by reporting the abstention rate on POPE, GQA, and TextVQA answerable questions. Please also report Rep2Act with the fused α used on VAD-R, not α = 0.
  4. Is the C-Acc advantage robust to a judge from a different model family (e.g., GPT- or Gemini-based) and to human grading?
  5. Can you provide human validation of the 300-item test set by 2–3 annotators, with inter-annotator agreement, and report how often the Qwen action and content judges agree with humans?
  6. Can you re-run the shortcut probes in Tables 1 and 9 on backbones not used for filtering (e.g., a text-only LLM, SigLIP, or InternVL)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.