acceptodds
Under review as a conference paper at ICLR 2027

Seeing Is Not Enough: Vision-Language Models Perceive Evidence but Fail to Act

Abstract

Vision-language models (VLMs) achieve strong performance on visual question answering benchmarks, yet often produce decisions that contradict visual evidence they have already correctly perceived. Existing work does not distinguish perceptual failure, where relevant evidence is not perceived, from process failure, where correctly perceived evidence fails to constrain the final decision. To address this gap, we first introduce VPAC-Bench, a benchmark of nine real-image process families in which each image is annotated with its current activity stage and the specific stage transition it falls near eg. whether a tool has made contact with an object (setup vs. active), or whether the intended outcome has been achieved (active vs. finished). Building on this structure, we develop SRT (State–Relevance–Target), a family of structured process-prior interventions that constrain how a model must use visible evidence before committing to a final answer. First, process failure is widespread and distinct from perceptual failure: models that correctly enumerate visual candidates still over-commit to a single answer at rates exceeding 95%, and an explicit process-structured intervention reduces this to below 13% without degrading unambiguous-referent performance. Second, process-prior transfer is model-sensitive: the same structured prior helps some models substantially while leaving others unchanged or degraded, and generic SRT alone does not reliably outperform strong CoT-style baselines. Third, when the correct stage transition is known, boundary-aligned SRT substantially outperforms generic process prompting and all CoT-style baselines across all four standardized process families (assembly, physical state transition, navigation/traffic, and object-use affordance). These findings indicate that the value of a process prior depends critically on its alignment with the image's specific decision boundary, motivating boundary-aware prior selection as a design strategy for process-grounded visual reasoning.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language models, visual reasoning, process failure, visual grounding, evidence-consistent reasoning, process priors, activity stage recognition, multimodal chain-of-thought, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RzSnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates process failure in VLMs, where models correctly perceive visual evidence but fail to use it consistently in their final decisions. It introduces VPAC-Bench and proposes SRT (State–Relevance–Target), a structured prompting framework. Experiments across multiple VLMs and visual reasoning tasks show that generic SRT has inconsistent benefits, while oracle boundary-aligned SRT achieves substantial improvements.

Critical strengths
  1. The distinction between perceptual failure and process failure provides an interesting perspective on VLM reasoning.
  2. The paper covers diverse models and tasks, with extensive baselines and ablation studies.
  3. Negative results and methodological limitations are reported transparently.
Critical weaknesses
  1. Lack of a coherent scientific hypothesis. The S1–S3 experiments address different tasks without a convincing methodological connection. The connection among S1–S3 is not sufficiently established, making it difficult to identify a common hypothesis that these experiments are intended to test.
  2. Limited methodological novelty. SRT largely reformulates existing structured prompting approaches, and S1 does not directly evaluate the proposed method.
  3. Weak practical effectiveness. Generic SRT does not consistently outperform CoT baselines, while the strongest improvements depend on oracle boundary annotations. The self-router offers negligible gains.
  4. Insufficient causal validation. The oracle comparison is confounded by differences in annotation access and output formatting. Several explanations, including process overload and residual uncertainty, remain post-hoc hypotheses rather than validated mechanisms.
Questions for authors
  1. What is the central falsifiable hypothesis connecting S1, S2, and S3, and how do the experiments systematically validate it?
  2. Could the authors clarify which aspects of SRT go beyond existing structured prompting methods, and whether these differences can be isolated experimentally?
  3. Could the authors provide schema-matched controls to isolate the contribution of boundary-aligned process reasoning?
Reviewer_SjGoOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper identifies process failure where VLMs perceive the decision-relevant evidence yet fail to let it constrain their answers, and introduces VPAC-BENCH together with SRT (State–Relevance–Target) for measurement. It presents a three-stage investigation covering referent ambiguity, spatial verification, and process-stage recognition. Its main finding is that generic process prompting has inconsistent effects, while an annotation-assisted boundary-specific condition achieves substantially higher performance on four standardized process families.

Critical strengths

The ambiguity experiments provide useful evidence that identifying multiple candidates does not necessarily lead to an appropriate answer policy. The marked-image comparison and unique-referent controls strengthen this observation. The evaluation also includes several valuable controls: CoT-style baselines, component ablations, incorrect-boundary prompting, label-only conditions, and self-routing. Reporting negative results and the reversals on supporting families is a strength.

Critical weaknesses
  1. Confounded boundary-alignment claim: Oracle SRT receives additional rationales and differs in candidate construction and output format, preventing attribution of its gains specifically to boundary alignment. Information-matched controls are needed, especially given generic SRT’s limited advantage over CoT and the small self-routing gains.

  2. Inconsistent evaluation reporting: "Missing Mistral outputs" mentioned in Appendix A.1 conflict with reported five-model summaries, while scoring alternates between excluding unparseable responses and counting them as incorrect. Actual sample counts, model coverage, parse rates, and scoring conventions should be double-checked.

  3. Overclaim: The abstract’s greater-than-95% over-commitment statement appears to combine default-condition over-answer rates with the separate evidence-enumeration finding. Table 2 reports substantially lower over-answer rates under evidence prompting.

  4. Incomplete Figure 1: Figure 1’s VPAC-Bench illustration is incomplete and should represent all nine process families.

Questions for authors
  1. What remains of the oracle gain when SRT and CoT receive identical boundary information, rationales, candidates, ordering, and output schemas?
  2. Can all primary results be regenerated from available paired records using one scoring convention and actual denominators?
  3. What is the inconsistency rate specifically among responses whose candidate enumeration is independently verified as correct, and what are the field-specific annotation agreement statistics?
Reviewer_FvEnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates process failure in vision-language models: cases where models correctly identify relevant visual evidence but fail to use it consistently in their final decisions. It introduces VPAC-Bench, covering nine real-image process families, and proposes SRT (State–Relevance–Target), a prompting framework for extracting evidence, verifying relevant relations, and making evidence-supported decisions. Experiments across five VLMs show that structured guidance can reduce unsupported answers, but generic SRT has inconsistent effects and does not reliably outperform strong chain-of-thought baselines. Annotation-assisted, boundary-aligned SRT achieves substantial improvements, highlighting the importance of matching process guidance to the image’s specific decision boundary, although reliable automatic boundary selection remains unresolved.

Critical strengths
  1. he paper distinguishes failing to perceive relevant evidence from failing to use correctly perceived evidence in a decision. Its ambiguous- and unique-referent experiments support this distinction while ruling out a trivial “always clarify” strategy.

  2. SRT explicitly separates evidence extraction, relevance checking, and final commitment. Unlike generic instructions to reason step by step, it targets the specific connection between visual observations and evidence-supported decisions.

  3. The experiments reveal that generic process guidance can help or harm depending on the model and task. Boundary-conditioned comparisons identify a promising direction while the substantial oracle–self-router gap clearly exposes the unresolved challenge of selecting appropriate guidance automatically.

  4. The results suggest that additional reasoning structure can be counterproductive when simpler information already supports a decision. This motivates studying both decision-boundary recognition and the ability to determine when additional reasoning is unnecessary, rather than treating more elaborate reasoning as universally beneficial.

Critical weaknesses
  1. Although the evaluation includes Qwen2.5-VL-72B and Gemini-3.1-Pro—not merely small models—it lacks controlled comparisons across model sizes and reasoning budgets. Consequently, it remains unclear whether process failure persists as capabilities improve or partly reflects particular instruction-following and direct-answer configurations. This matters because the paper presents the phenomenon as a broader limitation of VLM reasoning.

  2. S1 examines contradictions between explicitly enumerated evidence and final decisions, but S2 and S3 primarily report answer accuracy, which cannot distinguish perceptual errors from failures to use correctly perceived evidence. Moreover, a correct verbal description does not establish that the described evidence actually guided subsequent reasoning. Stage-level consistency measurements and interventions that hold extracted evidence fixed would provide stronger support for the central diagnosis.

  3. Oracle SRT receives gold boundary annotations and item-specific rationales, while comparison conditions differ in candidate construction and output schema. Label-only and oracle conditions also have substantially different parsing rates, so their performance gap cannot be attributed specifically to SRT’s process structure. Comparisons giving CoT and label-only baselines identical annotations, candidate labels, and output formats are necessary to isolate the proposed mechanism.

  4. Generic SRT does not reliably outperform strong CoT baselines, while self-routing improves over generic SRT by only 1.7 and 0.6 percentage points in the two tested families, without established significance. Furthermore, oracle SRT underperforms label-only prompting on Craft and Mobility despite receiving the correct boundary. These findings limit both practical utility and the explanatory reach of boundary alignment; the proposed residual-uncertainty account remains a hypothesis rather than a directly validated explanation.

Questions for authors
  1. How do you distinguish process failures from perceptual errors in S2 and S3, given that these experiments primarily measure final-answer accuracy rather than consistency between intermediate evidence and final decisions?

  2. Why were CoT and label-only baselines not given the same annotations, rationales, and output schema as oracle SRT, and would SRT’s advantage remain under these matched conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.