- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates process failure in VLMs, where models correctly perceive visual evidence but fail to use it consistently in their final decisions. It introduces VPAC-Bench and proposes SRT (State–Relevance–Target), a structured prompting framework. Experiments across multiple VLMs and visual reasoning tasks show that generic SRT has inconsistent benefits, while oracle boundary-aligned SRT achieves substantial improvements.
Critical strengths
- The distinction between perceptual failure and process failure provides an interesting perspective on VLM reasoning.
- The paper covers diverse models and tasks, with extensive baselines and ablation studies.
- Negative results and methodological limitations are reported transparently.
Critical weaknesses
- Lack of a coherent scientific hypothesis. The S1–S3 experiments address different tasks without a convincing methodological connection. The connection among S1–S3 is not sufficiently established, making it difficult to identify a common hypothesis that these experiments are intended to test.
- Limited methodological novelty. SRT largely reformulates existing structured prompting approaches, and S1 does not directly evaluate the proposed method.
- Weak practical effectiveness. Generic SRT does not consistently outperform CoT baselines, while the strongest improvements depend on oracle boundary annotations. The self-router offers negligible gains.
- Insufficient causal validation. The oracle comparison is confounded by differences in annotation access and output formatting. Several explanations, including process overload and residual uncertainty, remain post-hoc hypotheses rather than validated mechanisms.
Questions for authors
- What is the central falsifiable hypothesis connecting S1, S2, and S3, and how do the experiments systematically validate it?
- Could the authors clarify which aspects of SRT go beyond existing structured prompting methods, and whether these differences can be isolated experimentally?
- Could the authors provide schema-matched controls to isolate the contribution of boundary-aligned process reasoning?