- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces VIA-Bench, a benchmark of 1,004 human-verified image–question pairs covering six categories of visual illusions and anomalies. It tests whether multimodal language models answer according to the stipulated physical facts of an image when its appearance or common-sense priors suggest otherwise. The authors evaluate 30 models, alongside human and text-only baselines, and conduct smaller open-ended, prompting, and oracle-fact experiments. They report substantial variation across models and categories: the best model achieves 82.10% average accuracy, compared with 93.30% for humans. They also find limited average benefits from general-purpose chain-of-thought prompting and classify observed incorrect responses as visual misperception, self-negation or uncertainty, and overthinking.
Critical strengths
- A valuable research question and a substantial dataset. The authors bring six types of visual illusions and anomalies under a common framework of conflict between visual appearance and objective facts. They construct 1,004 question–answer pairs and have a second annotator verify each item. This provides a concrete testbed for studying model behavior on counterintuitive images.
- Broad evaluation with informative controls. The paper evaluates 30 models, reports category-level results, and includes human, text-only, open-ended, and prompting comparisons. The authors’ reporting of the high text-only score on motion illusions is particularly useful: it exposes a shortcut in their own benchmark and helps readers interpret the results more carefully.
Critical weaknesses
The paper attributes failures to visual misperception, but does not report model-specific image input settings or test whether the details required by pixel-level and high-resolution items survive preprocessing. This leaves the diagnostic attribution uncertain. Please report the available input settings and evaluate sensitivity to resolution and local crops; otherwise, narrow the claims to observed answer behavior.
Questions for authors
- What evidence shows that the illusion-inducing content causes an additional performance drop? Could you evaluate a representative subset using matched non-illusion images that preserve the target fact, question wording, and comparable visual complexity, while removing the illusion-inducing context? Without such a comparison, how can you distinguish an illusion-specific effect from the models’ baseline error rate on similar visual tasks?
- Section 2.2 distinguishes whether a model perceives an illusion from whether it gives the physically correct answer. Given this distinction, what justifies interpreting incorrect answers on VIA-Bench as evidence of illusion-induced perceptual conflict, rather than difficulty with the underlying visual task? Please clarify precisely what the benchmark measures.