acceptodds
Under review as a conference paper at ICLR 2027

Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomalies

Abstract

Multimodal Large Language Models (MLLMs) now match or exceed human performance on many vision-language benchmarks. However, these evaluations typically rely on standard in-distribution data, leaving the robustness of MLLMs largely unexamined under prior conflict scenarios. To address this gap, we introduce VIA-Bench, a diagnostic benchmark that probes whether MLLMs remain grounded in objective physical reality under such conflicts. VIA-Bench covers six categories: color illusions, motion illusions, gestalt illusions, geometric and spatial illusions, general visual illusions, and visual anomalies. Through careful human-in-the-loop review, we construct over 1K high-quality question-answer pairs supporting both multiple-choice and open-ended evaluation. Extensive evaluation of 30 state-of-the-art MLLMs, including proprietary, open-source, and reasoning-enhanced models, uncovers significant vulnerabilities. Beyond aggregate accuracy, we further find that general-purpose Chain-of-Thought (CoT) prompting offers negligible robustness, often yielding “brittle mirages” where the model's logic collapses under illusory stimuli. Even task-specific CoT helps only marginally. To better characterize these failures, we introduce an error taxonomy that disentangles visual misperception, self-negation, and overthinking. These findings expose a structural vulnerability of current MLLMs under perceptual conflict and motivate future work on visually grounded reasoning. The benchmark data and code will be released.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Large Language Models, Visual Illusions, Visual Anomalies, Evaluation, Chain-of-Thought

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qcwuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VIA-Bench, a benchmark of 1,004 human-verified image–question pairs covering six categories of visual illusions and anomalies. It tests whether multimodal language models answer according to the stipulated physical facts of an image when its appearance or common-sense priors suggest otherwise. The authors evaluate 30 models, alongside human and text-only baselines, and conduct smaller open-ended, prompting, and oracle-fact experiments. They report substantial variation across models and categories: the best model achieves 82.10% average accuracy, compared with 93.30% for humans. They also find limited average benefits from general-purpose chain-of-thought prompting and classify observed incorrect responses as visual misperception, self-negation or uncertainty, and overthinking.

Critical strengths
  1. A valuable research question and a substantial dataset. The authors bring six types of visual illusions and anomalies under a common framework of conflict between visual appearance and objective facts. They construct 1,004 question–answer pairs and have a second annotator verify each item. This provides a concrete testbed for studying model behavior on counterintuitive images.
  2. Broad evaluation with informative controls. The paper evaluates 30 models, reports category-level results, and includes human, text-only, open-ended, and prompting comparisons. The authors’ reporting of the high text-only score on motion illusions is particularly useful: it exposes a shortcut in their own benchmark and helps readers interpret the results more carefully.
Critical weaknesses

The paper attributes failures to visual misperception, but does not report model-specific image input settings or test whether the details required by pixel-level and high-resolution items survive preprocessing. This leaves the diagnostic attribution uncertain. Please report the available input settings and evaluate sensitivity to resolution and local crops; otherwise, narrow the claims to observed answer behavior.

Questions for authors
  1. What evidence shows that the illusion-inducing content causes an additional performance drop? Could you evaluate a representative subset using matched non-illusion images that preserve the target fact, question wording, and comparable visual complexity, while removing the illusion-inducing context? Without such a comparison, how can you distinguish an illusion-specific effect from the models’ baseline error rate on similar visual tasks?
  2. Section 2.2 distinguishes whether a model perceives an illusion from whether it gives the physically correct answer. Given this distinction, what justifies interpreting incorrect answers on VIA-Bench as evidence of illusion-induced perceptual conflict, rather than difficulty with the underlying visual task? Please clarify precisely what the benchmark measures.
Reviewer_pZbmOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VIA-Bench, a benchmark of 1,004 visual illusion and anomaly QA pairs across six categories. The authors evaluate 30 MLLMs and analyze their performance, the effects of COT prompting, and common failure patterns.

Critical strengths
  1. Important and relevant problem. The paper addresses an important challenge: whether MLLMs can correctly interpret visual evidence when it conflicts with learned priors. Visual illusions and anomalies provide an interesting setting for studying this problem, as they can expose failures that are often overlooked by conventional benchmarks.

  2. Comprehensive evaluation and analysis. The paper evaluates 30 MLLMs and the authors also conduct additional experiments, including COT prompting, open-ended evaluation, and error analysis. These experiments provide a useful overview of current models' limitations and highlight that stronger reasoning capabilities do not necessarily translate into more reliable visual perception.

Critical weaknesses
  1. Limited novelty and insights. Beyond collecting additional data, the paper provides limited investigation into why models fail. Compared with existing benchmarks such as VI-Probe and IllusionBench, the experimental analysis offers relatively few new insights. The dataset is also not particularly large, making its contribution as a new benchmark less compelling.

  2. Insufficient support for the central claim. The paper claims to diagnose conflicts between visual evidence and learned priors. However, since most images are collected rather than systematically constructed, it is difficult to manipulate visual factors or create controlled counterfactual comparisons. As a result, the benchmark cannot clearly distinguish failures caused by learned priors from general visual perception errors, weakening its central diagnostic claim.

Questions for authors

1, Error taxonomy: The paper categorizes model failures into visual misperception, self-negation, and overthinking. How are these categories reliably distinguished in practice? Could the authors clarify the annotation procedure?

  1. CoT analysis: The main general-purpose CoT ablation is conducted on only two models. Is this sufficient to support the broader conclusion that CoT generally fails to improve or even degrades performance? Could the authors provide additional evidence across reasoning models?
Reviewer_ia4wOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces VIA-Bench, a diagnostic benchmark that evaluates if VLMs are susceptible to visual illusions under misconceptions. By curating a human-in-the-loop process, the paper collects over 1000 VQA pairs spanning six categories of visual illusions and anamolies and evaluate 30 mainstream models. The results show that even the best-performing model still lags behind humans, achieving 82.10% accuracy compared with humans’ 93.30%. Further, an error analysis is conducted, demonstrating current failure modes of VLMs in these settings.

Critical strengths
  1. This benchmark contains over 1,000 human-curated VQA pairs, reflecting substantial annotation effort, and covers a broad range of visual illusions and anomalies across six categories.

  2. The evaluation covers 30 models, which provide a comprehensive landspace of how current mainstream VLMs perform.

  3. I find the use of visual illusions to probe VLMs’ reliance on language priors interesting, especially in cases where language priors still dominate the output space of VLMs.

Critical weaknesses
  1. The text-only GPT-4-Turbo baseline achieves 87.95% on motion illusions and 61.11% on geometric and spatial illusions, suggesting that high accuracy in these categories does not necessarily demonstrate visual grounding. Although the paper includes open-ended evaluation, its limited coverage leaves unclear whether the main model rankings and conclusions remain consistent without answer options.
  2. Several observations, including unreliable visual interpretation and limited benefits from CoT, are broadly consistent with prior findings, such as those in IllusionVQA[1]. While the oracle-fact probes are useful, the analysis provides limited evidence about when different failures occur and how the proposed taxonomy could inform targeted improvements.
  3. The authors show that removing answer options substantially reduces accuracy, yet the main model comparisons remain based on multiple-choice evaluation. This leaves unclear to what extent the reported scores reflect visual capabilities versus reliance on cues from the answer candidates.
  4. I think several of the diagnostic insights demonstrated in the paper have already been addressed in several works. For example, in BLINK-Twice[2], they also evaluate whether models can identify key visual clues from misguided cues and misleading visual scenes. The findings are closely related to those showed in VIA-Bench. It would be helpful to clarify what additional diagnostic insights VIA-Bench provides beyond these findings.

Overall, my main concern is that VIA-Bench’s challenges substantially overlap with those covered by prior benchmarks, and its additional diagnostic value over existing work remains unclear.

References:

[1] IllusionVQA: A Challenging Optical Illusion Dataset for Vision Language Models

[2] BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception

Questions for authors
  1. Given the substantial performance drop after removing answer options, why is open-ended evaluation not adopted more broadly? Could the authors extend this evaluation to the strongest models and verify whether the main rankings and conclusions still hold?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.