acceptodds
Withdrawn from ICLR 2027

DO VISION-LANGUAGE MODELS SEE ILLUSIONS LIKE HUMANS DO? A PARAMETRIC PSYCHOPHYSICS BENCHMARK FOR VLM VISUAL BIAS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) are routinely evaluated on optical illusions by scoring a single modified image “right” or “wrong,” collapsing a graded perceptual phenomenon into a binary label. We introduce ILLUSIONCURVE-BENCH, a benchmark that instead sweeps four classical illusions (Ebbinghaus, M¨uller-Lyer, Ponzo, Vertical–Horizontal) across 11 stimulus-magnitude steps each, fits psychometric functions to three open-weight VLMs’ (Qwen2.5-VL-3B/7B-Instruct, LLaVA-1.5-7B) forced-choice responses, and compares the resulting point-ofsubjective- equality (PSE) and slope estimates to published human norms – with the A/B label and position assignment counterbalanced across repeats so a model’s response-format bias cannot masquerade as a genuine perceptual offset. Of the 12 model–illusion response curves, only three are classified graded/human-like (one on a sparse cell where 86% of responses were “Equal”), while most (nine) are erratic or fail to converge at all, and 4 of the 8 that do converge place their PSE outside the tested range entirely. Doubling our repeat count mid-study to rule out sampling noise made this picture more, not less, extreme, and an auxiliary same/different confidence probe is similarly decoupled from stimulus ambiguity and unstable across reruns – one 7B-scale model stays within 0.25 of a constant “Same” verdict on every illusion, and which cells look dip-like at the point of maximal ambiguity changes across every rerun. Our results indicate that binary-accuracy illusion benchmarks systematically misattribute categorical, memorization-driven behavior to human-like perceptual bias, and that psychophysical curve-fitting with response-format counterbalancing is necessary – though not sufficient on its own, given the resampling instability we observe – to tell the two apart. We release ILLUSIONCURVE-BENCH, our sweep generators, and evaluation code.

Primary area:
applications to neuroscience & cognitive science
Keywords:
vision-language models, optical illusions, psychophysics, illusion benchmarking, psychometric curves, point of subjective equality, VLM evaluation, response bias, human perception, model interpretability, visual reasoning, benchmark design, Qwen2.5-VL, LLaVA. Provide your feedback on BizChat

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JFPfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IllusionCurve-Bench, which sweeps four classical illusions over 11 steps and fits psychometric functions to VLM forced-choice answers. On Qwen2.5 and LLaVA1.5, only 3 of 12 curves are classified graded/human-like. The authors conclude that binary benchmarks misattribute categorical, memorization-driven behavior to human-like bias.

Critical strengths
  1. The paper is honest about its instability (Tables 5 and 6).
Critical weaknesses
  1. The comparison with human norms is not valid. For Ebbinghaus and Muller-Lyer the two targets are physically equal at every step, so any symmetric observer has its PSE near x = 0, and "Equal" is the correct answer but is dropped from the fit. Model PSEs (normalized inducer units) are compared with human values in length units, and human slopes are missing for all illusions. I suggest the authors use the standard method of constant stimuli and report PSE in physical units.
  2. The measurement is too noisy to support the per-cell conclusions. Only 1 of 8 cells keeps its class across three runs, answers are sampled at temperature 0.7 with R = 16, and the taxonomy is not validated (e.g., the clear step of Qwen-7B on Ebbinghaus is labeled erratic). Since all models are open-weight, I suggest the authors read A/B/Equal token probabilities directly and validate the classifier on simulated observers.
  3. Several main claims are not supported. No experiment tests memorization, and Section 7 itself says the failure is not prior-knowledge override. The same/different probe compares against the x = 0 image, so "Different" is simply correct at every other step, and this measures change detection rather than confidence.
  4. The manuscript also still contains drafting notes or AI notes in Reference page 10-11 (e.g., "corroborated at abstract level in this session", "No peer-reviewed venue confirmed as of the most recent revision (April 2026) at the time of writing; verify before camera-ready in case a conference version appears.").

[*1] Yichi Zhang et al. Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? EMNLP 2023.

Questions for authors

As in weakness.

Reviewer_bP5EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces IllusionCurve-Bench, a parametric benchmark for evaluating visual biases in vision-language models. It generates stimulus-magnitude sweeps for four classical optical illusions and fits psychometric curves to model responses. Experiments on three VLMs reveal substantial deviations from human perceptual patterns and instability across repeated evaluations.

Critical strengths
  1. The benchmark extends binary illusion evaluation to continuous psychometric curves, providing a more detailed characterization of VLM perceptual behavior.
  2. The study incorporates label and position counterbalancing, repeated sampling, and psychometric analysis to investigate response biases and evaluation stability.
Critical weaknesses
  1. All three curves classified as human-like have PSE estimates outside the tested range. Classifications also change across repeated evaluations, raising concerns about their reliability.
  2. The results show unstable or non-monotonic responses but do not directly establish memorization-driven behavior. Moreover, the auxiliary same/different task does not directly measure model confidence.
Questions for authors
  1. How reliable are the human-like classifications when their fitted PSE values fall outside the tested range? How does excluding "Equal" responses affect these estimates?
  2. What evidence distinguishes memorization-driven behavior from unreliable visual discrimination or response noise?
  3. Can the authors validate the findings across more VLMs and stimulus configurations, and clarify how the same/different task reflects confidence?
Reviewer_pi8fOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a parametric benchmark for testing whether vision-language models show anything like human susceptibility to classical visual illusions. The study compares estimated points of subjective equality (PSEs) and response slopes against published human psychophysical measurements. The authors further investigate response-label counterbalancing and introduce an auxiliary same/different task to examine the relationship between stimulus ambiguity and model responses.

Critical strengths
  1. The paper addresses an interesting question regarding the relationship between VLM visual behavior and human perceptual phenomena. 2) The authors recognize potential confounding effects associated with response labels and positional biases and attempt to address them through counterbalancing. 3) The paper is relatively transparent about unsuccessful curve fitting, unstable results etc.
Critical weaknesses
  1. The authors did a direct copy of the human psychological experiments onto evaluating LLMs - with limited innovations
  2. The fitting procedure has fundamental concerns on wether the ways that the authors did are correct i.e. the conditional response function is not necessarily equivalent to the psychometric function assumed by the paper, and its fitted PSE may not provide a valid estimate of perceptual equality.
  3. The results are very unstable for any kinds of solid conclusions to be drawn.
  4. The experimental coverage of the models are extremely limited. They have only tested 3 checkpoints: Qwen2.5-VL-3B, Qwen2.5-VL-7B, and LLaVA-1.5-7B.
Questions for authors

Why should the same/different identification rate be interpreted as a measure of confidence rather than as a separate perceptual discrimination task?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.