- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes IllusionCurve-Bench, which sweeps four classical illusions over 11 steps and fits psychometric functions to VLM forced-choice answers. On Qwen2.5 and LLaVA1.5, only 3 of 12 curves are classified graded/human-like. The authors conclude that binary benchmarks misattribute categorical, memorization-driven behavior to human-like bias.
Critical strengths
- The paper is honest about its instability (Tables 5 and 6).
Critical weaknesses
- The comparison with human norms is not valid. For Ebbinghaus and Muller-Lyer the two targets are physically equal at every step, so any symmetric observer has its PSE near x = 0, and "Equal" is the correct answer but is dropped from the fit. Model PSEs (normalized inducer units) are compared with human values in length units, and human slopes are missing for all illusions. I suggest the authors use the standard method of constant stimuli and report PSE in physical units.
- The measurement is too noisy to support the per-cell conclusions. Only 1 of 8 cells keeps its class across three runs, answers are sampled at temperature 0.7 with R = 16, and the taxonomy is not validated (e.g., the clear step of Qwen-7B on Ebbinghaus is labeled erratic). Since all models are open-weight, I suggest the authors read A/B/Equal token probabilities directly and validate the classifier on simulated observers.
- Several main claims are not supported. No experiment tests memorization, and Section 7 itself says the failure is not prior-knowledge override. The same/different probe compares against the x = 0 image, so "Different" is simply correct at every other step, and this measures change detection rather than confidence.
- The manuscript also still contains drafting notes or AI notes in Reference page 10-11 (e.g., "corroborated at abstract level in this session", "No peer-reviewed venue confirmed as of the most recent revision (April 2026) at the time of writing; verify before camera-ready in case a conference version appears.").
[*1] Yichi Zhang et al. Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? EMNLP 2023.
Questions for authors
As in weakness.