- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SpatialUncertain to test whether VLMs recognize when visual evidence is missing or misleading. It uses controlled 3D scenes to vary occlusion and perspective, and evaluates answering, abstention, and view selection across eight models. Models often answer despite missing evidence or follow apparent rather than physical size. The authors support their findings with internal analyses, a small real-image study, and prompting, fine-tuning, and additional-view experiments.
Critical strengths
- The benchmark question setup is interesting and make the study possible. And there's not many benchmarks about the uncertainty of VLMs spatial cognition.
- The authors check the main findings through several methods, including real-image viewpoint consistency and answer-format comparisons.
Critical weaknesses
- The model coverage is incomplete. The authors evaluate eight models, but omit newest GPT and the Claude family, despite their availability before submission. I’ve noticed that the latest GPT and Claude models have improved significantly in spatial ability. Evaluating them may reveal some different findings.
- Most of the evidence comes from synthetic scenes. The real-image study includes only 35 object pairs and checks size judgments across viewpoints.
Questions for authors
- Can humans reliably answer the perspective-conflict questions from the single image? Have you done some human verifications?
- What was the evaluation cutoff date? Why were GPT-5.5 and vision-capable Claude models omitted, and do the main findings hold for them?
2 comments
Dear Reviewer UkRC,
As the discussion period progresses, we would like to gently follow up on our rebuttal.
We greatly appreciate your constructive feedback, which guided us to add evaluations for the latest GPT-5.5 and Claude models, expand the real-image study to 80 pairs, and conduct the human verification study you suggested.
Could you please let us know if our response has properly addressed your concerns? We would be happy to provide any further clarifications if needed.
Thank you again for your time and effort in reviewing our paper!
Best regards, Authors
We thank the reviewer for the constructive feedback. We appreciate the reviewer's recognition that the benchmark setup makes this study possible, that few benchmarks address the uncertainty of VLMs' spatial reasoning, and that our main findings are checked through several methods. The comments led to three changes in the revised paper:
We respond to each point below.
W1 & Q2. Evaluation on newer GPT and Claude models.
Our initial evaluation was conducted in April 2026, and the model selection was finalized at that time. To address the reviewer's concern, we have extended the evaluation to GPT-5.5 and two Claude models, following the same protocol. They are now included in Table 1.
Claude-Fable-5.1 achieves the highest 3D-consistent rate among all evaluated models (35.4%), surpassing the previous best, Gemini-2.5-Flash (32.5%). However, our main findings still hold:
Overall, while newer models show different patterns of accuracy and abstention, they remain vulnerable to misleading projections and do not consistently express uncertainty when visual evidence is insufficient. Within the GPT family, accuracy on answerable occlusion questions has improved from GPT-4o to the GPT-5 models (43.7% → 59.0–70.2%), but this has not been accompanied by better abstention: the GPT-5 models abstain far less on unanswerable questions (63.6% → 8.4–21.1%) and follow the 2D projection more often (35.6% → 61.1–72.6%).
W2. Limited real-image evaluation.
We expanded the real-image set from 35 to 80 object pairs, using the same collection protocol and exclusion rule. The pattern holds at this larger scale: GPT-5-mini gives contradictory size judgments across viewpoints on 66 of 80 pairs (82.5%), and Gemini-2.5-Flash on 57 of 80 (71.3%), each time favoring the object that projects larger. These rates are close to those on the original 35 pairs (86% and 66%).
Q1. Can humans reliably answer perspective-conflict questions from a single image?
We conducted a human study with four undergraduate students (none are co-authors) on 150 perspective-conflict questions and 30 clean controls, using the same images, questions, and answer options as the models, including the option to abstain. Models are evaluated on the same questions. The study is reported in Sec. 4.3 (Table 2).
These findings suggest that perspective-conflict views are challenging even for human observers. Humans achieve 36.0% 3D-consistent responses and abstain in 55.3% of cases, while only 8.7% of their responses follow misleading projections. In contrast, models frequently commit to projection-following answers (53.3–66.7%) and rarely abstain (2.7–8.7%). While human abstention does not establish that these questions are inherently unanswerable from a single image, the different response patterns highlight the importance of assessing observational uncertainty rather than spatial answer accuracy alone.