acceptodds
Under review as a conference paper at ICLR 2027

Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?

Abstract

Spatial reasoning benchmarks typically evaluate whether vision-language models can derive the correct answer from a visual observation. Yet in real 3D environments, the observation itself may be unreliable: occlusion can remove task-relevant evidence, while perspective can make visible geometry misleading. Reliable spatial reasoning therefore requires more than answering a question correctly. A model must also assess whether its current observation provides sufficient and trustworthy evidence for that answer. We introduce SpatialUncertain, a controlled evaluation framework for studying viewpoint-dependent observational uncertainty. By holding the underlying 3D scene fixed while systematically varying the observation, we study two complementary failure modes: missing evidence caused by occlusion and misleading evidence caused by perspective. We further evaluate whether models can recognize when the current view is unreliable and identify a more informative observation. Across eight open- and closed-source vision-language models from the Qwen2.5-VL, InternVL3, GPT, and Gemini families, we find that model behavior does not track the reliability of visual evidence. Models do not reliably become more cautious as evidence disappears, and under perspective conflict, their judgments increasingly follow projected appearance rather than the unchanged physical 3D relation. Internal analysis suggests a corresponding representational asymmetry: projected 2D relations are readily available, whereas the underlying physical 3D relation is barely decodable. Moreover, models that can identify an informative viewpoint when explicitly asked often fail to recognize when such an additional view is needed. These failures are not fully resolved by prompting or fine-tuning, and providing a better viewpoint is substantially more effective than adding depth information to the same misleading observation. Our results identify assessing the reliability of visual observations as a distinct and missing component of current spatial reasoning evaluation.

Primary area:
datasets and benchmarks
Keywords:
Spatial Reasoning, MLLM, Uncertainty

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UkRCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SpatialUncertain to test whether VLMs recognize when visual evidence is missing or misleading. It uses controlled 3D scenes to vary occlusion and perspective, and evaluates answering, abstention, and view selection across eight models. Models often answer despite missing evidence or follow apparent rather than physical size. The authors support their findings with internal analyses, a small real-image study, and prompting, fine-tuning, and additional-view experiments.

Critical strengths
  1. The benchmark question setup is interesting and make the study possible. And there's not many benchmarks about the uncertainty of VLMs spatial cognition.
  2. The authors check the main findings through several methods, including real-image viewpoint consistency and answer-format comparisons.
Critical weaknesses
  1. The model coverage is incomplete. The authors evaluate eight models, but omit newest GPT and the Claude family, despite their availability before submission. I’ve noticed that the latest GPT and Claude models have improved significantly in spatial ability. Evaluating them may reveal some different findings.
  2. Most of the evidence comes from synthetic scenes. The real-image study includes only 35 object pairs and checks size judgments across viewpoints.
Questions for authors
  1. Can humans reliably answer the perspective-conflict questions from the single image? Have you done some human verifications?
  2. What was the evaluation cutoff date? Why were GPT-5.5 and vision-capable Claude models omitted, and do the main findings hold for them?
2 comments

We thank the reviewer for the constructive feedback. We appreciate the reviewer's recognition that the benchmark setup makes this study possible, that few benchmarks address the uncertainty of VLMs' spatial reasoning, and that our main findings are checked through several methods. The comments led to three changes in the revised paper:

  • Three newer models (GPT-5.5, Claude-Sonnet-5.5, Claude-Fable-5.1) are added to Table 1 (W1, Q2).
  • The real-image study is expanded from 35 to 80 object pairs (W2).
  • A human study on perspective-conflict questions is added (Q1).

We respond to each point below.

W1 & Q2. Evaluation on newer GPT and Claude models.

Our initial evaluation was conducted in April 2026, and the model selection was finalized at that time. To address the reviewer's concern, we have extended the evaluation to GPT-5.5 and two Claude models, following the same protocol. They are now included in Table 1.

ModelOcc-Ans ↑Occ-Unans ↑Pers-Ans ↑3D-Cons. ↑Abstain2D-Foll. ↓
GPT-5.570.28.540.53.224.172.6
Claude-Sonnet-5.568.422.659.122.217.558.4
Claude-Fable-5.168.624.169.235.41.759.3

Claude-Fable-5.1 achieves the highest 3D-consistent rate among all evaluated models (35.4%), surpassing the previous best, Gemini-2.5-Flash (32.5%). However, our main findings still hold:

  • Models continue to rely heavily on 2D projections. All three newer models follow misleading 2D projections in 58–73% of perspective-conflict cases.
  • Abstention remains poorly aligned with visual evidence. Claude-Fable-5.1 abstains in only 1.7% of conflict cases. GPT-5.5 achieves only 40.5% accuracy on answerable perspective questions and abstains in 24.1% of perspective-conflict cases, yet produces the appropriate abstention response in only 8.5% of unanswerable occlusion cases. This further highlights the inconsistency of its uncertainty behavior across observation conditions.

Overall, while newer models show different patterns of accuracy and abstention, they remain vulnerable to misleading projections and do not consistently express uncertainty when visual evidence is insufficient. Within the GPT family, accuracy on answerable occlusion questions has improved from GPT-4o to the GPT-5 models (43.7% → 59.0–70.2%), but this has not been accompanied by better abstention: the GPT-5 models abstain far less on unanswerable questions (63.6% → 8.4–21.1%) and follow the 2D projection more often (35.6% → 61.1–72.6%).

W2. Limited real-image evaluation.

We expanded the real-image set from 35 to 80 object pairs, using the same collection protocol and exclusion rule. The pattern holds at this larger scale: GPT-5-mini gives contradictory size judgments across viewpoints on 66 of 80 pairs (82.5%), and Gemini-2.5-Flash on 57 of 80 (71.3%), each time favoring the object that projects larger. These rates are close to those on the original 35 pairs (86% and 66%).

Q1. Can humans reliably answer perspective-conflict questions from a single image?

We conducted a human study with four undergraduate students (none are co-authors) on 150 perspective-conflict questions and 30 clean controls, using the same images, questions, and answer options as the models, including the option to abstain. Models are evaluated on the same questions. The study is reported in Sec. 4.3 (Table 2).

ModelClean Ans. ↑3D-Cons. ↑Abstain2D-Foll. ↓
Human (4 annotators)86.736.055.38.7
GPT-5-mini76.724.78.760.0
Gemini-3.0-Flash76.726.72.766.7
Claude-Fable-5.166.732.74.753.3
  • Humans perform well on clean perspective views. They achieve 86.7% accuracy when the 2D projection is consistent with the underlying 3D relation.
  • Humans are much less susceptible to misleading 2D projections. Under perspective conflict, only 8.7% of human responses follow the projection, compared with 53.3–66.7% for the models. Instead, humans abstain in 55.3% of responses.
  • The key human–model difference lies in how conflicting evidence is handled. The human 3D-consistent rate (36.0%) is close to the best model (32.7%) and above the others (24.7–26.7%). In the remaining cases, humans acknowledge uncertainty, whereas models follow the misleading projection.

These findings suggest that perspective-conflict views are challenging even for human observers. Humans achieve 36.0% 3D-consistent responses and abstain in 55.3% of cases, while only 8.7% of their responses follow misleading projections. In contrast, models frequently commit to projection-following answers (53.3–66.7%) and rarely abstain (2.7–8.7%). While human abstention does not establish that these questions are inherently unanswerable from a single image, the different response patterns highlight the importance of assessing observational uncertainty rather than spatial answer accuracy alone.

Dear Reviewer UkRC,

As the discussion period progresses, we would like to gently follow up on our rebuttal.

We greatly appreciate your constructive feedback, which guided us to add evaluations for the latest GPT-5.5 and Claude models, expand the real-image study to 80 pairs, and conduct the human verification study you suggested.

Could you please let us know if our response has properly addressed your concerns? We would be happy to provide any further clarifications if needed.

Thank you again for your time and effort in reviewing our paper!

Best regards, Authors

Reviewer_6zYfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SpatialUncertain, a benchmark evaluating whether VLMs recognize when visual observations provide insufficient or misleading evidence for spatial reasoning. Using controlled 3D scenes, the authors systematically vary occlusion and camera perspective while preserving the underlying geometry (Figs. 1, 2 and 3). Across eight VLMs, they find that models frequently fail to abstain under occlusion and rely on misleading 2D projections rather than physical 3D relations (Table 1, Figs. 5 and 6). The key takeaway is that current VLMs struggle to assess the reliability of visual evidence, and that obtaining more informative viewpoints can be more effective than attempting to reason from the same unreliable observation (Fig. 8).

Critical strengths
  1. Careful benchmark design. Holding the underlying 3D geometry fixed while varying observation reliability helps isolate the effects of occlusion and perspective distortion on model behavior (Figs. 2 and 4). This makes the observed failures more interpretable than evaluations on unrelated images.

  2. Good empirical evidence across models and conditions. The evaluation covers eight VLMs and demonstrates systematic sensitivity to occlusion severity and perspective distortion (Table 1 and Fig. 6).

Critical weaknesses
  1. Limited investigation of active perception as a solution to SpatialUncertain. The paper primarily studies passive perception, where models must reason from fixed visual observations (Table 1, Figs. 5, 6). However, in realistic embodied settings, an agent can actively move to acquire better viewpoints. For instance, in Fig. 1(c), a human could simply change position to better determine the physical sizes of the benches. Similarly, a capable multimodal model such as GPT Astra, equipped with appropriate navigation and camera-control tools, might resolve such ambiguities through active exploration. Given that the simulated environments support camera control, evaluating tool-equipped agents would be a natural and important extension. Without this, it remains unclear how much of the observed failure persists when models can actively gather the evidence needed to answer correctly.
Questions for authors

Please see Weaknesses.

Additional question:

  1. Human evaluation. Can the authors include a human baseline to compare human and model performance? This would help contextualize the difficulty of the benchmark.

  2. The introduction needs some rewriting, currently it reads like a laundry list of observations.

====

Post Rebuttal

Thank you for the additional results. The camera-tool experiment directly addresses my concern on active perception, and the human study helps contextualize the benchmark difficulty. The introduction reads better now. I maintain my positive rating.

3 comments

We thank the reviewer for the constructive review. We appreciate the reviewer's recognition of the benchmark design, which holds the 3D geometry fixed while varying observation reliability, and of the evidence across models and conditions. The comments led to three changes in the revised paper:

  • A new camera-tool experiment, including GPT-6 Astra, tests active perception directly (W1; Sec. 4.4 and Appendix A.4).
  • A human study on perspective-conflict questions is added (Q1; Sec. 4.3, Table 2).
  • The introduction is rewritten around one central finding (Q2).

We respond to each point below.

W1. Limited investigation of active perception.

We agree that active perception is the natural way to resolve these cases. We agree that active perception is a natural extension of our passive evaluation. Following the reviewer's suggestion, we ran a new experiment with camera-control tools, including GPT-6 Astra. New experiment with camera tools. We evaluated GPT-5.5 and GPT-6 Astra on 200 wall-object conflict cases, in which the physically smaller object appears larger because of perspective. Each model is tested in two settings: answering from the single conflict view, and active perception, where it can move the camera with tools before answering. The models are given a single tool, move_camera(direction, steps), which moves the camera left or right along a fixed lateral track of 5–6 viewpoints that includes an equidistant reference view. Each episode allows at most six tool calls, and the model decides whether to move, how far, and when to answer. The prompt only describes the tool: it does not say that the initial view may be misleading or where the reference view is.

ModelSetting3D-Cons. ↑Abstain2D-Foll. ↓Avg. movesNo move
GPT-5.5Single view1.50.098.5––
GPT-5.5+ camera tools46.50.053.51.3425.5%
GPT-6 AstraSingle view10.50.588.0––
GPT-6 Astra+ camera tools75.50.019.51.876.0%

From a single view, both models follow the projection and almost never abstain. GPT-5.5 follows the misleading projection in 98.5% of cases and GPT-6 Astra in 88.0%. Even the newest model does not signal that this view is unreliable.

Active perception helps substantially, which confirms the reviewer's intuition. With camera tools, the 3D-consistent rate rises from 1.5% to 46.5% for GPT-5.5 and from 10.5% to 75.5% for GPT-6 Astra.

The failure is reduced but not removed, for two reasons. First, models do not always look: GPT-5.5 answers without moving the camera on 25.5% of questions and GPT-6 Astra on 6.0%, and none of these answers is correct. Second, looking is not always enough: 52.3% of GPT-5.5's errors and 75.5% of GPT-6 Astra's errors occur after the model has moved the camera.

Active perception has a cost. GPT-5.5 uses 1.34 camera moves per question on average and GPT-6 Astra uses 1.87. The table below gives the share of all 200 questions that a model answers correctly using at most k moves.

Modelk = 0k = 1k = 2k = 3Any
GPT-5.50.019.534.043.546.5
GPT-6 Astra0.018.057.573.575.5

The two models are similar within one move, and GPT-6 Astra's advantage comes from questions on which it uses two or three moves.

Relation to AbstainViewSel. AbstainViewSel is a single-step form of the same decision: a model must first recognize that the current view is insufficient and then choose a better one. The new experiment extends this to multi-step camera control, and the revised Table 1 reports the abstain-then-select route separately with a cost-aware score.

The experiment is reported in Sec. 4.4, with the full protocol and results in Appendix A.4 (Tables 9 and 10). It covers wall-object cases with lateral camera movement only; free navigation in the scene is an important direction for future work.

Q1. Human evaluation.

We conducted a human study with four undergraduate students (none are co-authors) on 150 perspective-conflict questions and 30 clean controls, using the same images, questions, and answer options as the models, including the option to abstain. Models are evaluated on the same questions.

ModelClean Ans. ↑3D-Cons. ↑Abstain2D-Foll. ↓
Human (4 annotators)86.736.055.38.7
GPT-5-mini76.724.78.760.0
Gemini-3.0-Flash76.726.72.766.7
Claude-Fable-5.166.732.74.753.3

Humans answer clean views reliably (86.7%). On conflict views, they give the 3D-consistent answer in 36.0% of responses, abstain in 55.3%, and follow the projection in only 8.7%. Models reach similar 3D-consistent rates (24.7–32.7%) but rarely abstain (2.7–8.7%), and most of their remaining answers follow the projection (53.3–66.7%). These results show that perspective-conflict views are challenging even for human observers, who frequently abstain rather than follow misleading projections. In contrast, models tend to commit to answers based on projected appearance. The main human–model difference is how this is handled. Humans withhold judgment, whereas models answer from projected appearance. The study is reported in Sec. 4.3 (Table 2). It covers the perspective setting; for occlusion, answerability was verified by human reviewers during benchmark construction (Sec. 3.5).

Q2. The introduction reads like a list of observations.

We have rewritten this part of the introduction. It now states one central finding, that model behavior does not track the reliability of the visual evidence, and organizes the results around it in a single paragraph: how models behave under occlusion and perspective conflict, how humans handle the same views, and what helps (better evidence helps more than reasoning over the same view).

Reviewer_6zYfOpenReview ↗

Reviewer response

Thank you for the additional results. The camera-tool experiment directly addresses my concern on active perception, and the human study helps contextualize the benchmark difficulty. The introduction reads better now. I maintain my positive rating.

Reviewer_iTBUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a controlled benchmark for evaluating whether vision-language models (VLMs) respond appropriately to missing or misleading visual evidence in spatial reasoning. With synthetic 3D indoor scenes, the benchmark varies target visibility via object occlusion and induces conflicts between projected 2D appearance and 3D physical size via viewpoint changes. It also introduces viewpoint-selection tasks to assess whether models can identify informative observations and recognize when additional evidence (new views) is needed. Experiments across eight VLMs show inconsistent abstention under occlusion and frequent reliance on projected size under perspective conflict. The authors supplement these evaluations with representation probes, attention analysis, a small real-image consistency study, and prompting, fine-tuning, and evidence interventions. Some models can select informative viewpoints when explicitly asked but struggle in the joint answer-or-abstain-and-select task. Fine-tuning reduces projection-following largely by increasing abstention, while an additional reference view improves physical-size judgments more than the tested depth inputs.

Critical strengths
  1. The paper goes beyond answer accuracy to examine how VLMs respond when visual evidence is missing or misleading. Its controlled occlusion and viewpoint changes help expose failures that conventional spatial benchmarks may overlook.

  2. Across eight models, physical-size judgments frequently follow misleading 2D projections. The real-image consistency study provides additional evidence that this behavior extends beyond synthetic scenes, making the finding relevant to practical spatial reasoning.

Critical weaknesses
  1. The size-comparison question templates are ambiguous. Figure 2 (top) uses the word “larger” without specifying whether it refers to physical or projected size, or whether the comparison concerns height, width, area, or volume. This ambiguity also appears in the QA data in supplementary material, e.g., occ_0004 asks, “Compared to the wall shelf, is the refrigerator larger or smaller?” Such wording may confound task interpretation with the capabilities being evaluated, particularly when apparent-size answers are classified as 2D-following errors. The authors should clarify the intended quantity in both questions and options, and verify the findings using explicitly disambiguated templates.

  2. The joint metric may reward models for avoiding difficult but answerable questions. A misleading 2D projection does not necessarily mean that the image lacks enough information to compare physical sizes, as the authors acknowledge (§3.3, lines 257–260). Walls, floors, and other scene context may provide enough cues to correct for perspective. However, the dataset does not identify which conflict images can be answered from these cues and which truly require another view. AbstainViewSel gives the same score for answering correctly from the original image and for abstaining and choosing the reference view (§3.4, lines 302–307). A model that cannot use the available 3D cues could therefore receive the same score as one that successfully reasons about perspective. The evaluation should separate cases where another view is necessary from cases where the model simply fails to solve the original question.

  3. In Figure 7(a), greater attention to the projected-larger object does not necessarily mean that the model relies on projected appearance. Larger objects occupy more pixels and are often more visually salient. Although the authors normalize attention by object area, the remaining bias could still reflect general visual salience that also occurs with unrelated question prompts. It does not establish that the model bases its physical-size judgments on projected size. And I have a similar concern about Figure 7(b). The objects in Holodeck-generated scenes used by the authors are low-poly assets with very simple textures and colors. Most objects have only a few uniform colors, making 2D relations easy for the probing classifier to learn from the extent of similarly colored regions in the input image. However, recovering physical 3D relations requires more reasoning and integration of scene context. The resulting information may therefore be represented elsewhere in the model, such as in question or answer token states, rather than in the mean-pooled features of patches within the object boxes. I find the reported difference in probing accuracy unsurprising: it shows that 2D relations are easier to read from these particular features, but does not demonstrate that the model relies on projected size when answering 3D relation questions.

Questions for authors
  1. The scores in Table 1 do not match the definition of AbstainViewSel. According to §3.4 (lines 302–307) and Appendix A.2.2 (lines 839–840), the joint score includes all correct Stage-1 answers, plus cases where the model abstains and selects the correct reference view. It therefore cannot be lower than the Stage-1 3D-Cons. score on the same samples. However, Qwen2.5-VL-32B scores 22.8% on 3D-Cons. but only 3.8% on AbstainViewSel, while InternVL3-8B scores 29.0% and 0.0%, respectively. Do these columns use different samples or predictions, or is there a reporting error?

  2. In indoor scenes, a human should be able to infer the 3D physical-size relation from a single image using object appearance, placement, and surrounding context. I would expect a strong VLM could achieve this too. However, AbstainViewSel gives equal credit for answering correctly from the original image and for abstaining and selecting a reference view. In my view, needing an additional view reflects weaker 3D reasoning and should be penalized by the metric rather than receive equal credit. How do the authors justify this scoring choice?

8 comments

We thank the reviewer for the careful and detailed review. We appreciate the reviewer's recognition that our benchmark goes beyond answer accuracy to examine how VLMs respond when visual evidence is missing or misleading, and that the tendency to follow misleading 2D projections, supported by the real-image study, is relevant to practical spatial reasoning. The comments led to several concrete changes in the revised paper:

  • Table 1 now reports the two routes of AbstainViewSel separately with a cost-aware score, and the reporting error for the three open-source models is corrected (W2, Q1, Q2).
  • A wording-control experiment with explicitly physical questions is added (W1).
  • The claims about attention and probing are revised after two additional controls (W3).

We respond to each point below.

W1. Ambiguity in size-comparison questions.

We agree that some questions could be more explicit about physical versus apparent size, and we have conducted additional experiments to examine whether this ambiguity affects our findings.

Perspective question design. We would like to first clarify that Figure 2 (top) illustrates the occlusion setting instead of the perspective setting. In the perspective setting, our questions already specify the comparison dimension, and some explicitly refer to physical size. For example, our original questions include:

  • Physical size (persp_0002): "There are two storage cabinets in this image. Based on this image, can you determine their physical sizes?" The options include "The storage cabinet on the left is physically larger" and "They are the same physical size."
  • Height (persp_0021): "There are two wall arts in this image. Based on this image, which one is taller?"
  • Width (persp_0022): "There are two wall arts in this image. Based on this image, which one is wider?"

For unequal-object pairs, the ground-truth labels are based on the corresponding physical height or width. For identical-object pairs, both objects share the same 3D asset, so their physical dimensions are identical. Figure 2 (bottom) shows these dimension-specific options (e.g., "Right is wider"). The remaining ambiguity is whether "wider" or "taller" refers to physical or projected size, which we test below.

New wording-control experiment. We re-evaluated all 477 perspective-conflict questions that did not explicitly specify physical size, using GPT-5-mini and Gemini-2.5-Flash. We kept the images, ground-truth labels, and evaluation settings unchanged, while making the questions and options explicitly refer to physical size (e.g., "Which one is physically wider?" instead of "Which one is wider?").

ModelWording3D-Cons. ↑Abstain2D-Foll. ↓
Gemini-2.5-FlashOriginal24.52.568.1
Gemini-2.5-FlashExplicit physical19.313.264.6
GPT-5-miniOriginal21.40.077.8
GPT-5-miniExplicit physical26.68.064.4

Findings. Explicit physical-size wording reduces 2D-following responses and increases abstention for both models. However, it does not consistently improve 3D reasoning accuracy. GPT-5-mini improves from 21.4% to 26.6% in 3D consistency, while Gemini-2.5-Flash decreases from 24.5% to 19.3%. On unequal-object pairs, accuracy remains similar for GPT-5-mini (28.8% vs. 28.1%) and decreases for Gemini-2.5-Flash (29.8% vs. 25.7%). Importantly, both models still follow misleading 2D projections in more than 64% of cases, even with explicit physical-size wording. These results show that question wording affects model responses but does not explain the main perspective-related failures. This wording-control experiment is added to the revised paper (Appendix A.3.3, Table 5).

Occlusion. In the occlusion setting (e.g., occ_0004), "larger/smaller" refers to overall physical size, defined using the volume of the 3D bounding box. Unlike perspective ambiguity, this setting does not introduce an explicit conflict between projected and physical size. We nevertheless tested explicit physical-size wording on 300 sampled questions and found similar accuracy on clean (63% vs. 61%) and partially occluded cases (51% vs. 52%).

W2 & Q2. The joint metric may reward avoiding answerable questions.

We agree that a misleading 2D projection does not necessarily make a question unanswerable, as contextual cues may still support correct 3D reasoning. We also acknowledge that AbstainViewSel gave equal credit to correct reasoning from the original image and abstention followed by selecting the reference view. We have revised the evaluation accordingly: Table 1 now reports the two routes separately together with a cost-aware score, and we added a human study that indicates how often conflict views can be resolved from a single image.

(i) The two routes are now reported separately.

In the revised Table 1 and Sec. 3.4, AbstainViewSel is split into (a) correct 3D answers from the original image, and (b) abstention followed by correct reference-view selection.

Model(a) Direct 3D-Cons.(b) Abstain → correct viewAbstainViewSel
Random25.05.030.0
Qwen2.5-VL-7B3.85.69.4
Qwen2.5-VL-32B22.83.826.6
InternVL3-8B29.00.029.0
GPT-4o15.011.026.0
GPT-5-mini26.67.534.1
GPT-5.415.612.928.5
GPT-5.53.222.125.2
Gemini-2.5-Flash32.51.333.8
Gemini-3.0-Flash22.62.224.8
Claude-Sonnet-5.522.211.834.0
Claude-Fable-5.135.40.435.8

The two components contribute very differently across models. View selection accounts for almost 90% of the joint score of GPT-5.5 and over 40% for GPT-5.4 and GPT-4o, compared with only 1–4% for Claude-Fable-5.1 and Gemini-2.5-Flash. Reporting these components separately makes this distinction explicit. Moreover, abstention alone earns no credit: the model must correctly select the reference view from five candidates.

(ii) Requesting another view is now discounted.

To avoid treating direct reasoning and requesting another view as equally valuable, following the reviewer's suggestion, we introduce a discounted variant, Score(λ) = (a) + λ × (b), where λ controls the credit assigned to successful view selection.

Modelλ = 1λ = 0.5λ = 0
Random30.027.525.0
Qwen2.5-VL-7B9.46.63.8
Qwen2.5-VL-32B26.624.722.8
InternVL3-8B29.029.029.0
GPT-4o26.020.515.0
GPT-5-mini34.130.426.6
GPT-5.428.522.115.6
GPT-5.525.214.23.2
Gemini-2.5-Flash33.833.232.5
Gemini-3.0-Flash24.823.722.6
Claude-Sonnet-5.534.028.122.2
Claude-Fable-5.135.835.635.4

Discounting view selection changes the comparison among models. For example, GPT-5-mini is ahead of Gemini-2.5-Flash at λ = 1 (34.1 vs. 33.8) but behind it at λ = 0.5 (30.4 vs. 33.2), and GPT-5.5 drops from 25.2 to 14.2. Claude-Fable-5.1 leads under every setting because almost all of its score comes from direct answers.

(iii) How often can conflict views be resolved from a single image?

We conducted a human study with four undergraduate students (none are co-authors) on 150 perspective-conflict questions and 30 clean controls, using the same images, questions, and answer options as the models. We also re-evaluated models on this subset for a direct comparison. The study is reported in Sec. 4.3 (Table 2) of the revised paper.

ModelClean Ans. ↑3D-Cons. ↑Abstain2D-Foll. ↓
Human86.736.055.38.7
GPT-5-mini76.724.78.760.0
Gemini-3.0-Flash76.726.72.766.7
Claude-Fable-5.166.732.74.753.3

Humans perform well on clean views (86.7%) but find perspective-conflict questions much harder, providing 3D-consistent answers in 36.0% of cases. Rather than relying on misleading 2D projections, they abstain in 55.3% of cases and follow the projection in only 8.7%. In contrast, models achieve comparable or lower 3D-consistent rates (24.7–32.7%), but rarely abstain (2.7–8.7%) and frequently follow the misleading projection (53.3–66.7%).

These results suggest that many conflict views are difficult to resolve reliably from a single image, even for humans. More importantly, humans and models respond differently to this uncertainty: humans tend to withhold judgment, while models often rely on projected appearance. While human abstention does not establish that an additional view is strictly necessary, it highlights the importance of recognizing when the current evidence may be insufficient. Our revised metric therefore distinguishes direct 3D reasoning from successful view selection and discounts the latter to reflect the cost of obtaining additional evidence.

W3. Attention and representation probing.

Thank you for pointing this out. We ran additional experiments to check both concerns. Based on the results, we clarify what Fig. 7 shows and revise our claims accordingly.

Attention (Fig. 7a). We repeated the analysis on the same 198 images using four unrelated questions, such as wall color, indoor/outdoor classification, and object counting. The results are very similar across questions. In the middle layers, the projected-larger object receives about 1.6× more area-normalized attention, even when the question has nothing to do with size. This suggests that the attention difference is mainly related to visual salience, rather than physical-size reasoning.

Probing (Fig. 7b). We ran two additional experiments (Table 8 in the revised appendix).

  1. Answer-token probing. Following the reviewer's suggestion, we probed the final answer token instead of object-region image tokens. The gap remains: at LM layer 20, the probing accuracy is 87.4% for the 2D relation and 63.3% for the 3D relation, compared with 89.6% and 65.6% from object-region image tokens. The answer token therefore does not expose more 3D information than the features we originally used. In both cases the 3D relation is decodable above the majority baseline (about 53%), so some of this information is present, but it is less linearly accessible than the 2D relation.
  2. Geometry-only baseline. We also trained a classifier using only the sizes and positions of object bounding boxes. It achieves 96.3% for the 2D relation and 75.6% for the 3D relation. The conflict set includes both reversed and non-reversed size relations, so simple geometric cues can also predict some 3D labels. This supports the reviewer's point that 2D relations are easy to predict in our dataset, even without model features.

Based on these results, we have revised the corresponding statements in the abstract, introduction, and Sec. 4.3. We removed the statements that the model "lacks a robust internal signal" for physical size and that it relies on projected appearance because the 3D relation is weakly represented. Sec. 4.3 now presents Fig. 7 as an exploratory analysis, describes Fig. 7(a) as a general attention bias, and states that Fig. 7(b) shows what is accessible at the probed positions, not which information the model uses.

Our main conclusion does not depend on Fig. 7. That models' physical-size answers follow the projected relation is shown by the controlled manipulation in Fig. 6b: with the 3D scene held fixed, answers shift toward the projected relation as the projected-size ratio grows.

Q1. Clarification and correction of AbstainViewSel scores.

Thank you for catching this inconsistency. The reviewer is correct that AbstainViewSel should include both correct Stage-1 answers and cases where the model abstains and selects the correct reference view. We identified a reporting error in Table 1 for the three open-source models: we inadvertently reported only the second component, without adding the correct Stage-1 predictions. The corrected AbstainViewSel scores are:

ModelCorrected AbstainViewSel (%)
Qwen2.5-VL-7B9.4
Qwen2.5-VL-32B26.6
InternVL3-8B29.0

We apologize for the oversight. These values are corrected in the revised Table 1, which now also reports the two components in separate columns, and Sec. 3.4 defines both components explicitly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.