acceptodds
Under review as a conference paper at ICLR 2027

Same Space or Not: Benchmarking Spatial Intelligence Through Cross-View Contradictions

Abstract

Spatial intelligence, the capacity to represent and reason about physical space, underlies embodied perception and real-world visual understanding. Existing benchmarks probe it through spatial relations, perspective-taking, and mental modeling. What remains underexamined is whether a model can maintain a coherent scene representation across views, since pairwise comparison alone cannot reveal which view departs from the truth. We thus introduce Same Space or Not (SSoN), a benchmark of cross-view contradiction resolution comprising 1,000 questions from seven indoor scene sources and six categories of controlled 3D edits. Each question presents three views of an unchanged scene and one view of an edited variant. Models must identify the inconsistent view and explain the contradiction without a designated trusted reference or a marked target. Our construction process iteratively translates human review into generation constraints. Automated checks and exhaustive human validation screen out both trivial visual shortcuts and questions lacking sufficient evidence for a unique answer. Humans achieve 83.3% accuracy, compared with 39.7% for the strongest vision-language models (VLMs), which further drops to 34.9% when requiring valid visual explanation for the correct selection. We investigate prompting, camera information, and 3D reconstruction as routes for improvement. On a 100-question intervention set with a 24% baseline, prompting and 3D reconstruction remain below 50% answer accuracy, except when ground-truth camera information is supplied, which reaches 61% answer accuracy but only 48% True solve accuracy. On a held-out 201-question test set, Qwen2.5-VL-7B achieves 21.89% accuracy, 33.33% after answer-only fine-tuning, and 41.79% after fine-tuning with both rationale and answer supervision. These results identify cross-view contradiction resolution as a substantial weakness of current VLMs that can be partly addressed by targeted supervision. The 1,000 questions and fine-tuned checkpoints will be made public.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, benchmark, spatial reasoning, multi-view consistency, 3D scene understanding, evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_3zW5OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Proposes a benchmark assessing the effectiveness of a model to detect the odd scene from a set of views where all except one are views of the same environment. Benchmark is evaluated on humans, basic models and models that have been enhanced to be more resilient to this task.

Critical strengths

-avoids focus on just pairwise comparisons

-provides a comprehensive collection of edits to constitute the odds scene out

-part of the task involves explaining the contradiction, making it more rich and challenging

-highlights surprising finding between the ability of humans versus machines for this task

Critical weaknesses

-benchmark is confined to indoor scenes

-focuses just on visual input without considering other types of sensor data that might assist with the task

Questions for authors

What would be the challenges to extend the benchmark to include outdoor scenes and other types of sensor data beyond visual??

1 comment

We sincerely thank you for your careful reading of our paper and for your encouraging assessment. Both points you raise are fair and inspiring, and we fully agree with them. During the rebuttal period we followed up on both.

1. Outdoor scenes

Prompted by your question, we built ten outdoor pilot items from TDW and Poly Haven assets, each set in a bounded space, and evaluated them under the protocol and prompt of the main benchmark. On these items, human accuracy is 7 of 10. The best-performing model, GPT-6 Astra Pro, answers 2 of 10 correctly, with a valid explanation for both. This follows the same pattern as the 1,000-item results, where humans reach 83.3% and the same model 39.7%. Although at pilot scale, the result indicates that the task carries over to outdoor scenes.

The main challenge is preserving a unique answer. Indoors, the walls, corners and doorways of a room tie all four views to one enclosed space (the room-reference requirement, Section 3.1 and Appendix A.4). Outdoors this shared reference is no longer given. In an open or repetitive scene, a view that disagrees with the others may simply show another location, so the contradiction can no longer be attributed to a single view. The bounded spaces of our pilot restore this reference. Real outdoor scenes add one more consideration, because they are never static. Light and shadow shift naturally, and foliage or water may move between shots, all of which we define as acceptable noise (Appendix B.9). With that definition in place, the benchmark can extend to real outdoor scenes as well.

2. Other sensor data

You are right that sensor data beyond visual input could assist with the task. The one input of this kind in our experiments is the ground-truth camera intrinsics and extrinsics of the four views (the Cam condition, Section 4.3). These parameters tell the model how the cameras are positioned and oriented relative to one another, geometric information that it would otherwise have to infer from the pixels and that a physical device would obtain from its calibration and pose tracking. The outcome supports your view, as Cam brings the largest overall improvement among all the interventions in Table 4. For convenience, the table is reproduced below.

Category (n)BaseCompSceneTaxComp+TaxReconCam
Overall (100)24.00 / 12.0040.00 / 23.0037.00 / 29.0044.00 / 32.0048.00 / 34.0047.00 / 34.0061.00 / 48.00
Swap (18)66.67 / 33.3377.78 / 55.5672.22 / 66.6772.22 / 61.1177.78 / 61.1177.78 / 66.6766.67 / 61.11
Move (20)10.00 / 5.0020.00 / 20.0020.00 / 20.0030.00 / 25.0025.00 / 15.0015.00 / 10.0070.00 / 70.00
Rotate (18)11.11 / 5.5616.67 / 0.0022.22 / 11.1116.67 / 11.1122.22 / 5.5616.67 / 5.5638.89 / 22.22
Mirror (15)13.33 / 6.6740.00 / 13.3333.33 / 20.0066.67 / 53.3366.67 / 53.3346.67 / 40.0040.00 / 20.00
Delete (12)16.67 / 8.3325.00 / 8.3325.00 / 25.0066.67 / 33.3375.00 / 50.0075.00 / 50.0075.00 / 58.33
Detail (17)23.53 / 11.7658.82 / 35.2947.06 / 29.4123.53 / 11.7635.29 / 29.4164.71 / 41.1876.47 / 52.94

Each cell gives answer accuracy and True solve in percent (Acc / TS). Thank you so much for your suggestion, which led us to analyze these results further. Overall, Cam helps more than any other intervention we tested.

The ten outdoor items are available at https://huggingface.co/datasets/AnonymousAnonymousAccount/Anonymous as SSoN_Outdoor_Pilot_Data.zip. In the revised paper, we will add them to the appendix. We are deeply grateful for your time and for your meaningful contributions to SSoN.

Reviewer_KxsdOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Same Space or Not (SSoN), a 1,000-item benchmark in which a model identifies and explains one inconsistent view among four rendered views of an indoor scene. Six controlled edit categories and viewpoint distractors test whether models can reconcile observations without a designated trusted view. The authors combine automated screening with three-stage human verification and evaluate six VLMs, 17 vision encoders, and humans. The strongest VLM achieves 39.7% answer accuracy versus 83.3% for humans, falling to 34.9% when its explanation must provide valid evidence. Diagnostic interventions and fine-tuning on disjoint rooms explore ways to improve performance.

Critical strengths
  1. SSoN adds a useful evaluation target: attributing a contradiction within a set of views, rather than only detecting a difference between a pair. Its controlled edits, corroborating anchors, and viewpoint distractors make this distinction concrete. The seven scene sources and item-level checks for answer uniqueness and specified shortcuts are substantial dataset contributions (Sections 3.1–3.3; Appendix A.4).
  2. The experiments provide informative diagnosis beyond an aggregate leaderboard. Table 2 reveals a particularly large rotation gap (at most 9.54% for the evaluated VLMs versus 76.01% for humans), and the held-room fine-tuning study establishes that performance can improve. The paper also reports order sensitivity and appropriately qualifies the additional rationale-supervision gain given its single seed and p = 0.064 (Section 4.5; Appendix A.1).
Critical weaknesses
  1. The reliability of the True solve labels needs a broader independent check. Every correct-choice response receives human adjudication, which is reassuring, but the second-human agreement result covers only unanimously agent-TS responses (Appendix A.6, p. 16). It therefore does not measure agreement on unsupported explanations or agent-disputed cases, where the sufficiency boundary is most consequential. A stratified, independent second-human audit across final labels, reporting sample counts and agreement, would strengthen this proposed metric. This is a fixable concern and does not undermine the answer-accuracy gap.
  2. Reproducibility is currently limited by the absence of a linked code repository or downloadable benchmark materials. The reproducibility statement promises the questions and checkpoints, and the appendices describe the protocols in considerable detail, but the item metadata, split assignments, grading records, and implementation cannot yet be inspected. An anonymous release containing these materials and the evaluation/training scripts would make the benchmark contribution verifiable and usable (p. 10; Appendix A.4).
Questions for authors
  1. Could you report an independent second-human audit sampled across final True solve and Partly guessed labels, including agent-disputed responses, with sample counts and agreement? This would clarify how reliably the evidence-sufficiency criterion can be reproduced.
  2. Can you provide an anonymous archive or repository with the benchmark, item metadata, exact split identifiers, grading records, and evaluation/training scripts? Access would allow verification of the released artifacts against the documented protocols.
1 comment

We sincerely thank you for reading our paper so carefully, and for your kind words about the cross-view contradiction attribution task and the diagnostic experiments. Both points you raise are fair and fixable, and during the rebuttal period we addressed each of them.

1. Independent second-human audit of the True solve labels

Following your suggestion, an independent second human with PhD-level training labeled the responses again under the same Appendix A.6 standard. We had this second human cover all 1,368 correct-choice responses of the six models, so both final labels and every automation class are included. The table reports the labels assigned in this audit before any discussion.

Automated agent screening (responses)Original final human labelIndependent second humanAgreement before discussion
Unanimous TS (483)483 True solveTrue solve for all 483483/483 (100%)
Unanimous HALLUCINATION (134)134 Partly guessedPartly guessed for all 134134/134 (100%)
DISPUTE (751)481 True solve, 270 Partly guessedAgreed on all 481 True solve and on 263 of the 270 Partly guessed, labeling the other 7 True solve744/751 (99.07%)

The second human reproduced the original evidence-sufficiency judgment for 1,361 of the 1,368 responses (99.49%). By final label, agreement is 964 of 964 for True solve and 397 of 404 for Partly guessed (98.27%). All seven disagreements lie where you expected the sufficiency boundary to matter most, among the agent-disputed responses labeled Partly guessed. The seven responses all come from GPT-5.6 Sol Pro and have one feature in common. Each indicates a general area where the views disagree but leaves the change to the edited object unstated, which the second human first accepted as sufficient. After discussion, the first and the second human agreed to keep Partly guessed for all seven. The second human also marked four other responses as borderline while agreeing with their labels. These eleven cases are all that came into question among the 1,368 responses, so we believe that mislabeling, residual cases included, is below 1%. The original API responses and the grading records from every stage are released, so the community can examine each case directly and apply the same criterion.

DISPUTE holds just over half of the responses because the automated screening is strict by design. A response is classed as TS or HALLUCINATION only when all ten agent seats agree, and a single differing vote sends it to DISPUTE (Appendix A.6). This strictness cost more human effort, but it made the two unanimous classes dependable, as all 617 of their labels were confirmed in the audit, and we consider the trade-off worthwhile. Every DISPUTE response received a human decision, and the second human confirmed 744 of the 751 (99.07%). DISPUTE therefore reflects the caution of the screening rather than widespread ambiguity in the labels.

We will add this audit to the appendix, with the counts above and the borderline cases.

2. Anonymous release

We fully agree that the materials you list are important to release, and all of them can now be downloaded from https://huggingface.co/datasets/AnonymousAnonymousAccount/Anonymous as separate archives.

MaterialArchive
Benchmark, all 1,000 questionsSSoN_1000_Data.zip
Item metadata, correct options, reference explanationsManifest.zip
Exact split identifiers (700/99/201), evaluation and training scriptscode.zip
Grading recordsSSoN_Independent_True_Solve_Label_Audit.zip
Visible model responsesAPI_Records.zip
Fine-tuned checkpoints of the paper (seed 1)Main_Tuning_Weights_seed1.zip

The six CSV files of the audit archive give, for each response, the automation class, the original human label, the second human's label and the disagreement notes, and the item ID links them to the images and the model's API response. For the split, the build_groups function in Code/prepare_data.py connects questions that share a scene, a scene family or a panel SHA-256 hash, and each connected group stays within one partition (Appendix A.1). The remaining archives and the extra_diagnostic_experiment folders in API_Records.zip were prepared during the review period, and the results in the submission are unchanged.

Thank you again for these two suggestions. Because of them, the True solve metric now rests on a complete second-human audit, and every aggregate figure in the paper can be traced to the released items, responses and labels. We are grateful for your meaningful contributions to SSoN and thank you so much for your time.

Reviewer_caQjOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a new benchmark called “Same Space or Not” to test if vision language models can understand physical space across different viewpoints. Instead of just comparing two images, the test shows four views where three are unchanged, and one has a hidden edit, forcing the model to find the odd one out and explain why. The dataset contains one thousand carefully checked questions from seven indoor scene sources, making sure answers require real spatial thinking rather than simple visual tricks. Results show a huge gap between humans, who get over 83% accuracy, and the best models, which only reach about 40%, dropping even lower when a valid explanation is required. The authors also find that giving models exact camera information or training them with detailed reasoning steps helps improve their scores, though they still fall far short of human ability.

Critical strengths

The benchmark design prevents random guessing by requiring three consistent views to validate the fourth, removing the ambiguity of simple pairwise comparisons; and the validity-guided constraint discovery process ensures high data quality by systematically turning human review failures into automated generation rules, which is also interesting.

Critical weaknesses

The critical weaknesses are:

  1. My first critical weakness is about representation learning of the whole mechanism. The entire benchmark relies on rendered indoor scenes from simulators, which lack the noise, lighting variations, and sensor artifacts found in real photographs. If models are only tested on perfect synthetic renders, their spatial reasoning abilities might not translate to messy, real world embodied tasks. This limits the practical value of the benchmark for developing agents that must operate in actual physical environments. Also, the paper says that providing explicit ground-truth camera information boosts accuracy significantly, but this is an unrealistic crutch for real-world deployment. In practical scenarios, models must infer camera poses from pixels alone, and the benchmark does not sufficiently prove that models can learn this inference. Relying on provided metadata masks the true difficulty of the spatial reasoning task. While fine-tuning a specific model improves accuracy to about 41%, this is still barely above random guessing when considering the four options. The paper admits the gain is only suggestive with weak statistical significance, indicating that the model might just be memorizing specific edit patterns. This raises doubts about whether the model is learning generalizable spatial reasoning or just overfitting to the training data.

  2. Then, comes the evaluation. The true solve metric relies on human reviewers checking if the model explanation is factually valid and sufficient. However, the text mentions that high similarity object names or partial descriptions count, which introduces subjectivity into the grading process. Without a strict, automated way to verify these textual explanations, the scores might fluctuate based on reviewer leniency rather than actual model capability. The 100 question diagnostic set used for prompting and reconstruction tests was specifically chosen to balance certain outcomes from one specific model. This means the results on this subset cannot be fairly generalized to the full 1000-question benchmark or to other models. Consequently, the claims about intervention effectiveness might be skewed by this selected diagnostic set. The benchmark presents four static views simultaneously, ignoring how humans naturally perceive space through continuous movement over time. Real embodied agents navigate spaces sequentially, and testing them only on static, disjointed snapshots misses a crucial dimension of real-world spatial intelligence. This static setup might not accurately reflect the dynamic reasoning required in actual navigation tasks.

  3. I have some concerns about the process too. The reconstruction intervention required rerunning noncompliant responses until they produced a valid evidence archive, which hides the true failure rate of the model. This manual enforcement of compliance inflates the perceived usefulness of 3-dimensional reconstruction. A real world system would not have a human loop to force the model to try again until it succeeds. Then come the augmentations. Although the paper mentions varying edit magnitudes and object sizes, the distribution is heavily skewed, with medium objects making up over half of the dataset. This imbalance might allow models to develop biases toward medium sized objects. It remains unclear if the models are truly struggling with scale or if the medium category is simply driving the average performance.

Questions for authors
  1. Could you provide a small pilot evaluation of top models on a set of real-world, unrendered photographic images with similar spatial edits to prove the findings hold outside simulators?

  2. Is there a way to evaluate models that estimate their own camera poses from the images, and how does their performance compare to the ground-truth camera condition?

  3. Can you run the fine-tuning experiment across multiple random seeds and larger model scales to confirm if the rationale supervision gain is statistically significant and stable?

  4. Could you release the detailed rubric and a sample of graded model explanations so the community can audit the subjectivity of the true solve metric?

  5. Would it be possible to run the prompting and reconstruction interventions on a randomly sampled subset to show that the improvements are not just an artifact of the selected diagnostic set?

  6. What is the actual first-try success rate of the 3D reconstruction workflow before human-enforced reruns, to give a more realistic picture of its current viability?

  7. Can you break down the model performance specifically by the small and large object categories to show if the models are truly struggling with scale?

  8. Do you have any plans or preliminary data on how models perform when the views are presented as a sequential video or a step-by-step navigation path instead of static side-by-side images?

4 comments

Thank you for your detailed questions. We address all eight in order.

Q1. Real-world photographs

We photographed real scenes with physical edits, constructed ten four-view questions, and evaluated them using the main benchmark's prompt and scoring protocol. The photographs contain recognizable features of our institution, so we will release them with the camera-ready version.

We also constructed ten separate outdoor items using TDW and Poly Haven assets. On these rendered outdoor items, humans answered 7/10 correctly. GPT-6 Astra Pro, the strongest VLM on the main benchmark, answered 2/10 correctly and gave valid explanations for both. On the 1,000 indoor items, the corresponding answer accuracies were 83.30% and 39.70%. The 1,000 main items and the ten rendered outdoor items are available at https://huggingface.co/datasets/AnonymousAnonymousAccount/Anonymous.

Weakness 1 and Q2. Camera information and pose estimation

No model in the main benchmark or the fine-tuning evaluation receives scene or camera metadata. The six VLMs in Table 2 receive the four images and the task prompt. Section 4.1 explicitly excludes camera parameters, top-down views, filenames, and edit annotations. Appendix A.1 likewise states that "scene metadata are not model inputs" in fine-tuning (p. 13). The best model's 39.70% accuracy was obtained under this image-only protocol.

Table 4 reports a comparison relevant to your question. Recon performs sparse 3D reconstruction and inspects reconstructed views "without supplied geometry" (Section 4.4). Its prompt states that "No ground-truth camera parameters or additional scene observations are provided" (Appendix B.7, p. 22). Cam supplies the ground-truth camera intrinsics and extrinsics (Section 4.3).

SampleRecon, Acc / True solveCam, Acc / True solve
Original 100 questions (Table 4)47% / 34%61% / 48%
Random 100 questions (Q5)56% / 43%68% / 55%

Cam scored higher on both samples.

Weakness 1 and Q3. Multiple seeds and larger models

Both fine-tuned models significantly improved over zero-shot in the original experiment. On the same 201 held-out questions, accuracy rose from 21.89% to 33.33% with answer-only supervision (p = 0.015) and to 41.79% with rationale and answer supervision (p = 7.9 × 10⁻⁵), under two-sided exact McNemar tests. The entire 95% Wilson interval for 41.79% (35.2% to 48.7%) lies above the four-choice chance level of 25%. The result the paper calls "suggestive" is the additional gain from 33.33% to 41.79% (p = 0.064, Section 4.5). Reviewer Kxsd likewise distinguishes the finding that "performance can improve" from "the additional rationale-supervision gain". The split keeps related rooms and scene families together and checks for identical panel images across partitions, so the test scenes were unseen during training.

Following your suggestion, we repeated both training conditions at seeds 0 and 4. All three seeds use the same 201 test questions and share the same zero-shot baseline. The table reports answer accuracy (%). Each p-value compares the two supervision targets within that seed using a two-sided exact McNemar test.

SeedZero-shotAnswer SFTRationale + answer SFTRationale gain (points)p
021.8924.3841.29+16.929.6 × 10⁻⁵
1 (paper)21.8933.3341.79+8.460.064
421.8927.3640.80+13.430.004
Mean21.8928.3641.29+12.94

Rationale + answer supervision averaged 41.29% accuracy (sample SD 0.50 percentage points) and significantly exceeded zero-shot at every seed (all p < 0.0001). It also exceeded answer-only supervision at every seed, by 8.46 to 16.92 percentage points. The mean additional gain was 12.94 points, and two of the three within-seed comparisons reached p < 0.05. The additional checkpoints are available in Extra_Multiple_Seeds_Tuning_Weights.zip.

The experiments and API calls cost approximately US$5,000, and training larger models exceeded our budget. For context, the original Thinking in Space (Yang et al., CVPR 2025) and BLINK (Fu et al., ECCV 2024) benchmark evaluations report zero-shot results. Multi-SpatialMLLM (Xu et al., CVPR 2026) uses rank-16 LoRA on the language backbone of InternVL2-8B in its main training configuration. Our three-seed experiment uses Qwen2.5-VL-7B-Instruct.

Weakness 2 and Q4. Rubric and graded explanations

The submitted paper gives the grading criteria in Appendix A.6 (pp. 15 to 16) and the reviewer instructions, including the three rubric questions and voting table, in Appendix B.9 (pp. 23 to 31). The rule states that "sufficient partial descriptions count." For example, in a swap, correctly locating one object's displacement can suffice. Claiming that the object vanished from the room when it remains elsewhere fails the criterion.

Every correct-choice response underwent automated screening. An automated TS or HALLUCINATION classification required agreement across all ten seats. Any disagreement or DISPUTE vote routed the response to DISPUTE. A human first judged the response blind to the automated classification, then examined the agents' evidence before assigning the final label.

An independent rater with PhD-level training has now applied the same standard to all 1,368 correct-choice responses from the six models. Before discussion, the rater agreed with 1,361/1,368 original labels (99.49%), including 964/964 True solve labels and 397/404 Partly guessed labels (98.27%). All seven disagreements came from agent-disputed responses. The full responses, screening classes, both human labels, and disagreement notes are in SSoN_Independent_True_Solve_Label_Audit.zip. The visible API responses are also in API_Records.zip.

Q5. Randomly sampled subset

Following your suggestion, we sampled 100 distinct questions uniformly from the full 1,000 without conditioning on edit type or model outcome. We evaluated all seven conditions. Base reuses Astra Pro's main-benchmark responses. Twelve items overlap with the original diagnostic set and have the same visible responses across the six API conditions. For Recon, we scored the first run on every sampled item. Each cell gives correct choices, with True solves in parentheses.

Category (n)BaseCompSceneTaxComp+TaxReconCam
Swap (10)8 (8)10 (8)9 (9)8 (8)9 (9)8 (8)9 (8)
Move (23)7 (4)7 (5)12 (8)10 (4)7 (4)11 (7)16 (15)
Rotate (29)0 (0)8 (1)13 (3)8 (4)9 (3)7 (1)12 (7)
Mirror (15)9 (7)5 (4)11 (8)11 (9)11 (11)10 (8)10 (6)
Delete (12)8 (8)9 (8)10 (10)9 (8)9 (9)10 (10)11 (10)
Detail (11)8 (8)11 (9)9 (8)8 (8)10 (10)10 (9)10 (9)
Random sample (100)40 (35)50 (35)64 (46)54 (41)55 (46)56 (43)68 (55)
Original diagnostic set, Table 4 (100)24 (12)40 (23)37 (29)44 (32)48 (34)47 (34)61 (48)

The original diagnostic set was selected to include 12 Base True solves and 12 Base Partly guessed responses. On the random sample, Base reached 40% answer accuracy and 35% True solve, close to its full-benchmark scores of 39.70% and 34.90%. All six interventions improved answer accuracy over Base on both samples, and Cam ranked first on both metrics in both samples. The gains varied in magnitude. For example, Scene improved answer accuracy by 13 points on the original set and 24 points on the random set. The six API conditions are documented in the extra_diagnostic_experiment folders of API_Records.zip.

Weakness 3 and Q6. First-try success of the reconstruction workflow

Only two of the original 100 Recon items required a rerun. Each was rerun once because the first attempt lacked the required reconstruction evidence, and both answers remained incorrect after the rerun. Every other item met the execution and evidence requirements on its first attempt. As specified in Section 4.4 and Appendix A.8, the first compliant response was retained regardless of answer correctness.

On the new random subset, we scored the first Recon run for every item, regardless of reconstruction success. With no whole-item retries, Recon answered 56 of 100 questions correctly and earned 43 True solves.

Weakness 3 and Q7. Small and large objects

The six VLMs in Table 2 were evaluated zero-shot on SSoN. Medium items account for 56.8% of the test set and therefore carry the most weight in the overall score. An uneven size distribution is common in well-known benchmarks. In Microsoft COCO (Lin et al., ECCV 2014), about 41% of the objects are small, 34% medium and 24% large, and results are reported separately for each size group, as we do below. Section 3.3 defines size by the longest side of the unchanged object's axis-aligned bounding box after scene scaling. Each cell below gives answer accuracy / True solve (%).

Object size (n)GPT-6 Astra ProGPT-5.6 Sol ProGemini 3.5 FlashClaude Fable 5Gemini 3.6 FlashGrok 4.5
Small (276)59.78 / 55.0740.22 / 32.9731.16 / 21.3829.71 / 21.7428.99 / 20.2918.84 / 12.68
Medium (568)25.35 / 21.8316.90 / 9.8616.37 / 7.5713.73 / 8.1013.38 / 7.2210.39 / 4.58
Large (156)56.41 / 46.7922.44 / 14.1017.31 / 8.9721.15 / 16.0319.87 / 14.1020.51 / 12.18
Overall (1,000)39.70 / 34.9024.20 / 16.9020.60 / 11.6019.30 / 13.1018.70 / 11.9014.30 / 8.00

Medium items had the lowest answer accuracy and True solve rate for all six models. Small items had the highest answer accuracy for five of the six and the highest True solve rate for all six.

The edit distribution helps explain this pattern. Of the 568 medium items, 319 are rotations, the lowest-accuracy edit type for every model. Of the 276 small items, 184 are swaps, deletions, or detail edits. Every model scored higher on each of those three edit types than on either moves or rotations. The size breakdown therefore reflects substantial differences in the edits each group contains.

Weakness 2 and Q8. Sequential video or navigation

Humans reached 83.30% accuracy and the strongest VLM reached 39.70% from the same four static views (Table 2). Sequential video or navigation is a harder task, and this gap already appears in the simpler static setting. Our benchmark measures whether models can attribute a contradiction when all four observations are available simultaneously.

Sequential presentation is a natural next step, but it is beyond the scope of this paper. We plan to test continuous navigation paths to measure how the additional motion and intermediate views affect attribution.

Thank you again for the questions that led to the additional training seeds, the random-sample intervention study, the size breakdown, and the pilot evaluations. We would be grateful if you would reconsider your assessment in light of these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.