acceptodds
Under review as a conference paper at ICLR 2027

SIGHT-Bench: Diagnosing Reference-Frame Consistency and Global Spatial Reasoning in Multimodal Models

Abstract

We introduce SIGHT-Bench, a diagnostic benchmark for spatial reasoning in real-world elevated outdoor scenes. It draws on stationary camera installations whose positions remain fixed while viewing orientations may vary. SIGHT-Bench contains 1,000 four-choice questions from 400 cameras, organized into five dimensions and 12 subtasks spanning geometry, scene layout, viewpoint changes, world-frame motion, and topology. We evaluate proprietary, general-purpose open, and spatially specialized multimodal models against a human reference. Even the strongest evaluated models retain substantial overall gaps to human performance, and those gaps vary across spatial dimensions. Across model configurations, spatial consistency is more reliable than direct connectivity. Additional thinking yields marked gains in world-frame motion but much smaller gains elsewhere. Subtask analyses further show that improvements in one spatial relation need not extend to related judgments. These results motivate treating spatial reasoning as a profile of distinct capabilities rather than a single aggregate score.

Primary area:
datasets and benchmarks
Keywords:
Multimodal large language models, Spatial reasoning, Spatial intelligence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aQffOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose SIGHT-bench, a dataset featuring elevated images captured from 400 cameras paired with 1,000 multiple-choice questions (MCQs) designed to evaluate the spatial understanding of Vision-Language Models (VLMs) across five categories: Geometric Comparison, Scene Configuration, Viewpoint Analysis, World-Frame Motion, and Topological Navigation. Experiments across a wide range of model families demonstrate varying capabilities across tasks, with GPT-5.5 achieving over 95% accuracy in World-frame motion and 70%+ accuracy in two other categories.

Critical strengths

S1. Introduces a realistic, diverse dataset of elevated outdoor images captured across multiple camera angles.

S2. Evaluates the dataset across a broad range of VLMs to demonstrate model capabilities across different spatial reasoning sub-categories.

Critical weaknesses

W1. It is difficult to distinguish the novelty of the core results, as several of these sub-categories have been evaluated in prior work. The authors should explicitly clarify the gaps in spatial understanding that differentiate the proposed benchmark from related papers such as SpatialLab and Frontier Spatial.

W2. The benchmark lacks evaluations against recent specialized spatial agents (e.g., SpatialClaw, PySpatial), leaving an unaddressed gap in understanding performance differences between humans and dedicated spatial agents.

W3. The qualitative analysis outcomes appear generic (for example, models retaining correct left–right ordering while confusing front and rear); these failure modes are already commonly reported across existing spatial evaluation literature.

W4. The evaluation scope would be stronger if the authors explored recent reasoning-focused open-source models (such as DeepSeek reasoning variants) as well as efficient closed-source models like Gemini Flash, both of which report strong performance on visual task

Questions for authors

Addressing W1 and W2 is critical to clarifying the core gaps in existing work and justifying the need for another benchmark.

4 comments

Thank you for identifying the distinction from prior benchmarks and the evaluation of spatial agents as the central issues. We address these first, using the revised comparison and the new agent experiments.

W1 and Q1. Contribution Relative to Existing Spatial Benchmarks

SIGHT-Bench provides a focused setting for examining reference-dependent spatial judgments across views and time under a common outdoor acquisition regime. Within each camera installation, the viewing direction can change while the camera center remains fixed. This permits cross-view correspondence and viewpoint judgments without introducing camera translation. Temporally paired observations, together with the supplied camera heading, support geographic-motion judgments; visible road structure supports connectivity and route questions.

This setting lets us ask whether a model or inference configuration that performs well on one spatial operation also performs well on the other operations required to interpret these observations. For example, all 200 World-Frame Motion questions explicitly provide camera heading. GPT-5.5 thinking yields 125 net additional correct answers on this task, versus three on Camera Rotation. The two tasks therefore respond very differently even though both involve orientation information.

To clarify the relationship to prior work:

  • SpatiaLab evaluates a broad taxonomy across diverse real-world sources, including viewpoint visibility and navigation, and studies both multiple-choice and open-ended responses. It also reports task-dependent effects of reasoning. We acknowledge these precedents. SIGHT-Bench adds a more specific acquisition regime in which elevated views, a fixed camera center, temporal pairs, and heading information support the joint examination of reference-frame and scene-structure judgments.
  • FrontierSpatial provides broad coverage by consolidating 23 datasets into 65 tasks. SIGHT-Bench complements this breadth with a benchmark constructed under one acquisition regime across its task families.

Together, the acquisition protocol and analysis provide a testbed for assessing whether improvements in interpreting supplied headings also extend to cross-view correspondence and physical connectivity. This is the additional measurement role we claim for SIGHT-Bench. Comparisons across distinct question subsets describe performance profiles; they do not by themselves isolate internal abilities or equalize task difficulty.

Revised manuscript: Section 2.1 and Table 1; Sections 3, 4.4-4.5, and 5.


W2 and Q1. Evaluation of Spatial Agents

We added SpatialClaw-based evaluations on all 1,000 questions with two Qwen backbones. The evaluated configuration retains iterative code execution while disabling external perception modules and backbone thinking mode.

BackboneOverall directOverall agentWorld-Frame Motion direct to agent
Qwen3.8-27B51.6%61.6%34.5% to 86.5%
Qwen3.5-35B-A3B50.7%55.4%24.5% to 69.5%

These experiments reveal a substantial change in the capability profile. World-Frame Motion contributes 104 and 90 additional correct answers, respectively, exceeding the overall increases of 100 and 47. Consequently, the remaining 800 questions have four and 43 fewer correct answers under the reported agent protocols. Viewpoint Analysis and Topological Navigation decline for both backbones. These counts are an arithmetic decomposition of Table 3, with subtask results in Table 11.

The strongest evaluated agent configuration reaches 61.6%, compared with the approximately 90.1% human reference. This is a descriptive comparison for the tested configuration. Full perception-tool-enabled SpatialClaw and PySpatial remain unevaluated, so the revision adds evidence about agentic inference while leaving broader agent coverage open.

Appendix A.10 specifies the agent protocol: temperature 0.6, up to 4,096 output tokens per call, 30 agent steps, and up to two finalization actions. Direct and agentic inference use the separately documented inference and answer-extraction protocols; the archived Qwen3.8 direct run includes a recorded-score fallback described in Appendix A.11. We therefore interpret these as configuration comparisons, without claiming matched compute or attributing every difference specifically to reasoning quality.

Revised manuscript: Section 4.5 and Table 3; Appendices A.10-A.12, including Table 11.

W3. Specificity of the Failure Analysis

We extended the existing paired analysis to all 12 subtasks, added statistical tests, and linked the answer changes to concrete returned explanations. For GPT-5.5, thinking produces 226 corrections and 67 regressions, yielding 159 net additional correct answers. World-Frame Motion contributes 125 of these net gains, or 78.6%. Its 127 corrections and two regressions contrast with seven corrections and ten regressions on Camera-Relative Distance, and one correction and four regressions on Direct Connectivity.

After Holm correction across the 12 subtasks, only the World-Frame Motion and Multi-Object Layout improvements are statistically significant. The small observed declines in Camera-Relative Distance and Direct Connectivity are therefore presented as changes in this evaluation, not established systematic degradation.

Figure 6 connects these counts to concrete answer changes: a successful image-to-world heading conversion, an unsupported use of image-row position to compare depth across views, a corrected junction relation, and a mistaken association between a ground-level road and an elevated road. These are examples of familiar error types; their role is to make the measured corrections and regressions inspectable. The returned explanations do not establish the causal origin of an error inside the model.

Revised manuscript: Sections 5.4-5.5, Figure 6, and Appendix A.8.


W4. Additional Model Coverage

We added the requested Gemini Flash and DeepSeek configurations as part of the expanded evaluation:

ModelOverall accuracy95% confidence interval
Gemini-3.8-Flash74.5%[71.7, 77.1]
DeepSeek-V4.1-Flash54.8%[51.7, 57.9]

These results broaden coverage of the model families identified in your review. The DeepSeek result covers the named configuration; a systematic comparison of its reasoning variants is still outside the evaluated set.

Revised manuscript: Table 6 and Appendix A.11, Tables 13 and 15.

Thank you for the detailed rebuttal, the expanded model evaluations, and the statistical breakdown of the error analysis. While the positioning against SpatiaLab and FrontierSpatial clarifies the acquisition setup, my core concerns regarding the evaluation of full perception-enabled spatial agents (such as PySpatial and full SpatialClaw) and high-effort DeepSeek reasoning configurations remain only partially addressed. Given these remaining gaps and the overlap in spatial sub-categories with prior benchmarks, I will maintain my current rating.

Dear Reviewer,

Thank you for your follow-up and for acknowledging the additional evaluations and analyses. We appreciate the clarification of the remaining concerns. In response, we have completed further experiments on full perception-enabled spatial agents and high-effort DeepSeek reasoning.

1. Full perception-enabled spatial agent evaluation

We evaluated pySpatial with its full perception toolchain enabled on all 1,000 SIGHT-Bench questions using six model configurations, spanning open-weight models at different scales and proprietary models. Unlike the previously reported SpatialClaw experiments, these evaluations enable pySpatial's external perception components.

All values are accuracy.

BackboneGCSCVAWFTNOverall
Qwen3.8-27B39.1%39.9%51.0%75.5%55.1%52.5%
Qwen3.5-35B-A3B39.1%41.1%53.0%30.5%49.8%42.9%
Qwen3.5-397B-A17B34.9%44.3%54.0%33.5%52.4%43.9%
Qwen3.8-Max47.9%44.3%48.5%46.5%60.4%50.0%
DeepSeek-V4.1-Flash37.7%40.5%49.0%30.5%52.4%42.3%
Kimi-K338.1%41.8%53.0%45.5%54.2%46.8%

The results show that a full perception-enabled agent does not consistently improve accuracy across model configurations. For the two Qwen backbones previously evaluated with SpatialClaw, pySpatial achieves 52.5% and 42.9% Overall accuracy, compared with 51.6% and 50.7% under direct inference. The corresponding SpatialClaw configurations reach 61.6% and 55.4%, with gains concentrated in World-Frame Motion.

The differences are also substantial within pySpatial. World-Frame Motion accuracy ranges from 30.5% to 75.5% across the six configurations, while Overall accuracy ranges from 42.3% to 52.5%. These results extend the agent evaluation beyond code execution alone and provide a more specific diagnostic: access to external perception tools does not, by itself, imply consistent gains in spatial reasoning.

We interpret these as configuration-level comparisons, not controlled estimates of the isolated contribution of perception modules. Although the full pySpatial pipeline was enabled, individual runs could use fallback answer paths, and the configurations were not compute-matched.

Full perception-enabled SpatialClaw remains unevaluated. Its implementation requires access to Meta's gated SAM3 weights. We have submitted an access request but have not yet received approval. We therefore retain the earlier SpatialClaw results as evaluations without external perception modules and intend to evaluate the full configuration once access is granted.

2. High-effort DeepSeek reasoning

We additionally evaluated DeepSeek-V4.1-Flash with high reasoning effort on the same 1,000 questions.

All values are accuracy.

ConfigurationGCSCVAWFTNOverall
DeepSeek-V4.1-Flash (high-effort)59.5%44.9%73.0%97.5%69.2%69.7%

The high-effort configuration achieves 69.7% Overall accuracy, compared with 54.8% for the previously reported DeepSeek-V4.1-Flash configuration, an observed increase of 14.9 percentage points. World-Frame Motion reaches 97.5%, whereas Scene Configuration remains at 44.9%.

This result provides a second model family in which reasoning-intensive inference is associated with pronounced differences across spatial capabilities. As in the GPT-5.5 analysis, strong performance on World-Frame Motion does not imply comparable performance on scene configuration or other spatial operations. We report the observed differences between inference configurations without attributing them to a particular internal reasoning mechanism.

3.Contribution and task overlap

SIGHT-Bench is distinguished by its focus on spatial reasoning from elevated outdoor viewpoints, where perspective effects, occlusion, and multi-level scene structures complicate the relationship between visual appearance and physical geometry or connectivity. Its fixed-center cameras, changing viewing directions, and temporal observations support joint evaluation of reference-frame interpretation, cross-view correspondence, and road topology within this setting. The contribution lies in a purpose-built real-world testbed for assessing how reliably models infer scene structure from partial, viewpoint-dependent evidence. This provides a concrete complement to SpatiaLab and FrontierSpatial beyond overlap in broad task categories.

Revised manuscript: Section 2.1 and Table 1; Section 3; Sections 4.4–4.5 and Table 3; Section 5; Appendix A.7 and Table 6; Appendix A.10

Reviewer_Jr93OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SIGHT-Bench, a diagnostic benchmark designed to evaluate spatial reasoning in multimodal foundation models using real-world elevated outdoor observations. Unlike existing benchmarks, SIGHT-Bench uses the scene where camera positions remain fixed and viewing angles vary. The dataset comprises 1,000 four-choice questions from 400 cameras across five core dimensions (Geometric Comparison, Scene Configuration, Viewpoint Analysis, World-Frame Motion, and Topological Navigation) and 12 subtasks. Across comprehensive evaluations covering proprietary, open-weight, and spatially specialized models, the authors highlight significant human-model performance gaps, pronounced capability trade-offs across spatial subtasks, and the asymmetric benefit of chain-of-thought "thinking" compute.

Critical strengths
  1. Clear Diagnostic Focus and Well-Motivated Setup: The stationary elevated camera setting isolates viewpoint orientation and spatial transformations without introducing camera translation artifacts. Treating spatial intelligence as a multidimensional diagnostic profile rather than an aggregate scalar score provides actionable insights for model development.

  2. Comprehensive and Timely Model Evaluation: The benchmark tests leading proprietary models (e.g., GPT-5.5 with and without thinking, Claude Opus 4.8), large open-weight models (Qwen3-VL, InternVL3.5 families), and spatial specialists (SenseNova-SI, Spatial-SSRL). The human reference and text-only control further ensure robust calibration.

Critical weaknesses
  1. Lack of Constructive Solutions or Baselines: The submission operates purely as a diagnostic benchmark. It does not explore fine-tuning strategies, architectural interventions (e.g., explicit camera pose/intrinsics injection), or prompting solutions to mitigate the identified failure modes.

  2. Discretization and Multiple-Choice Formulation: Formulating all tasks as 4-choice QA reduces evaluation friction, but potentially allows models to leverage elimination heuristics. Adding open-ending evaluations or metric space (e.g., metric depth error or continuous degree estimation for orientation changes) would provide a deeper geometric assessment.

  3. Imbalance in Thinking Compute Gains: While the analysis in Section 4.5 and Section 5.3 is compelling, the paper mostly characterizes where thinking fails rather than conducting deeper error analysis on intermediate chain-of-thought traces (e.g., whether visual grounding failed initially or reasoning collapsed midway).

Questions for authors
  1. For model developing, how is the benchmark or the failure cases can shows ways to improve the current models?

  2. For case analysis, could the authors analyze the chain-of-thought traces from VLM (like GPT-5.5 - thinking) to clarify why camera-relative distance and direct connectivity exhibited regressions compared to the no-reasoning setting?

4 comments

Thank you for recognizing the diagnostic motivation and for asking us to demonstrate how the benchmark can guide improvement. We have strengthened the response with an inference intervention, paired error analysis, and clearer limits on what four-choice accuracy measures.

W1 and Q1. How the Benchmark Informs Model Development

The new agent experiments provide a concrete use of the diagnostic: checking whether an inference change improves the spatial operation of interest while preserving performance elsewhere. We evaluate the same questions with each backbone under direct inference and the documented SpatialClaw-based configuration, and complement this with the paired GPT-5.5 thinking comparison.

Configuration changeOverall accuracyWorld-Frame MotionRemaining 800 questions
GPT-5.5 with thinking55.9% to 71.8%34.0% to 96.5%61.4% to 65.6%
Qwen3.8-27B with agent51.6% to 61.6%34.5% to 86.5%55.9% to 55.4%
Qwen3.5-35B-A3B with agent50.7% to 55.4%24.5% to 69.5%57.3% to 51.9%

The last column is derived by subtracting the 200 World-Frame Motion outcomes from the reported totals. For the two agent runs, the gains on World-Frame Motion exceed the overall gains, so accuracy on the other 800 questions decreases. An overall score alone would hide this distinction. These are descriptive results under the reported protocols; the agent runs disable external perception modules and are not compute-matched to direct inference.

This yields concrete development priorities. First, all World-Frame Motion questions already provide camera heading, and the large gains show that performance on this task depends strongly on the inference configuration even when the relevant metadata is supplied. Second, success on this task should be checked separately from cross-view correspondence and topology: both agent runs have lower Viewpoint Analysis and Topological Navigation scores. Third, the remaining errors identify testable targets for future interventions, such as maintaining object and road identity across views, distinguishing overpasses from accessible junctions, and preserving the specified reference frame in directional judgments.

These are hypotheses for targeted data, representations, or inference checks. The present experiments demonstrate how the benchmark can evaluate an intervention and reveal its trade-offs; they do not validate a particular new training or architectural remedy.

Revised manuscript: Sections 4.4-4.5 and 5; Tables 3, 9, and 11.


W2. Multiple Choice and Geometric Evaluation

We agree that four-choice accuracy can admit elimination heuristics and does not directly measure continuous geometric estimation. The format gives heterogeneous tasks a common scoring rule, while our claims concern the spatial judgments measured by these questions.

Two checks address limited sources of bias. Correct-answer positions are approximately balanced (A/B/C/D: 247/261/256/236), reducing the usefulness of a fixed-position preference. Removing visual input reduces Qwen3-VL-8B's reported accuracy from 40.8% to approximately 29.3%, against a 25% chance baseline. This indicates that visual input contributes to the score for this model.

Neither check rules out semantic elimination or other shortcuts. In particular, the removal control also removes image-based answer choices, so the decrease cannot be attributed exclusively to scene understanding. We have made the evaluation scope explicit in Appendix A.12. Open-ended answers and calibrated depth or continuous-angle errors would provide complementary measurements, and are not measured by the present four-choice protocol.

Revised manuscript: Section 3.1, Section 4.2, and Appendices A.6 and A.12.

W3 and Q2. Why Some Answers Regress with Thinking

We strengthened the existing paired comparison with complete subtask-level correction and regression counts, statistical tests, and analysis of the associated returned explanations. The two GPT-5.5 runs are aligned by question ID, with matching questions, labels, and ground truths. Across all 1,000 questions, thinking corrects 226 answers and introduces 67 regressions.

SubtaskQuestionsCorrectionsRegressionsNet correct answersHolm adjusted p
World-Frame Motion2001272+125< 0.001
Camera-Relative Distance51710-31.0000
Multi-Object Layout72173+140.0283
Direct Connectivity5114-31.0000
Route Planning42144+100.2780

The tests are two-sided exact McNemar tests with Holm adjustment across all 12 subtasks. World-Frame Motion and Multi-Object Layout retain significance at 0.05; the observed Camera-Relative Distance and Direct Connectivity declines do not. The tests operate at question level and do not adjust for dependence among questions from the same camera.

Figure 6 provides more specific evidence about the requested regressions:

  • Camera-Relative Distance: the returned explanation uses vertical image position as a proxy for distance and changes the correct blue-point answer to the red point. Across these views, image-row position alone is insufficient to establish the required distance ordering. The case illustrates an unsupported geometric shortcut in the explanation.
  • Direct Connectivity: one explanation reassigns a marked ground-level road to an elevated highway, changing a correct crossing-without-access answer to no meeting. A separate example shows the opposite outcome: thinking correctly identifies a junction or merge. These cases distinguish errors in cross-view road association from successful interpretation of a connection.

The explanations make the answer changes more interpretable, but cannot uniquely determine whether an error began during visual grounding or later reasoning. We therefore report observable error patterns and paired outcomes, while avoiding a causal claim about hidden intermediate computation.

Revised manuscript: Sections 5.4-5.5, Figure 6, and Appendix A.8, Tables 9-10.

Thanks for the explanation. The new experiments address my concerns, and I will keep my original score.

Thank you for your follow-up and for taking the time to consider our additional experiments and clarifications. We appreciate your acknowledgment that the new experiments address your concerns. Your feedback has been valuable in helping us strengthen the experimental evaluation and clarify the scope and limitations of SIGHT-Bench.

Reviewer_w5ZAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SIGHT-bench is a solid resource and evaluate paper looking at the ability of VLMs versus humans to understand spatial relationships in real world outdoor scenes taken by video cameras. The paper is clear and well written, related benchmarks are briefly reviewed and given a feature matrix which is clear and easy to understand, and a gap is identified. SIGHT-bench investigates a diverse group of cameras in elevated, stationary positions with the goal of answering MCQs across 5 dimensions (e.g. geometric comparisons, object motion), which themselves are partitioned into sub-dimensions making for a total of 12 diagnostic dimensions. The final benchmark consists of 1000 4-way MCQs across 2688 images from 400 cameras.

Critical strengths
  • Clear motivation and positioning, easy to read and follow. Nice emphasis on real-world challenges of scene understanding with real data that will be welcomed by others.
  • The benchmark itself has been well thought about and structured. It's balanced and the taxonomy looks reasonable, clearly yielding interesting and differentiated results across models and categories of relationships.
  • Decent baselines and interesting results (e.g. WF, thinking versus non-thinking, open versus vendor models). It might be worth ordering the performance by model release data - most likely you'll see improvements over time.
Critical weaknesses
  • Question construction used a VLM so danger of bias and limitation on difficulty (although as the authors rightly point out this isn't a unique methodological limitation for their study so I do understand this is a pragmatic choice and just note it here). To compensate for this human review was performed both for quality control and question selection.
  • Figures are tiny and difficult to read without zoom.
  • It's not clear how many people participated in reviewing, what their qualifications were and whether they were independent from the study or how disagreements were handled. No IAA reported.
  • Results are reported to two dps - this implies precision.
Questions for authors
  • Who were the annotators? Were they the authors or independent of the study, any IAA? This would help make the study stronger.
  • What are the confidence intervals here in Table 2 to help readers understand meaningful differences from sample variation?
1 comment

Thank you for the positive assessment of the motivation and taxonomy, and for the specific requests concerning annotation transparency, readability, and uncertainty. We have expanded the methodological description and statistical reporting as follows.

W1, W3, and Q1. Question Construction and Annotation Independence

Human authors assigned the ground-truth answers, and a non-author reviewer who had not constructed the items performed final verification. VLM assistance supported template-based drafting of straightforward questions. Human authors wrote or revised questions involving multiple images or more complex spatial reasoning, finalized each item, assigned its answer from the visual evidence and task context, and designed three plausible distractors.

Six individuals with research experience in computer vision and multimodal learning contributed to construction and verification. Their roles were:

  1. Source screening: two reviewers assessed camera suitability, visibility, and available spatial evidence. Only sources approved by both were retained, yielding 400 cameras from 1,580 candidates.
  2. Question construction: the authors prepared and annotated 1,500 candidate questions from the screened observations.
  3. Final verification: a non-author reviewer checked visual grounding, task alignment, wording, answer correctness and uniqueness, and distractor plausibility. Wording or option problems were revised; insufficient evidence, unresolved ambiguity, unintended textual cues, or task mismatches led to removal. The final set contains 1,000 questions.

This process provides human control over the answers and a separate verification stage, although it cannot eliminate construction or selection bias. It also differs from blind independent answer annotation: the verifier checked proposed answers, and parallel independent labels were not collected. We therefore cannot report a quantitative inter-annotator agreement score and explicitly acknowledge this limitation.

The five undergraduate participants in the human reference were separate from construction and verification. Their benchmark performance is reported as a human reference, not an annotation-agreement estimate.

Revised manuscript: Section 3.1 and Figure 3; Appendices A.6 and A.12.


W2. Figure Readability

We agree that the compact overview figures are difficult to inspect at normal page scale. We will further enlarge their labels and simplify dense annotations in the final presentation. The task definitions in Section 3, explicit question counts in Table 5, and the 24 question examples in Appendix A.9 provide detailed references, but do not replace improving the overview figures themselves.


W4 and Q2. Numerical Precision and Confidence Intervals

We now report accuracy to one decimal place and provide two-sided, pointwise 95% Wilson confidence intervals. The intervals use question-level counts before rounding: 1,000 questions overall, and 215, 158, 200, 200, and 227 questions for GC, SC, VA, WF, and TN, respectively.

ModelOverall accuracy95% confidence interval
Claude Opus 5.579.9%[77.3, 82.3]
GPT-6.1-Sol79.3%[76.7, 81.7]
GPT-5.5 with thinking71.8%[68.9, 74.5]

Tables 12-15 provide overall and dimension-level intervals, and Table 2 now points readers to them. These intervals describe question-level sampling uncertainty under the item-based calculation; they do not account for dependence among questions from the same camera or variability across repeated decoding runs. The close observed scores of the two leading models are not, by themselves, evidence of a statistically reliable ordering.

Appendix A.11 also identifies the provenance of the counts and the scoring exceptions. We leave intervals unavailable where retained human or text-only summaries do not support the corresponding estimate. Random guessing is an analytical 25% expectation rather than an empirical run.

Revised manuscript: Section 4.1, Table 2, and Appendix A.11, Tables 12-15.


Suggestion. Ordering Models by Release Date

A release-date view would provide useful additional context. The revision retains model-family grouping and identifies the evaluated model sources in Appendix A.5. We do not infer a controlled trend over time from models that also differ in training data, architecture, and inference configuration.

Comments on the paper

2 comments

General response and revision summary

We thank the reviewers for their careful feedback. We have strengthened the manuscript around the questions most relevant to the contribution: what SIGHT-Bench adds to existing spatial benchmarks, how its measurements inform model development, and how reliably the reported differences can be interpreted.

The revision includes the following substantive changes:

  1. More precise positioning. Section 2.1 and Table 1 distinguish task coverage from acquisition constraints. SIGHT-Bench examines complementary spatial judgments under a common regime of stationary elevated outdoor cameras, with changing viewing directions and temporally paired observations.
  2. Additional inference experiments and broader model coverage. We evaluate SpatialClaw-based code execution on all 1,000 questions with two Qwen backbones, with external perception modules disabled. The broader evaluation now covers 37 distinct models and includes the requested Gemini Flash and DeepSeek configurations.
  3. Paired analysis of improvements and regressions. We align GPT-5.5 predictions by question, report corrections and regressions across all 12 subtasks, and add exact McNemar tests with Holm correction and examples of answer changes.
  4. Clearer annotation and statistical reporting. We document the construction and verification roles, provide pointwise 95% confidence intervals, use one decimal place for accuracy, and state the remaining annotation and evaluation limitations.

The additional analysis makes the benchmark's practical use more concrete: in the evaluated configurations, a higher overall score can conceal unchanged or lower accuracy on other spatial tasks. The responses below explain the evidence for each concern. All section, figure, and table references refer to the revised manuscript.

AuthorsOpenReview ↗

Author Statement

Dear Reviewers,

Thank you for your careful feedback on SIGHT-Bench. We have responded to each point in the corresponding review threads and revised the manuscript accordingly. As the discussion period draws to a close, we hope you will reconsider your assessments in light of our responses and the revised manuscript. If any concerns remain, we would be happy to provide further clarification.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.