- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose SIGHT-bench, a dataset featuring elevated images captured from 400 cameras paired with 1,000 multiple-choice questions (MCQs) designed to evaluate the spatial understanding of Vision-Language Models (VLMs) across five categories: Geometric Comparison, Scene Configuration, Viewpoint Analysis, World-Frame Motion, and Topological Navigation. Experiments across a wide range of model families demonstrate varying capabilities across tasks, with GPT-5.5 achieving over 95% accuracy in World-frame motion and 70%+ accuracy in two other categories.
Critical strengths
S1. Introduces a realistic, diverse dataset of elevated outdoor images captured across multiple camera angles.
S2. Evaluates the dataset across a broad range of VLMs to demonstrate model capabilities across different spatial reasoning sub-categories.
Critical weaknesses
W1. It is difficult to distinguish the novelty of the core results, as several of these sub-categories have been evaluated in prior work. The authors should explicitly clarify the gaps in spatial understanding that differentiate the proposed benchmark from related papers such as SpatialLab and Frontier Spatial.
W2. The benchmark lacks evaluations against recent specialized spatial agents (e.g., SpatialClaw, PySpatial), leaving an unaddressed gap in understanding performance differences between humans and dedicated spatial agents.
W3. The qualitative analysis outcomes appear generic (for example, models retaining correct left–right ordering while confusing front and rear); these failure modes are already commonly reported across existing spatial evaluation literature.
W4. The evaluation scope would be stronger if the authors explored recent reasoning-focused open-source models (such as DeepSeek reasoning variants) as well as efficient closed-source models like Gemini Flash, both of which report strong performance on visual task
Questions for authors
Addressing W1 and W2 is critical to clarifying the core gaps in existing work and justifying the need for another benchmark.
4 comments
W3. Specificity of the Failure Analysis
We extended the existing paired analysis to all 12 subtasks, added statistical tests, and linked the answer changes to concrete returned explanations. For GPT-5.5, thinking produces 226 corrections and 67 regressions, yielding 159 net additional correct answers. World-Frame Motion contributes 125 of these net gains, or 78.6%. Its 127 corrections and two regressions contrast with seven corrections and ten regressions on Camera-Relative Distance, and one correction and four regressions on Direct Connectivity.
After Holm correction across the 12 subtasks, only the World-Frame Motion and Multi-Object Layout improvements are statistically significant. The small observed declines in Camera-Relative Distance and Direct Connectivity are therefore presented as changes in this evaluation, not established systematic degradation.
Figure 6 connects these counts to concrete answer changes: a successful image-to-world heading conversion, an unsupported use of image-row position to compare depth across views, a corrected junction relation, and a mistaken association between a ground-level road and an elevated road. These are examples of familiar error types; their role is to make the measured corrections and regressions inspectable. The returned explanations do not establish the causal origin of an error inside the model.
Revised manuscript: Sections 5.4-5.5, Figure 6, and Appendix A.8.
W4. Additional Model Coverage
We added the requested Gemini Flash and DeepSeek configurations as part of the expanded evaluation:
| Model | Overall accuracy | 95% confidence interval |
|---|---|---|
| Gemini-3.8-Flash | 74.5% | [71.7, 77.1] |
| DeepSeek-V4.1-Flash | 54.8% | [51.7, 57.9] |
These results broaden coverage of the model families identified in your review. The DeepSeek result covers the named configuration; a systematic comparison of its reasoning variants is still outside the evaluated set.
Revised manuscript: Table 6 and Appendix A.11, Tables 13 and 15.
Thank you for the detailed rebuttal, the expanded model evaluations, and the statistical breakdown of the error analysis. While the positioning against SpatiaLab and FrontierSpatial clarifies the acquisition setup, my core concerns regarding the evaluation of full perception-enabled spatial agents (such as PySpatial and full SpatialClaw) and high-effort DeepSeek reasoning configurations remain only partially addressed. Given these remaining gaps and the overlap in spatial sub-categories with prior benchmarks, I will maintain my current rating.
Dear Reviewer,
Thank you for your follow-up and for acknowledging the additional evaluations and analyses. We appreciate the clarification of the remaining concerns. In response, we have completed further experiments on full perception-enabled spatial agents and high-effort DeepSeek reasoning.
1. Full perception-enabled spatial agent evaluation
We evaluated pySpatial with its full perception toolchain enabled on all 1,000 SIGHT-Bench questions using six model configurations, spanning open-weight models at different scales and proprietary models. Unlike the previously reported SpatialClaw experiments, these evaluations enable pySpatial's external perception components.
All values are accuracy.
| Backbone | GC | SC | VA | WF | TN | Overall |
|---|---|---|---|---|---|---|
| Qwen3.8-27B | 39.1% | 39.9% | 51.0% | 75.5% | 55.1% | 52.5% |
| Qwen3.5-35B-A3B | 39.1% | 41.1% | 53.0% | 30.5% | 49.8% | 42.9% |
| Qwen3.5-397B-A17B | 34.9% | 44.3% | 54.0% | 33.5% | 52.4% | 43.9% |
| Qwen3.8-Max | 47.9% | 44.3% | 48.5% | 46.5% | 60.4% | 50.0% |
| DeepSeek-V4.1-Flash | 37.7% | 40.5% | 49.0% | 30.5% | 52.4% | 42.3% |
| Kimi-K3 | 38.1% | 41.8% | 53.0% | 45.5% | 54.2% | 46.8% |
The results show that a full perception-enabled agent does not consistently improve accuracy across model configurations. For the two Qwen backbones previously evaluated with SpatialClaw, pySpatial achieves 52.5% and 42.9% Overall accuracy, compared with 51.6% and 50.7% under direct inference. The corresponding SpatialClaw configurations reach 61.6% and 55.4%, with gains concentrated in World-Frame Motion.
The differences are also substantial within pySpatial. World-Frame Motion accuracy ranges from 30.5% to 75.5% across the six configurations, while Overall accuracy ranges from 42.3% to 52.5%. These results extend the agent evaluation beyond code execution alone and provide a more specific diagnostic: access to external perception tools does not, by itself, imply consistent gains in spatial reasoning.
We interpret these as configuration-level comparisons, not controlled estimates of the isolated contribution of perception modules. Although the full pySpatial pipeline was enabled, individual runs could use fallback answer paths, and the configurations were not compute-matched.
Full perception-enabled SpatialClaw remains unevaluated. Its implementation requires access to Meta's gated SAM3 weights. We have submitted an access request but have not yet received approval. We therefore retain the earlier SpatialClaw results as evaluations without external perception modules and intend to evaluate the full configuration once access is granted.
2. High-effort DeepSeek reasoning
We additionally evaluated DeepSeek-V4.1-Flash with high reasoning effort on the same 1,000 questions.
All values are accuracy.
| Configuration | GC | SC | VA | WF | TN | Overall |
|---|---|---|---|---|---|---|
| DeepSeek-V4.1-Flash (high-effort) | 59.5% | 44.9% | 73.0% | 97.5% | 69.2% | 69.7% |
The high-effort configuration achieves 69.7% Overall accuracy, compared with 54.8% for the previously reported DeepSeek-V4.1-Flash configuration, an observed increase of 14.9 percentage points. World-Frame Motion reaches 97.5%, whereas Scene Configuration remains at 44.9%.
This result provides a second model family in which reasoning-intensive inference is associated with pronounced differences across spatial capabilities. As in the GPT-5.5 analysis, strong performance on World-Frame Motion does not imply comparable performance on scene configuration or other spatial operations. We report the observed differences between inference configurations without attributing them to a particular internal reasoning mechanism.
3.Contribution and task overlap
SIGHT-Bench is distinguished by its focus on spatial reasoning from elevated outdoor viewpoints, where perspective effects, occlusion, and multi-level scene structures complicate the relationship between visual appearance and physical geometry or connectivity. Its fixed-center cameras, changing viewing directions, and temporal observations support joint evaluation of reference-frame interpretation, cross-view correspondence, and road topology within this setting. The contribution lies in a purpose-built real-world testbed for assessing how reliably models infer scene structure from partial, viewpoint-dependent evidence. This provides a concrete complement to SpatiaLab and FrontierSpatial beyond overlap in broad task categories.
Revised manuscript: Section 2.1 and Table 1; Section 3; Sections 4.4–4.5 and Table 3; Section 5; Appendix A.7 and Table 6; Appendix A.10
Thank you for identifying the distinction from prior benchmarks and the evaluation of spatial agents as the central issues. We address these first, using the revised comparison and the new agent experiments.
W1 and Q1. Contribution Relative to Existing Spatial Benchmarks
SIGHT-Bench provides a focused setting for examining reference-dependent spatial judgments across views and time under a common outdoor acquisition regime. Within each camera installation, the viewing direction can change while the camera center remains fixed. This permits cross-view correspondence and viewpoint judgments without introducing camera translation. Temporally paired observations, together with the supplied camera heading, support geographic-motion judgments; visible road structure supports connectivity and route questions.
This setting lets us ask whether a model or inference configuration that performs well on one spatial operation also performs well on the other operations required to interpret these observations. For example, all 200 World-Frame Motion questions explicitly provide camera heading. GPT-5.5 thinking yields 125 net additional correct answers on this task, versus three on Camera Rotation. The two tasks therefore respond very differently even though both involve orientation information.
To clarify the relationship to prior work:
Together, the acquisition protocol and analysis provide a testbed for assessing whether improvements in interpreting supplied headings also extend to cross-view correspondence and physical connectivity. This is the additional measurement role we claim for SIGHT-Bench. Comparisons across distinct question subsets describe performance profiles; they do not by themselves isolate internal abilities or equalize task difficulty.
Revised manuscript: Section 2.1 and Table 1; Sections 3, 4.4-4.5, and 5.
W2 and Q1. Evaluation of Spatial Agents
We added SpatialClaw-based evaluations on all 1,000 questions with two Qwen backbones. The evaluated configuration retains iterative code execution while disabling external perception modules and backbone thinking mode.
These experiments reveal a substantial change in the capability profile. World-Frame Motion contributes 104 and 90 additional correct answers, respectively, exceeding the overall increases of 100 and 47. Consequently, the remaining 800 questions have four and 43 fewer correct answers under the reported agent protocols. Viewpoint Analysis and Topological Navigation decline for both backbones. These counts are an arithmetic decomposition of Table 3, with subtask results in Table 11.
The strongest evaluated agent configuration reaches 61.6%, compared with the approximately 90.1% human reference. This is a descriptive comparison for the tested configuration. Full perception-tool-enabled SpatialClaw and PySpatial remain unevaluated, so the revision adds evidence about agentic inference while leaving broader agent coverage open.
Appendix A.10 specifies the agent protocol: temperature 0.6, up to 4,096 output tokens per call, 30 agent steps, and up to two finalization actions. Direct and agentic inference use the separately documented inference and answer-extraction protocols; the archived Qwen3.8 direct run includes a recorded-score fallback described in Appendix A.11. We therefore interpret these as configuration comparisons, without claiming matched compute or attributing every difference specifically to reasoning quality.
Revised manuscript: Section 4.5 and Table 3; Appendices A.10-A.12, including Table 11.