- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SpaceCast-Bench to evaluate whether vision-language models can infer spatial relations after a specified transformation using only observations of the original scene. It contains 3,862 questions from 182 indoor scenes across 16 task types, covering static perception, local prediction, and global prediction, with answers derived from 3D geometry and checked by human annotators. Experiments on 21 models show a substantial gap from human performance and examine the effects of prompting, bridge views, and additional spatial information. Fine-tuning Qwen3-VL-4B on data generated by the same pipeline improves performance on SpaceCast-Bench and raises its average score across six external benchmark subsets.
Critical strengths
- The benchmark combines controlled scene transformations with questions about direction, distance, occlusion, and attachment, including direction questions in three reference frames. Its geometry-based construction and independent human checks provide a useful basis for studying how models reason about spatial changes that are not shown in the input.
- The experimental coverage is substantial, with 21 evaluated models and controlled comparisons of prompting, bridge views, and additional spatial evidence. The training experiments also improve Qwen3-VL-4B from 34.0% to 65.7% and show gains on five of six external benchmark subsets. These results demonstrate that the data pipeline has value for both evaluation and training.
Critical weaknesses
- Section 2.2 acknowledges existing benchmarks involving hypothetical object motion and spatial state updates, but the paper does not provide a detailed comparison showing which reasoning requirements are new in SpaceCast-Bench. The distinction from MindCube, SpaMEM, and SAT therefore remains unclear. Since the paper presents a new benchmark as its main contribution, this limits my assessment of its novelty beyond broader task coverage and a real-scene construction pipeline.
- The experiments do not clearly separate errors in understanding the original scene from errors in applying the transformation. The static tasks provide a useful baseline, but the paper does not report predictive accuracy conditioned on correctly answering matched questions about the initial state. Gains from supplying 3D metadata also do not isolate transformation reasoning from scene understanding. The conclusion that existing spatial training fails to support state updating is therefore not fully established by the reported results.
Questions for authors
- What specific reasoning requirements does SpaceCast-Bench introduce beyond the tasks in MindCube, SpaMEM, and SAT discussed in Section 2.2? A direct comparison of their inputs, transformations, and queried relations would help clarify the novelty.
- Are there matched questions about the original and transformed states? If so, please report predictive accuracy on examples where the model correctly answers the initial-state question, particularly for the spatially specialized models.
- Do you have results for language-only inputs or a baseline that ignores the transformation and answers from the original scene? Please separate relation-changing and relation-preserving questions to show how much performance depends on the visual evidence and the specified transformation.