acceptodds
Withdrawn from ICLR 2027

SpaceCast-Bench: Evaluating Predictive Spatial Reasoning in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing spatial reasoning benchmarks mainly test spatial perception: reading off relations already visible in the input. Yet real-world spatial intelligence demands predictive spatial reasoning: constructing a scene from observations, anticipating how an intervention changes it, and reasoning about the unseen outcome. We introduce SpaceCast-Bench, the first benchmark to directly and diagnostically evaluate this capability. Built around an observe–transform–infer framework, its 3,862 questions from 182 real-world scenes span 16 task types at three levels: static perception, local prediction, and global prediction, progressively requiring scene understanding, spatial state updating, and relational inference over unobserved outcomes. Evaluating 21 models exposes a stark gap: the strongest model reaches only 58.0% against 87.2% human performance, while spatially specialized models remain near random chance. Controlled analyses further reveal that bridge views are critical for integrating distributed observations, and that explicit 3D evidence benefits models more reliably than generated outcome images or videos. Fine-tuning on our programmatically generated data lifts Qwen3-VL-4B from 34.0% to 65.7% with macro-average gains across six out-of-domain benchmarks.

Primary area:
datasets and benchmarks
Keywords:
Spatial Reasoning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Do5jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpaceCast-Bench to evaluate whether vision-language models can infer spatial relations after a specified transformation using only observations of the original scene. It contains 3,862 questions from 182 indoor scenes across 16 task types, covering static perception, local prediction, and global prediction, with answers derived from 3D geometry and checked by human annotators. Experiments on 21 models show a substantial gap from human performance and examine the effects of prompting, bridge views, and additional spatial information. Fine-tuning Qwen3-VL-4B on data generated by the same pipeline improves performance on SpaceCast-Bench and raises its average score across six external benchmark subsets.

Critical strengths
  1. The benchmark combines controlled scene transformations with questions about direction, distance, occlusion, and attachment, including direction questions in three reference frames. Its geometry-based construction and independent human checks provide a useful basis for studying how models reason about spatial changes that are not shown in the input.
  2. The experimental coverage is substantial, with 21 evaluated models and controlled comparisons of prompting, bridge views, and additional spatial evidence. The training experiments also improve Qwen3-VL-4B from 34.0% to 65.7% and show gains on five of six external benchmark subsets. These results demonstrate that the data pipeline has value for both evaluation and training.
Critical weaknesses
  1. Section 2.2 acknowledges existing benchmarks involving hypothetical object motion and spatial state updates, but the paper does not provide a detailed comparison showing which reasoning requirements are new in SpaceCast-Bench. The distinction from MindCube, SpaMEM, and SAT therefore remains unclear. Since the paper presents a new benchmark as its main contribution, this limits my assessment of its novelty beyond broader task coverage and a real-scene construction pipeline.
  2. The experiments do not clearly separate errors in understanding the original scene from errors in applying the transformation. The static tasks provide a useful baseline, but the paper does not report predictive accuracy conditioned on correctly answering matched questions about the initial state. Gains from supplying 3D metadata also do not isolate transformation reasoning from scene understanding. The conclusion that existing spatial training fails to support state updating is therefore not fully established by the reported results.
Questions for authors
  1. What specific reasoning requirements does SpaceCast-Bench introduce beyond the tasks in MindCube, SpaMEM, and SAT discussed in Section 2.2? A direct comparison of their inputs, transformations, and queried relations would help clarify the novelty.
  2. Are there matched questions about the original and transformed states? If so, please report predictive accuracy on examples where the model correctly answers the initial-state question, particularly for the spatially specialized models.
  3. Do you have results for language-only inputs or a baseline that ignores the transformation and answers from the original scene? Please separate relation-changing and relation-preserving questions to show how much performance depends on the visual evidence and the specified transformation.
Reviewer_WTqBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark named SpaceCast-Bench to evaluate "predictive spatial reasoning". Each test case consists of a set of images from one scene, combined with a text description of the capture, and a question on the spatial relations between the objects in the scene for the model to answer. The question, depending on the difficulty level, can be either a "static" task, where a model needs to answer based on the current observation, or a predictive task, where the question is in a form of "if we perform and action A...". The benchmark therefore requires to model to be able not only recognise the spatial relations of object, but also be able to predict the effect of actions on them. Empirical evaluation ona set of proprietary and open VLMs/MLLMs show a significant gap between human an AI performance on this task. SpaceCast consists of 182 scenes, with 3862 questions in total.

Critical strengths
  • The topic of spatial reasoning is an important and timely issue in LLM development, as current models become more and more capable of embodied interactions.
  • The action-outcome prediction type of benchmark, while not able to capture if a model can actually act in the world or predict physical interactions between objects, is a convenient proxy for evaluation of the above mentioned spatial reasoning capabilities.
  • The paper provides a wide evaluation of both proprietary and open models, as well as a human baseline.
  • The authors describe the dataset preparation pipeline as well as provide the code for review. Human verification was conducted.
  • A simple SFT and GRPO baselines show the overall range of improvement expected of a small model.
  • The general writing is generally understandable.
Critical weaknesses
  • While the exact combination of spatial relations and action-outcome prediction to the best of my knowledge is novel, both of the ideas are well known. Action-outcome prediction is a popular tool for both training and evaluation (e.g. the CLEVRER dataset and it's derivatives), while spatial relation question answering and spatial representation probing is an even more common idea (e.g. Open3D-VQA, SpatialVLM, SpaRRTa, etc.).
  • The paper positions the benchmark as evaluating action-outcome prediction on spatial relations. However, the benchmark itself tests only a narrow set of synthetic transforms of positions, without considering physical consequences and interactions. The claimed contribution should be narrowed to reflect that.
  • The proposed benchmark is a very small set (only 182 scenes), even those are as I understand mostly procedurally generated test cases from images from ScanNet(++), with generally similar content (indoor, typical rooms). Questions generated from the same scene share objects, geometry, appearance, and spatial configurations. They are not independent observations of a model's generalization capability.
  • Manual inspection of a several test cases show that at least some of them have low quality images, where the main issue with completing the test case is bad image quality rather than spatial reasoning itself. This is likely why the human performance is lower on some cases, and introduces a major perception confound the the spatial reasoning evaluation.
  • Fine tuning results are somewhat odd - the GRPO result only 3.7% better overall than the Qwen3-vl-4B baseline, while SFT is 31.7% better. My guess here is that the GRPO training setup is too simplistic or incorrect, especially as SFT for question answering tasks usually at first learns to just follow the answer template which improves results significantly for small models. Moreover, regardless of the SFT/GRPO choice, the improvement on other benchmarks is significantly smaller, which would support SFT just learning the template or finding some shortcut in the data. An interesting here would be to run GRPO on the SFT trained checkpoint.
Questions for authors
  • One of my main concerns is if the spatial perception limits the spatial reasoning capabilities shown in the benchmark - could you provide a text-only baseline for the task, based on the annotations of the source dataset? I.e. provide the evaluated model with ground-truth 3d object coordinates instead of the image input to test how the result will change when the perception errors are eliminated.
  • Are the objects affected by the actions always the target of the benchmark question? It would be interesting to see a negative test - what if we ask a question about an object that is NOT affected by the action to see if the action text affect the model reasoning about the static part of the scene.
  • For the global transformation tasks the rotations are limited to 90, 180 and 270 deg, which essentially limits the evaluation to very simple cases - have you tried more complex transformations?
  • Can you check if the supervised fine-tuning actually learns general predictive reasoning rather than just adapting to the benchmark rules and templates? A comparison against a similarly sized static-spatial training dataset would also help isolate the contribution of predictive training.
Reviewer_ZE9GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing spatial reasoning benchmarks mainly assess models’ perception of observed scenes, with limited evaluation of their ability to predict unobserved states after scene transformations. SpaceCast-Bench addresses this gap using 3D geometry from real indoor scenes and an “Observe–Transform–Infer” framework. It contains 3,862 questions across 16 task types, covering static perception, local prediction, and global prediction. Across 21 vision-language models, the best accuracy is 58.0%, versus 87.2% for humans. Fine-tuning on generated data also improves performance on the benchmark and out-of-domain tasks.

Critical strengths
  1. Ten annotators are involved; Each question is independently checked by at least two people, with disagreements adjudicated by a third. This reduces ambiguity and annotation errors, improving evaluation reliability.
  2. The pipeline generates multi-view observations from real scenes, applies controlled transformations, and automatically derives answers, offering a practical foundation for expanding the benchmark.
Critical weaknesses
  1. L3 applies rigid rotations to entire scenes, and its questions mainly concern directions. A model may identify initial object relations and mechanically rotate directional labels without updating a comprehensive global scene representation.
  2. L1 and L2 differ not only in transformation requirements but also in scene distributions: nearly all L2 questions come from ScanNet++, compared with about two-thirds of L1. Thus, lower L2 performance may reflect harder scenes or distribution shifts rather than weaker state updating.
Questions for authors
  1. Could a baseline identify the initial spatial relations and apply mathematical rotation rules to predict L3 answers? Comparing its performance with VLMs would clarify whether L3 tests reasoning beyond directional conversion.
  2. Could L1 and L2 scenes be evaluated under matched conditions to determine whether their initial perception difficulty is comparable? This would help isolate the contribution of spatial state updating to the performance drop.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.