acceptodds
Under review as a conference paper at ICLR 2027

From Sight to Foresight: Predictive Spatial Reasoning in Vision-Language Models

Abstract

Predicting future spatial states supports collision avoidance and timely decision-making in dynamic environments. However, existing vision-language models (VLMs) and benchmarks for spatial reasoning primarily focus on observed scenes, leaving predictive spatial reasoning beyond the observed interval underexplored. To this end, we introduce SpatialMind, a metric-scale VLM for spatial reasoning and future prediction. Its metric depth adapter anchors spatial reasoning to real-world scale, while its progressive state chain establishes current spatial states and observed dynamics as the foundation for future prediction. Given a video prefix, SpatialMind predicts distances, motion directions, and spatial relations in both observed and unseen future frames. For training and evaluation, we build a scalable data engine that grounds entity descriptions in metric geometry to generate question-answer pairs and state supervision. Using this engine, we construct the SpatialMind-30K dataset and the SpatialMind-2K benchmark, both covering driving and everyday egocentric scenes. The benchmark spans eight tasks across three levels: current-state understanding, observed-dynamics understanding, and future prediction. Experiments show that SpatialMind substantially outperforms both general and spatially specialized models on our benchmark while achieving competitive zero-shot performance on VSI-Bench, OSI-Bench, and VLM4D.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Reasoning, Vision-language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FBbMOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Existing VLMs and spatial reasoning benchmarks mostly focus on observed scenes, while predicting future spatial states is still underexplored. This paper proposes SpatialMind, a metric-scale VLM for spatial reasoning and future prediction. Given a video prefix, it predicts distances, motion directions, and spatial relations in both observed and unseen future frames. A metric depth adapter anchors reasoning to real-world scale. A progressive state chain then has the model work out the current state and the observed motion before predicting the future. The authors also build a data engine on Waymo and Ego-Exo4D, which produces the SpatialMind-30K dataset and the SpatialMind-2K benchmark. SpatialMind clearly beats existing VLMs on this benchmark and shows competitive zero-shot results on VSI-Bench, OSI-Bench, and VLM4D.

Critical strengths
  1. The paper is clearly written and easy to follow. The three-level view is well motivated, and the figures make the data pipeline and the model easy to understand.
  2. Each step of the method is explained clearly and the logic is easy to follow. The data engine is also carefully built.
  3. The experiments are thorough, with a wide range of baselines, ablations of both components, and extra diagnostic analyses. The worked examples in the appendix make the method very easy to understand.
Critical weaknesses
  1. Most of the gain comes from the new data. Fine-tuning Qwen3-VL-8B on SpatialMind-30K alone raises the score from 20.20 to 45.70, while the two new modules add only about 7 more points (to 52.65).
  2. The future-prediction claim is only tested on the authors' own benchmark, but the paper did not evaluate on the most related future-prediction benchmarks like STRIDE-QA and SCP-Bench.
  3. On VSI-Bench, the base model Qwen3-VL-8B is not listed. The Qwen3-VL Technical Report [1] reports 59.4 for Qwen3-VL-8B-Instruct, and SenseNova-SI [2] reports 57.9, but SpatialMind only gets 46.4.
  4. Some highly related work is not cited, such as ST-VLM [3], which builds a very similar data pipeline for kinematic QA. Spatial-MLLM [4] is used as a baseline in Table 2 but never cited.
  5. Several implementation details are missing, including the values of ρρ, λλ, γgγ_g, γlγ_l, the LoRA and training settings, the MRA tolerance settings.

References: [1] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, et al. Qwen3-VL Technical Report. arXiv:2511.21631, 2025. [2] Scaling Spatial Intelligence with Multimodal Foundation Models (SenseNova-SI). arXiv:2511.13719, 2025. [3] Dohwan Ko, Sihyeon Kim, Yumin Suh, Vijay Kumar B.G, Minseo Yoon, Manmohan Chandraker, Hyunwoo J. Kim. ST-VLM: Kinematic Instruction Tuning for Spatio-Temporal Reasoning in Vision-Language Models. arXiv:2503.19355, 2025. [4] Diankun Wu, Fangfu Liu, Yi-Hsin Hung, Yueqi Duan. Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence. NeurIPS 2025.

Questions for authors
  1. Please report zero-shot Qwen3-VL-8B on VSI-Bench under your protocol and explain the gap to the published 57.9 to 59.4.
  2. If possible, please evaluate on STRIDE-QA and SCP-Bench. If this is not feasible, please explain why.
  3. Please provide the hyperparameters listed above.
  4. Please cite and discuss closely related work such as ST-VLM and Spatial-MLLM.
6 comments
AuthorsOpenReview ↗

Response to Reviewer FBbM [1/5]

We sincerely appreciate your valuable time and efforts in reviewing this paper. We thank the thoughtful feedback you provided, which significantly improved the quality of this paper. For the potential concerns you bring up, we would like to answer/address them here.

Contributions of Dataset and Method

Most of the gain comes from the new data. Fine-tuning Qwen3-VL-8B on SpatialMind-30K alone raises the score from 20.20 to 45.70, while the two new modules add only about 7 more points (to 52.65).

Thank you for highlighting the substantial contribution of SpatialMind-30K. This improvement aligns with the domain gap between existing spatial reasoning datasets and tasks requiring future spatial prediction. As shown in Table 1 of the main paper, most existing models achieve scores below 20% on SpatialMind-2K, reflecting the importance of task-specific supervision for adapting general-purpose VLMs to predictive spatial reasoning. This observation also motivates our development of the data engine and benchmark to support future spatial prediction. Building on this, the metric adapter and progressive state chain further improve performance from 45.70% to 52.65% (+6.95 pp), supporting their additional contributions beyond supervised fine-tuning alone.

AuthorsOpenReview ↗

Response to Reviewer FBbM [2/5]

Evaluation on External Future Prediction Benchmarks

The future-prediction claim is only tested on the authors' own benchmark, but the paper did not evaluate on the most related future-prediction benchmarks like STRIDE-QA and SCP-Bench.

If possible, please evaluate on STRIDE-QA and SCP-Bench.

We extend our evaluation to STRIDE-QA [1] and SCP-Bench [2] in a zero-shot setting, following their original input and scoring protocols. First, STRIDE-QA evaluates future localization in urban driving, requiring models to predict a target agent's distance and direction relative to the ego vehicle at different future horizons. As shown in Table 1, SpatialMind achieves consistent improvements over our baseline, Qwen3-VL-8B, across the three prediction horizons, with notable gains at 1 and 2 seconds compared with the other methods.

Table 1. STRIDE-QA: Future localization success rate (%, ↑).

ModelLSR @ 1 sLSR @ 2 sLSR @ 3 s
InternVL2.5-8B1.41.71.7
Cosmos-Reason1-7B3.22.01.5
Qwen3-VL-8B2.71.72.7
SpatialMind9.38.86.8

We further evaluate on SCP-Bench, which examines spatial reasoning about unobserved past and future states from partially observed videos. As shown in Table 2, SpatialMind achieves consistent improvements over Qwen3-VL-8B and Spatial-MLLM, with competitive performance compared to the larger Qwen3-VL-30B across overall, relation, and relative-distance metrics, indicating improved spatial reasoning about unobserved states.

Table 2. SCP-Bench: Accuracy (%, ↑).

ModelOverallRelationRelative distance
Spatial-MLLM39.7638.3349.73
Qwen3-VL-8B47.5242.3349.47
Qwen3-VL-30B54.1646.2256.65
SpatialMind54.9650.8059.57

Overall, these zero-shot evaluations support the generalization of SpatialMind's spatial prediction capabilities beyond our benchmark. We have added the zero-shot evaluation results on STRIDE-QA and SCP-Bench, together with the evaluation settings, in Appendix O.

AuthorsOpenReview ↗

Response to Reviewer FBbM [3/5]

VSI-Bench Evaluation and Input Settings

On VSI-Bench, the base model Qwen3-VL-8B is not listed. The Qwen3-VL Technical Report [3] reports 59.4 for Qwen3-VL-8B-Instruct, and SenseNova-SI [4] reports 57.9, but SpatialMind only gets 46.4.

Please report zero-shot Qwen3-VL-8B on VSI-Bench under your protocol and explain the gap to the published 57.9 to 59.4.

We thank the reviewer for highlighting the need to clarify the evaluation settings and include the base model. Our original VSI-Bench evaluation uses 8 input frames, while SenseNova-SI [4] uses 32 frames. To comprehensively analyze performance, we evaluate Qwen3-VL-8B and SpatialMind on VSI-Bench in a zero-shot setting with both 8 and 32 input frames.

As shown in the table, SpatialMind achieves comparable performance to the baseline under both settings. Furthermore, increasing the input to 32 frames substantially improves the performance of both models, bringing their scores close to the published 57.9–59.4% [3, 4] and suggesting that input frame count contributes substantially to the reported score gap.

Table 3. VSI-Bench performance under different input frame settings (%, ↑).

Input framesQwen3-VL-8B-InstructSpatialMind
8 frames49.3546.40
32 frames57.2958.63

We are not surprised that SpatialMind achieves comparable performance to the baseline on VSI-Bench, because SpatialMind-30K does not include training examples for room-layout reasoning. VSI-Bench is built from room-scanning videos and evaluates spatial layouts and relationships across viewpoints, while SpatialMind-30K, as introduced in Sec. 3, focuses on dynamic scenes and future spatial prediction.

Despite this domain gap, SpatialMind maintains comparable performance on VSI-Bench while achieving notable improvements in dynamic scene reasoning (OSI-Bench, VLM4D) and future spatial prediction (STRIDE-QA, SCP-Bench), suggesting improved predictive spatial reasoning capabilities while preserving general spatial understanding. We include the VSI-Bench results under 8- and 32-frame inputs and the corresponding discussion in Appendix P.

Reviewer_wJ3XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies predictive spatial reasoning in vision-language models (VLMs). The authors argue that existing VLMs mainly focus on understanding observed scenes, while their ability to predict future spatial states from past observations remains relatively underexplored.

To address this problem, the authors introduce SpatialMind, along with the SpatialMind-30K dataset and SpatialMind-2K benchmark. They use object tracking, depth estimation, and geometric information from videos to construct question-answer pairs grounded in real-world spatial measurements. The tasks are organized into three levels:

  1. Current-state understanding
  2. Observed-dynamics understanding
  3. Future prediction

The proposed model includes two main components:

  1. Metric Depth Adapter: Incorporates depth and geometric information to improve the VLM's understanding of metric-scale spatial relationships.
  2. Progressive State Chain: Supervises the model to generate intermediate states, progressing from target grounding and current-state estimation to observed dynamics and future prediction.

Experiments show that SpatialMind outperforms several general-purpose and spatially specialized VLMs on SpatialMind-2K, while achieving competitive results on external spatial reasoning benchmarks.

Overall, the paper addresses a meaningful problem and presents a systematic approach with a well-developed data construction pipeline and experimental evaluation.

Critical strengths

Strengths

  1. Meaningful problem setting. Future spatial prediction is an important capability for embodied AI and autonomous driving. Compared with conventional VQA and static spatial understanding tasks, this paper focuses on predicting future spatial relationships from current observations and past dynamics, which has clear practical relevance.

  2. Valuable dataset construction. Compared with conventional VQA datasets based mainly on textual descriptions, the proposed dataset provides richer supervision grounded in metric geometry.

  3. Relatively comprehensive experiments. The full model improves over the SFT baseline trained on the same data, and the ablation studies provide evidence for the effectiveness of the proposed components.

Critical weaknesses

Weaknesses

  1. Insufficient evidence for the claimed predictive spatial reasoning capability. The authors claim that SpatialMind demonstrates strong spatial reasoning and future prediction capabilities. However, the current experiments mainly show performance gains from structured supervision. It remains unclear whether the model has learned transferable spatial dynamics or primarily benefits from structured CoT supervision.

  2. Limited methodological novelty. The Metric Depth Adapter is closely related to existing geometry-enhanced VLM approaches, while the Progressive State Chain shares substantial similarities with structured CoT and intermediate-state supervision.

  3. Lack of simple but important baselines for future prediction. Appendix E.1 shows that, for 81.8% of the future-distance questions in Ego-Exo4D, simply using the annotated current distance as the future prediction falls within the primary evaluation tolerance. This suggests that some test cases may not require complex motion prediction. The authors should include simple baselines such as persistence and constant-velocity extrapolation, and analyze performance under different magnitudes of future-state change to better assess the model's predictive capability.

Questions for authors

1.How can we tell whether SpatialMind actually learns to reason about future spatial states, rather than simply benefiting from structured CoT supervision? For example, how much does its future prediction performance change when only the last observed frame is provided instead of the full video?

2.Could the authors compare Progressive State Chain with a standard CoT-SFT baseline trained on the same data and with a comparable training budget? The free-form CoT baseline in Table 4 is not fine-tuned, making it difficult to judge the specific benefit of the proposed state chain.

3.How does SpatialMind perform against simple prediction baselines such as persistence and constant-velocity extrapolation? Given the findings in Appendix E.1, it would be particularly useful to see whether the model still performs better on samples where the future state differs substantially from the current state.

11 comments
AuthorsOpenReview ↗

Response to Reviewer wJ3X [1/4]

We sincerely appreciate your valuable time and efforts in reviewing this paper. We thank the thoughtful feedback you provided, which significantly improved the quality of this paper. For the potential concerns you bring up, we would like to answer/address them here.

Predictive Spatial Reasoning and Transferability

Insufficient evidence for the claimed predictive spatial reasoning capability. ... However, the current experiments mainly show performance gains from structured supervision. It remains unclear whether the model has learned transferable spatial dynamics or primarily benefits from structured CoT supervision.

How can we tell whether SpatialMind actually learns to reason about future spatial states, rather than simply benefiting from structured CoT supervision? For example, how much does its future prediction performance change when only the last observed frame is provided instead of the full video?

Thank you for highlighting the importance of evaluating SpatialMind's predictive spatial reasoning capability and its transferability.

The substantial improvement from supervised fine-tuning (SFT) on SpatialMind-30K highlights the contribution of our data engine in bridging the domain gap between general-purpose VLMs and predictive spatial reasoning. By constructing metric annotations of current spatial states, observed dynamics, and future states, our data engine provides supervision for learning these capabilities. As shown in Table 1 of the main paper, fine-tuning Qwen3-VL-8B on SpatialMind-30K improves performance from 20.20% to 45.70%. Building on this foundation, the metric adapter and progressive state chain further improve performance to 52.65% (+6.95 pp), demonstrating their additional contributions beyond SFT alone.

Temporal Input Ablation: To examine whether SpatialMind uses observed temporal information for future prediction, we compare its performance under three input settings: (1) original video, which provides the complete observed sequence; (2) last observed frame, which provides only the final frame; and (3) temporally shuffled video, which provides the original frames in random temporal order.

Table 1. Future prediction with different input observations.

Input observationFuture direction (ACC)Future distance (MRA)Future relation (ACC)
Last observed frame46.1540.8958.60
Temporally shuffled video45.4537.4949.12
Original video65.1342.8964.48

As shown in Table 1, the original video input yields the highest scores across all three future prediction tasks, with particularly large differences in Future Direction and Future Relation. Since the model and reasoning structure remain unchanged across these settings, these results support SpatialMind's use of observed temporal information to predict future spatial states.

Zero-Shot Transferability: To further address the reviewer's concern about the transferability of predictive spatial reasoning, we conduct zero-shot evaluations on two external benchmarks, STRIDE-QA [1] and SCP-Bench [2]. STRIDE-QA evaluates future localization at different prediction horizons, while SCP-Bench examines spatial reasoning about unobserved past and future states. As shown in Table 2, SpatialMind achieves notable improvements over Qwen3-VL-8B on both benchmarks, supporting its ability to generalize predictive spatial reasoning beyond our SpatialMind-2K benchmark.

Table 2. Zero-shot evaluation on external spatial prediction benchmarks.

BenchmarkMetricQwen3-VL-8BSpatial-MLLMGPT-4oSpatialMind
STRIDE-QAMean LSR (%)2.37—6.778.30
SCP-BenchOverall ACC (%)47.5239.76—54.96

Together, these results support SpatialMind's use of observed dynamics to predict future spatial states and its zero-shot generalization to external future prediction benchmarks. We further examine the specific contribution of Progressive State Chain through the matched CoT-SFT comparison in the following response. We have added the experimental details and results for the temporal input ablation and zero-shot external benchmark evaluations in Appendices J and O, respectively.

AuthorsOpenReview ↗

Response to Reviewer wJ3X [2/4]

Progressive State Chain and CoT Comparison

Progressive State Chain shares substantial similarities with structured CoT and intermediate-state supervision.

Could the authors compare Progressive State Chain with a standard CoT-SFT baseline trained on the same data and with a comparable training budget? The free-form CoT baseline in Table 4 is not fine-tuned, making it difficult to judge the specific benefit of the proposed state chain.

We thank the reviewer for encouraging a comparison between Progressive State Chain (PSC) and structured Chain of Thought (CoT). Though both share intermediate reasoning steps to guide the model, PSC organizes them into three temporally connected stages: estimating current spatial states, analyzing observed dynamics, and predicting future states. Standard spatial CoT, such as the DETECT, THINK, and ANSWER format in SR-ReaL [3], guides the model through object grounding (DETECT) and geometric reasoning (THINK) to derive the final answer, with less emphasis on how observed dynamics inform future spatial states.

To compare Progressive State Chain with standard CoT-SFT baseline, we conduct three SFT experiments using Qwen3-VL-8B under the same training settings: (1) Direct SFT, which directly predicts the answer; (2) CoT-SFT, which follows the DETECT, THINK, and ANSWER CoT format of SR-ReaL [3]; and (3) PSC-SFT, which follows our progressive state chain. Specifically, we construct 30,000 CoT training examples from the same computation chains in SpatialMind-30K, with the same number of training examples as PSC-SFT. To ensure a fair comparison, we train all three models with RGB video inputs, without our metric adapter, using the same training budget of 8,000 updates over 20 epochs on six NVIDIA B200 GPUs.

Table 3. Comparison of Direct SFT, CoT-SFT, and PSC-SFT.

MethodOverallL1: Motion stateL1: Spatial stateL2: Ego distanceL2: Object distanceL2: Relative MotionL3: Future directionL3: Future distanceL3: Future relation
Direct SFT45.8335.9458.1624.5738.9755.9456.6441.2557.19
CoT-SFT47.3935.3264.3426.2347.5666.4347.5541.6550.18
PSC-SFT51.1436.2766.4428.6349.1669.6360.2442.7560.05

As shown in the table, both CoT-SFT and PSC-SFT improve overall performance over Direct SFT, indicating the benefits of intermediate reasoning supervision. PSC-SFT further achieves consistent improvements over CoT-SFT, particularly in Future Direction and Future Relation prediction. These results suggest that the proposed Progressive State Chain helps the model establish current spatial states, analyze observed dynamics, and further leverage these intermediate estimates to guide accurate future spatial prediction. We provide the detailed training setup and full comparison of Direct SFT, CoT-SFT, and PSC-SFT in Appendix L.

AuthorsOpenReview ↗

Response to Reviewer wJ3X [3/4]

Absolute Metric Depth Estimation

The Metric Depth Adapter is closely related to existing geometry-enhanced VLM approaches.

We appreciate the reviewer's comment on the connection between our Metric Depth Adapter and existing geometry-enhanced VLMs. Existing spatial reasoning methods leverage pretrained geometry to support spatial understanding: VLM-3R utilizes a frozen CUT3R [4] to fuse geometric features into the VLM, while SpatioLM directly uses DA3 as a geometry teacher to train its spatial module [5]. Unlike these approaches, our Metric Depth Adapter explicitly adjusts depth predictions to accommodate the substantially different distance scales of indoor and outdoor scenes. We train the adapter with metric depth supervision to learn scene-dependent corrections, enabling adaptive scale adjustment and more accurate predictions of real-world distances across diverse environments.

To evaluate absolute metric depth accuracy, we compare CUT3R (used in VLM-3R), DA3 (used in SpatioLM), and our Metric Depth Adapter on Waymo, Ego-Exo4D, and the external KITTI-360 dataset [6]. We compare predicted depth against metric depth references from the source datasets, including Waymo LiDAR and Ego-Exo4D Aria 3D points. For Waymo and Ego-Exo4D, we use the same sampled frames and reference points across models to ensure a consistent comparison. For KITTI-360, we evaluate on the test split using its ground-truth depth. We evaluate all predictions at their native output scale, without scale alignment using the evaluation depth references.

Table 4. Absolute metric depth accuracy across different datasets.

Geometry modelWaymo AbsRel ↓Waymo MSE ↓Ego. AbsRel ↓Ego. MSE ↓KITTI-360 AbsRel ↓KITTI-360 MSE ↓
CUT3R0.22550.100.6480.7730.19667.17
DA30.97472.050.5360.9910.874384.3
Our0.12229.640.2180.3430.14061.34

As shown in the table, our Metric Depth Adapter achieves consistently lower AbsRel and MSE across the three datasets, indicating improved alignment with absolute metric depth and supporting its contribution to metric spatial reasoning. We describe the absolute metric depth evaluation protocol and report the full results in Appendix M.

Reviewer_15TfOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpatialMind, a vision-language framework for metric-scale spatial reasoning and future-state prediction. The method combines an absolute metric depth adapter with a progressive state chain that connects current spatial understanding, observed dynamics, and future prediction. The authors also introduce SpatialMind-30K and SpatialMind-2K, a training dataset and benchmark covering eight spatial reasoning tasks across driving and egocentric scenarios. Experiments demonstrate substantial improvements over a fine-tuned VLM baseline on the proposed benchmark, together with competitive zero-shot performance on three external spatial reasoning benchmarks.

Critical strengths
  1. Meaningful problem formulation and dataset contribution. The paper extends spatial reasoning beyond observed scenes to metric-scale future prediction, addressing a relevant problem for vision-language models. The proposed dataset and benchmark cover multiple reasoning levels and two distinct visual domains, providing a potentially valuable resource for studying predictive spatial understanding.

  2. Competitive results across multiple benchmarks. SpatialMind substantially improves over the fine-tuned Qwen3-VL baseline on SpatialMind-2K and demonstrates competitive zero-shot performance on VSI-Bench, OSI-Bench, and VLM4D. These external evaluations provide useful evidence that the learned spatial representations transfer beyond the proposed benchmark.

Critical weaknesses
  1. Insufficient evidence of genuine future predictive reasoning. A central claim of the paper is that SpatialMind learns to predict future spatial states from observed dynamics. However, Appendix E.1 reveals substantial temporal persistence in the benchmark: retaining the annotated current distance satisfies the future-distance tolerance for 81.8% of egocentric questions, while current spatial relations remain unchanged in a large fraction of future-relation questions. Although these diagnostics are not directly comparable to model accuracy, they raise concerns about whether the benchmark sufficiently distinguishes genuine future prediction from static-state persistence. The lack of simple temporal extrapolation baselines and explicit temporal-dependency ablations makes the central claim less convincing.

  2. Limited ablation of methodological contributions and insufficient dataset validation. The proposed metric adapter and progressive state chain provide relatively modest improvements over simpler variants. In particular, removing the grounding step slightly improves Level 3 performance, raising quest1ions about the necessity of the full progressive reasoning structure for future prediction. Additionally, because the benchmark relies on automatically reconstructed geometric annotations, more rigorous validation of label accuracy and training-test independence is necessary to establish the reliability of the reported improvements.

Questions for authors
  1. Could the authors compare SpatialMind against last-observation, constant-velocity extrapolation, and temporally shuffled-video baselines? It would also be valuable to report performance across different future prediction horizons, particularly on examples where future states differ substantially from the last observed states. These results would materially affect my assessment of the paper's central contribution.

  2. Can the authors provide ablations separately removing the CURRENT and DYNAMICS stages, together with evaluations of intermediate state prediction accuracy? This would help establish whether the proposed reasoning chain genuinely leverages observed dynamics rather than primarily benefiting from additional structured supervision.

  3. How are the training and evaluation sets separated at the recording or sequence level? Additionally, what quantitative validation was performed to assess errors in depth-derived geometric annotations, object trajectories, and automatically generated numerical answers?

  4. Could the authors report the inference overhead introduced by the geometric models and progressive state generation, as well as uncertainty estimates for the relatively small performance improvements observed in some ablations and external benchmarks?

12 comments
AuthorsOpenReview ↗

Response to Reviewer 15Tf [1/4]

We sincerely appreciate your valuable time and efforts in reviewing this paper. We thank the thoughtful feedback you provided, which significantly improved the quality of this paper. For the potential concerns you bring up, we would like to answer/address them here.

Assessing Genuine Future Spatial Prediction

Insufficient evidence of genuine future predictive reasoning. A central claim of the paper is that SpatialMind learns to predict future spatial states from observed dynamics. However, Appendix E.1 ... The lack of simple temporal extrapolation baselines and explicit temporal-dependency ablations makes the central claim less convincing.

Could the authors compare SpatialMind against last-observation, constant-velocity extrapolation, and temporally shuffled-video baselines? It would also be valuable to report performance across different future prediction horizons, particularly on examples where future states differ substantially from the last observed states. These results would materially affect my assessment of the paper's central contribution.

Thank you for highlighting the importance of distinguishing genuine future prediction from static-state persistence. To address this concern, we conduct two additional studies: temporal extrapolation and temporal-dependency ablations.

Temporal Extrapolation Ablation: We first introduce two baselines and conduct temporal extrapolation experiments: (1) Last-observation persistence, which uses the estimated last-observed state as the future prediction; and (2) constant-velocity extrapolation , which linearly extrapolates the estimated observed motion to the queried future time.

Besides, to evaluate performance across different magnitudes of future-state change, we group questions by annotated state changes: (1) Future Distance Change: Small/Large distinguishes changes within or above 0.3 m in indoor scene (Ego-Exo4d) and 7 m in outdoor scene (Waymo); (2) Future Relation Change: Unchanged/Changed indicates whether the spatial relation changes between current and future states.

As shown in the Table, SpatialMind achieves higher scores than both baselines on the full sets and, particularly on the subsets with large distance changes and changed spatial relations. We additionally evaluate future prediction on STRIDE-QA [1] at various future horizons of 1, 2, and 3 seconds and report the results in Appendix O. We have added the detailed results and analysis of the temporal extrapolation baselines in Appendix I.

Table 1. Temporal extrapolation across future-state changes.

Prediction methodDistance: FullDistance: SmallDistance: LargeRelation: FullRelation: UnchangedRelation: Changed
Last-observation36.7959.737.5944.5665.0914.66
Constant-velocity39.5458.5115.4045.9657.9928.45
SpatialMind43.2760.2421.6764.2185.2133.62

Temporal-Dependency Ablation: We then evaluate SpatialMind’s temporal-dependency capability by comparing with different input: the original video, the last observed frame which provide only last frame as input, and temporally shuffled video which shuffled the input video order. As shown in the Table below, SpatialMind achieves higher scores across all three future prediction tasks with the original video,showing the importance of observed scenes and temporal order for Spatialmind to make future spatial prediction. We have added the detailed results and analysis of the temporal input ablation in Appendix J.

Table 2. Temporal dependency across future prediction tasks.

Video inputFuture direction (ACC)Future distance (MRA)Future relation (ACC)
Last observed frame46.1540.8958.60
Temporally shuffled video45.4537.4949.12
Original video65.1342.8964.48

Together, these studies provide complementary evidence for genuine future spatial prediction from both benchmark and model perspectives. The temporal extrapolation ablation shows that SpatialMind achieves better performance than simple prediction baselines on SpatialMind-2K, particularly when future states differ substantially from observed states. The temporal-dependency ablation further demonstrates that SpatialMind leverages the complete, temporally ordered observations to reason about future spatial states.

AuthorsOpenReview ↗

Response to Reviewer 15Tf [2/4]

Ablation on Methodological Contributions

The proposed metric adapter and progressive state chain provide relatively modest improvements over simpler variants. In particular, removing the grounding step slightly improves Level 3 performance, raising quest1ions about the necessity of the full progressive reasoning structure for future prediction.

Can the authors provide ablations separately removing the CURRENT and DYNAMICS stages, together with evaluations of intermediate state prediction accuracy? This would help establish whether the proposed reasoning chain genuinely leverages observed dynamics rather than primarily benefiting from additional structured supervision.

We thank the reviewer for encouraging a more detailed evaluation of our methodological contributions. The substantial improvement from supervised fine-tuning (SFT) on SpatialMind-30K highlights the contribution of our data engine in bridging the domain gap between general-purpose VLMs and predictive spatial reasoning. By providing metric annotations of current spatial states, observed dynamics, and future states, our data engine supports learning these capabilities, improving Qwen3-VL-8B from 20.20% to 45.70%. Building on this foundation, the metric adapter and progressive state chain further improve overall performance to 52.65% (+6.95 pp). To further examine the contribution of the progressive state chain, we evaluate its individual stages and intermediate state estimates below.

Explicit grounding plays different roles across reasoning levels. GROUND first explicitly guides the model to identify the queried entity and its spatial reference, improving L1 and L2 performance. For L3, without the explicit GROUND step, CURRENT and DYNAMICS implicitly perform the necessary grounding as a foundation for subsequent spatial reasoning, allowing SpatialMind to maintain comparable L3 performance, with a difference of approximately 0.1 percentage points.

Individual Stage Ablation: To evaluate the contribution of individual stages, we conduct inference-time ablations across all three reasoning levels using the same checkpoint, prompting the model to skip generating GROUND (identify the target), CURRENT (describe spatial state), and DYNAMICS (estimate observed changes) separately. As shown in the table, removing each stage lowers overall performance. GROUND contributes to L1 and L2, while CURRENT and DYNAMICS support L2 and L3. Together, these stages complement one another, allowing the full chain to achieve higher overall performance than the ablated settings. We provide further details on the stage ablations in Appendix K.

Table 3. Ablation of individual reasoning stages.

State chain settingOverallL1L2L3
w/o GROUND51.5549.4846.7457.58
w/o CURRENT46.2446.0845.3352.31
w/o DYNAMICS49.2151.3144.5051.83
Full state chain52.6551.8950.3057.47

Intermediate State Accuracy: We further evaluate the intermediate state estimates that SpatialMind computes while answering future-distance questions. For comparison, we directly query Qwen3-VL-8B + SFT to obtain the corresponding estimates as a baseline. Specifically, we measure CURRENT against the annotated distance at the last observed frame and DYNAMICS against the annotated distance change over the observed interval. As shown in the table, SpatialMind reduces CURRENT MAE by 45.60% and DYNAMICS MAE by 26.08% compared with the baseline, indicating more accurate intermediate state estimation and supporting its use of observed dynamics for future spatial prediction. We also include the intermediate-state evaluation details and results in Appendix K.

Table 4. Intermediate state estimation on future-distance questions.

MethodCURRENT MAE (m) ↓DYNAMICS MAE (m) ↓
Baseline3.4212.926
SpatialMind1.8612.163
Relative MAE reduction45.60%26.08%

Overall, these results support the contribution of the full progressive state chain to overall spatial reasoning, with CURRENT and DYNAMICS playing important roles in future prediction. Further, the improved intermediate state accuracy suggests that SpatialMind effectively estimates and leverages observed spatial dynamics to guide future predictions.

AuthorsOpenReview ↗

Response to Reviewer 15Tf [3/4]

Dataset Independence and Annotation Quality

More rigorous validation of label accuracy and training-test independence is necessary to establish the reliability of the reported improvements.

How are the training and evaluation sets separated at the recording or sequence level?

Training and Evaluation Separation: We separate training and evaluation data at the source recording level. For Waymo, we follow the official sequence split; for Ego-Exo4D, we partition data by source recording, keeping all clips and QA pairs from the same recording within one partition. Our training set contains 30,677 QA pairs from 5,996 video clips, while the evaluation set contains 2,000 QA pairs from 357 clips derived from 172 source recordings, including 85 Waymo segments and 87 Ego-Exo4D source recordings. This separation ensures that training and evaluation use disjoint source recordings, maintaining training-test independence at the recording level. We have added further details on training–test independence in Appendix D.1.

More rigorous validation of label accuracy and training-test independence is necessary to establish the reliability of the reported improvements.

What quantitative validation was performed to assess errors in depth-derived geometric annotations, object trajectories, and automatically generated numerical answers?

Annotation Quality Validation: We construct spatial annotations using depth references and camera poses from the source datasets. To further assess geometric accuracy, we compare our estimated dense metric depth with the source depth references, obtaining AbsRel errors of 0.1133 on Waymo and 0.1842 on Ego-Exo4D. These depth estimates provide a geometric foundation for spatial QA construction.

For object trajectories in Waymo, we use the official 3D tracking annotations from the dataset as our trajectory references. For Ego-Exo4D, we further assess the consistency of reconstructed object positions by measuring the 3D position drift of static targets across video frames, with a mean drift of 0.0348 m indicating stable position estimates for these targets.

For numerical answers, we incorporated human review into our QA generation pipeline during dataset construction to assess the calculation process and answer correctness. We sampled questions across subtasks, identified the sources of annotation errors, and corrected and rechecked the affected questions. We further recompute numerical answers from geometric measurements and verify their agreement within the original tolerances. Together with the quantitative checks above, these procedures help establish the label quality of spatial annotations and numerical answers in our training and evaluation datasets.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

Dear ACs, SACs, and Reviewers,

We sincerely thank the reviewers, ACs, and SACs for their time and constructive feedback. We appreciate the reviewers’ recognition of the importance of future spatial prediction for VLMs and the value of our dataset contribution (Reviewers 15Tf, wJ3X), as well as the comprehensive experimental evaluation (Reviewers wJ3X, FBbM). The reviewers also highlighted the competitive results across multiple benchmarks and evidence of transfer beyond our benchmark (Reviewer 15Tf), together with the carefully built data engine and clear, easy-to-follow presentation (Reviewer FBbM).

We have posted detailed responses to all reviews and follow-up questions. To facilitate the discussion, we summarize below the main points raised by the reviewers and how we have addressed them:

  1. Evidence for future spatial prediction (Reviewers 15Tf, wJ3X). We add persistence and constant-velocity baselines using SpatialMind’s intermediate estimates, showing further improvements on large distance changes and changed spatial relations. We also conduct last-frame and shuffled-video ablations, supporting SpatialMind’s use of ordered observations for future prediction.

  2. Contributions of the data and model components (Reviewers 15Tf, wJ3X, FBbM). We clarify the dataset’s contribution and show that the progressive state chain consistently outperforms CoT-SFT across tasks under matched data and training settings. We also provide stage ablations, intermediate-state evaluations, and metric-depth comparisons.

  3. Transferability to external benchmarks (Reviewers wJ3X, FBbM). We add zero-shot evaluations on STRIDE-QA and SCP-Bench, showing improvements over Qwen3-VL-8B, including across three prediction horizons on STRIDE-QA. We also conduct matched 8- and 32-frame evaluations on VSI-Bench to clarify the effect of input frame count on performance.

  4. Data split and annotation quality (Reviewer 15Tf). We clarify the independence of the training and test sets, annotation quality checks, and our manual review and refinement process.

  5. Inference efficiency and reproducibility (Reviewers 15Tf, FBbM). We report end-to-end latency and performance variability over five runs and clarify practical considerations for time-sensitive applications. We also supplement the manuscript with detailed training and evaluation settings and an expanded discussion of related work.

The additional results and implementation details have been incorporated into the revised manuscript. We are more than happy to answer any further questions. We sincerely thank all reviewers, ACs, and SACs again for their time, effort, and constructive feedback, which have substantially helped us improve our paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.