acceptodds
Desk rejected at ICLR 2027

T4D: Comprehending the Dynamic World with MLLMs

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have made rapid progress on 3D spatial reasoning, yet these successes remain confined to a static world. Real scenes evolve, and understanding them requires reasoning jointly over space and time—4D spatial reasoning. Existing benchmarks judge spatial change only qualitatively and leave dynamic egocentric–allocentric transformation untested; at comparable scale, open-source and expert models remain weak on dynamic scenes. We present T4D, a systematic study of dynamic spatial intelligence. We revisit spatial intelligence and, for the first time, construct the taxonomy of this field from a dynamic perspective. This yields six dimensions, instantiated as T4D-Bench: 10 tasks and 3,250 expert-verified QA pairs. Two difficulties stand in the way of scale: the clips do not share one 4D annotation—outdoor video supplies boxes, tracks, and ego poses, while egocentric video needs another geometric readout—and a language reference is ambiguous among similar objects. Our automated pipeline computes each answer from the native measurements and marks the queried entity on the frame, producing the benchmark and about 600K training QA pairs. On the algorithm side, (1) we propose the T4D network. It fuses explicit 3D point trajectories of instances with vision, and it attains state-of-the-art on T4D-Bench among MLLMs of comparable scale. (2) Trajectory-aligned training (TAP), a training recipe, supervises those instance trajectories in a fixed reference frame, aligning 2D vision tokens with the trajectory tokens and improving dynamic metric reasoning.

Primary area:
datasets and benchmarks
Keywords:
dynamic world comprehension, 4D spatial intelligence, video question answering, multimodal large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KWkmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper, T4D: Comprehending the Dynamic World with MLLMs (under review at ICLR 2027), addresses the limitation that existing multimodal large‑language models excel at static 3D spatial reasoning but struggle with 4D space‑time reasoning for dynamic real‑world scenes. It first establishes a six‑dimensional taxonomy for dynamic spatial intelligence and introduces T4D‑BENCH, a benchmark with 10 tasks and 3,250 expert‑validated QA pairs covering all six dimensions, alongside an automated pipeline generating around 600K training QA pairs from real‑world indoor and outdoor videos. The authors design the T4D network, which fuses explicit instance‑level 3D object trajectories with visual features via Position‑Addressed 4D Fusion (PAF), and propose Trajectory‑Aligned Training (TAP), a multi‑stage training recipe to prevent the model from taking 2D‑appearance shortcuts by aligning visual tokens and trajectory tokens. Evaluations show that the 4B‑parameter T4D model achieves state‑of‑the‑art performance on T4D‑BENCH among comparable‑scale open‑source MLLMs, though it still lags behind top‑tier proprietary models.

Critical strengths
  1. The paper builds a 6-dimensional classification framework for dynamic spatial intelligence from a dynamic perspective, points out the gaps in capabilities covered by the existing 4D benchmarks, and clarifies what abilities need to be evaluated for 4D spatial reasoning.

  2. The benchmark mentioned in the paper fully covers six major dynamic spatial dimensions and includes 3,250 QA entries manually verified by experts. It features both multiple-choice questions and open-ended physical value answers (like distance and speed), not just qualitative judgments.

  3. A lot of ablation experiments were done, confirming all the effects of the proposed method.

Critical weaknesses
  1. The model's performance heavily depends on the frozen SpatialTrackerV2 trajectory extractor. If the trajectory estimation is wrong, the error gets passed directly to the MLLM; once the trajectory module fails, the whole 4D inference is greatly affected, and there's no end-to-end joint optimization.

  2. The method proposed in the paper requires running 3D trajectory extraction in advance, which is more costly than regular video MLLM inference; during evaluation, it's limited to a maximum of 128 frames, so 4D inference on very long sequence videos hasn't been fully tested.

  3. Although T4D is SOTA among open-source models, its overall metrics are far below GPT‑6 Astra, being outperformed in all 10 tasks, which shows that dynamic 4D space reasoning still has a lot of room for improvement. Of course, this isn’t really the main weaknesses of the paper.

Questions for authors

This issue mainly affected my impression of this piece of work: Unfortunately, all key empirical findings are evaluated exclusively against the authors’ proposed T4D‑BENCH. Additional evaluations on out‑of‑distribution external benchmarks would be desirable to better verify the cross‑dataset generalisation capacity of the T4D method. Furthermore, most compared baseline models are trained on out‑of‑distribution data disjoint from T4D, which casts doubt on whether the reported performance improvements represent genuine advances in dynamic spatial reasoning or stem from training‑test distribution mismatches.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KWkm (1/2)

We thank the reviewer for the thoughtful review and for recognizing the six-dimension taxonomy, the expert-verified benchmark, and the ablation studies. We address each point below.

W1: Dependence on the frozen SpatialTrackerV2

We thank the reviewer for raising this important point. We acknowledge that a frozen tracker may pass errors downstream and is not jointly optimized with the MLLM. It provides explicit 3D trajectories beyond RGB and remains modular. On the same test set, ADEs are 29.73 m (Qwen3-VL-4B), 10.45 m (T4D), and 8.36 m (SpatialTrackerV2). Qualitatively, the backbone's decoded tracks tend to collapse toward the image center, as illustrated in the revised Figure 8. T4D scores 61.0 versus 46.5 for the same-data Qwen3-VL-4B on T4D-Bench, showing a clear system benefit without ruling out error propagation. Freezing the expert keeps the 4D evidence fixed, which makes the gains in Table 2 attributable to the training recipe. Replacing the tracker requires adapting only the lightweight projection/fusion modules, not the MLLM itself.

W2: Precomputation cost and the 128-frame limit

We appreciate the reviewer pointing this out. External experts add computation, an overhead inherent to pipelines that use an external geometry expert. During training, we pre-extract and cache expert features to avoid repeated tracker runs. At inference, the expert still adds cost, and T4D is slower than a native MLLM. Sparser feature-extraction points balance runtime and geometric coverage. All Table 1 models use 1 FPS and at most 128 frames, ensuring a consistent comparison; this is also in line with frame-sampled evaluation protocols used in spatial video benchmarks such as VSI-Bench and STI-Bench. This does not establish long-video ability: T4D-Bench targets short-clip geometry, kinematics, and cross-view reasoning, and we make no long-video claim.

AuthorsOpenReview ↗

Response to Reviewer KWkm (2/2)

W3: Gap relative to GPT-6 Astra

We thank the reviewer for the balanced comment and agree that GPT-6 Astra (78.0) substantially outperforms T4D (61.0), showing room for progress. T4D uses a 4B backbone and exceeds GPT-5.6 Sol (59.3) and Gemini 3.8 Flash (58.4). We do not speculate about proprietary models' parameter counts or training data; our claim is not that T4D surpasses them.

Question: External benchmarks and training-data distribution

We thank the reviewer for this important question, and we address it with two independent lines of evidence.

First, on external benchmarks: T4D obtains 69.0 Avg. on VSI-Bench, above VLM-3R (60.9), VG-LLM (47.3), and Spatial-MLLM (47.0); please see Appendix Table 6 for the full results.

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. OrderAvg.
Spatial-MLLM-4B65.334.863.145.141.346.233.546.347.0
VG-LLM-4B66.037.855.259.244.645.633.536.447.3
VLM-3R-7B70.249.469.267.165.480.545.440.160.9
T4D (4B)71.853.375.969.970.682.945.981.469.0

VSI-Bench evaluates static spatial understanding, so we treat it as complementary cross-benchmark evidence, not direct proof of dynamic 4D generalization. Baseline scores are from published results and were not rerun by us.

Second, on the training-distribution concern: for a controlled comparison, Qwen3-VL-4B scores 46.5, the best same-data spatial specialist 49.1, and T4D 61.0—a gap that training–test distribution mismatch alone cannot explain. We cannot verify whether other models' pretraining data overlaps with T4D-Bench and make no assumption about such overlap.

Thank you for the detailed responses. While most of my concerns have been addressed, I still see some limitations in the practical value of this work. I lean to borderline, but will finalize my assessment during the AC-reviewer discussion.

We are glad that most of your concerns have been addressed, and we hope the additional results and revisions will be helpful as you finalize your assessment during the AC discussion. We remain happy to answer any further questions.

Reviewer_K97BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

T4D introduces a benchmark, training data, and a geometry-augmented model for dynamic spatial understanding. T4D-Bench contains 3,250 questions across ten tasks, organized into six dimensions of spatial intelligence. The associated data pipeline generates approximately 600K training QA pairs from video datasets with native annotations or estimated geometric information. The model combines Qwen3-VL-4B with a frozen SpatialTrackerV2 through position-addressed fusion (PAF). Trajectory-aligned training (TAP) includes pointmap supervision, fixed-reference-frame trajectory supervision, and spatial QA. An additional mismatch task, TrajDQA, trains the model to reject geometry features taken from another video. The model achieves an overall score of 61.0 on T4D-Bench, compared with 46.5 for the same-data Qwen3-VL-4B baseline. The main methodological contribution is the adaptation of tracking representations to a language model through spatial addressing and trajectory supervision, rather than a new tracking algorithm.

Critical strengths

1.The benchmark combines several useful evaluation settings that are usually studied separately. It includes open numerical distance and speed estimation, camera motion, and paired-view tasks across indoor and outdoor video. The use of native annotations for metric questions, explicit entity markers, and manual screening of every benchmark item provides a concrete foundation for evaluation. Separating pseudo-trajectory data from the official evaluation split is also an appropriate design choice.

2.The experiments establish a substantial improvement over a same-data backbone and include relevant component ablations. T4D improves the overall score from 46.5 to 61.0 over Qwen3-VL-4B. Removing Stage B reduces absolute-speed performance from 42.0 to 26.9, supporting the usefulness of trajectory supervision. Removing PAF while retaining the full training recipe reduces the overall score from 61.0 to 58.7. These comparisons provide evidence for the system’s components beyond comparisons against zero-shot models alone.

Critical weaknesses

1.The experiments do not establish that T4D’s answers depend on the supplied dynamic geometry, despite this being the central justification for TAP and TrajDQA. Section 4.1 states that the tracker tokens concatenate 2D and 3D features. Improvements from adding these tokens therefore do not isolate the contribution of 3D trajectories from additional appearance or tracking features. More importantly, replacing the entire feature stream with one from another video does not close the appearance-only answering shortcut claimed in Section 4.3. A model can detect a coarse cross-video mismatch, output INCONSISTENT for those examples, and still answer matched examples primarily from RGB. The mismatch task does not require it to use the metric values or temporal evolution of the supplied trajectories. Table 2 shows that the training components improve accuracy, but it does not test this mechanism. The reported trajectory ADE of 10.45 m, compared with 8.36 m for SpatialTrackerV2, shows that a trajectory can be decoded with some fidelity; it does not establish the claim that the language model uses “nearly the full capability” of the encoder. Controls that preserve appearance while removing or altering the dynamic geometric information are missing. This leaves the paper’s principal methodological interpretation unverified.

2.The benchmark’s claimed coverage of six dimensions is not supported by tests that distinguish the intended capabilities from simpler solutions. Figures 3-5 assign tasks to dimensions, but task assignment alone does not establish what solving a task requires. Relative-size questions constructed from concatenated clips can be answered through independent object-size estimates rather than dynamic reasoning. Egocentric contact is explicitly defined as apparent touch or occlusion, not physical 3D contact. Perspective matching may benefit from appearance and scene correspondence without demonstrating metric transformation across viewpoints. These can all be useful tasks, but their scores do not by themselves establish comprehensive dynamic spatial understanding. In particular, the main text does not provide a controlled evaluation of the identity retention across occlusion described under D6, or single-frame and temporally disrupted baselines that establish which tasks require temporal information. This matters because the largest model gain is on perspective matching, whereas the gain over the strongest same-data specialist on absolute speed is only 0.4 points. The overall score therefore cannot substitute for evidence that the proposed model and benchmark address the advertised dynamic capabilities.

Questions for authors

1.Dependence on dynamic geometry: With RGB and the tracker’s 2D features preserved, how does performance change when the 3D component is removed or its temporal evolution is disrupted? Please distinguish the ability to detect mismatched streams from the ability to use trajectory geometry to answer matched examples. A direct geometric readout from the same tracker would also provide a useful reference for the distance and speed tasks.

2.Temporal addressing in PAF: In Equation (1), do (uℓ,vℓ)(u_\ell,v_\ell) denote the original query location or the tracked location in the current frame? For a moving point, these are different addresses. Please specify how the position attached to a per-frame track token corresponds to the current-frame vision token, including when the point is occluded. This determines whether PAF implements the spatial correspondence used to motivate the method.

3.Validation of benchmark capabilities: Which task scores remain high under a single-frame or temporally shuffled input? For D6, please identify the subset that requires maintaining object identity across an actual visibility gap and report performance on that subset. For perspective matching, a sparse-view control would help establish whether the improvement depends on temporal reasoning or primarily on cross-view appearance matching.

7 comments
AuthorsOpenReview ↗

#Response to Reviewer K97B (1/3)

We sincerely thank the reviewer for the exceptionally detailed and constructive assessment. The questions on geometry dependence, temporal controls, and PAF addressing identify exactly the issues that needed direct validation. Following these suggestions, we added analyses with geometry-feature zeroing, temporal shuffling, single-frame and sparse-view controls, an occlusion subset, and alternative PAF addressing, and we have narrowed our claims about the model and benchmark accordingly. We believe these controls make the evidence for T4D's dynamic-geometric reasoning substantially stronger, and we are grateful for the reviewer's guidance in strengthening the study.

W1: Do the experiments show that the model depends on dynamic geometry?

We agree that comparing the full T4D model only against Qwen3-VL-4B, or comparing performance before and after adding the complete tracker-token stream, cannot by itself isolate the contribution of 3D trajectories, because each SpatialTrackerV2 sync token contains both 2D and 3D features.

First, at inference time, we zeroed the entire tracker-token stream while retaining RGB. Under the same evaluation setting, Overall decreased from 61.0 to 54.9 (−6.1) for the full model with TAP, from 52.5 to 50.8 (−1.7) without TAP, and from 57.4 to 52.1 (−5.3) without TrajDQA. These results show that the trained model uses the tracker information. However, zeroing the complete stream alone does not distinguish the contributions of its 2D and 3D components.

To isolate the 3D component, we kept RGB and the 2D features from SpatialTrackerV2 unchanged and zeroed only its 3D features:

TaskNormal input3D features zeroed
Absolute distance66.557.5
Absolute speed42.036.9
Distance trend52.044.2
Relative size74.271.4
Relative speed51.947.4
Object count42.840.4
Camera path76.577.0
Camera yaw51.552.0
EgoLife / perspective matching79.272.6
EgoExo4D / egocentric contact73.072.0
Overall (mean over ten tasks)61.057.1

Zeroing only the 3D features reduced Overall from 61.0 to 57.1. The largest drops occurred on tasks that rely more heavily on 4D geometry: absolute distance decreased from 66.5 to 57.5, absolute speed from 42.0 to 36.9, distance trend from 52.0 to 44.2, and relative speed from 51.9 to 47.4. These results show that, even when RGB and 2D tracking features are preserved, the 3D tokens provide clear benefits for understanding spatial information in video.

To further test the temporal organization of the tracker information, we compared four input controls. 4D-token shuffle shuffles only the temporal order of the tracker tokens while preserving the RGB order. RGB + token shuffle applies the same permutation to the RGB frames and their corresponding tracker tokens, preserving frame-wise correspondence while disrupting temporal order. Single frame retains only the middle frame and its corresponding tracker tokens. Overall is the mean score across the ten tasks.

TaskNormal input4D-token shuffleRGB + token shuffleSingle frame
Absolute distance66.551.344.030.1
Absolute speed42.038.838.135.0
Distance trend52.048.645.239.2
Relative size74.270.271.836.4
Relative speed51.949.551.925.4
Object count42.841.124.922.9
Camera path76.580.083.554.5
Camera yaw51.550.050.549.0
EgoLife / perspective matching79.277.874.440.4
EgoExo4D / egocentric contact73.071.069.056.0
Overall (mean over ten tasks)61.057.855.338.9

The exact ten-task mean under 4D-token shuffle is 57.83, reported as 57.8. Overall is 55.3 under RGB + token shuffle and 38.9 under single-frame input. Compared with normal input, single-frame input reduces performance on all ten tasks, showing that multi-frame temporal information benefits the entire task suite. Both shuffle controls also reduce Overall, indicating that the temporal organization of the visual and tracker information matters for reasoning. Because 4D-token shuffle changes the temporal order of both the 2D and 3D components in each tracker token, this control alone does not attribute the effect to either component; the contribution of the 3D component is separately supported by the 3D-only zeroing experiment above.

AuthorsOpenReview ↗

Response to Reviewer K97B (2/3)

We also thank the reviewer for suggesting the trajectory output of Qwen3-VL-4B as a reference. We have updated Figure 8 and the main text. On the same trajectory test set, the mean ADE is 29.73 m for Qwen3-VL-4B, 10.45 m for T4D, and 8.36 m for SpatialTrackerV2. We further inspected the trajectories predicted by Qwen3-VL-4B and found that many collapse to a nearly fixed point around the image center; Figure 8 now shows one such example. Together with the 3D-only zeroing result, this comparison further supports that TAP alignment helps the MLLM use information supplied by the 4D encoder to understand spatial relations in video. Because T4D's ADE remains higher than that of SpatialTrackerV2, we have also removed our previous statement that the language model uses "nearly the full capability" of the tracker.

W2: Coverage of the six benchmark dimensions and validation of dynamic capabilities

We agree that assigning a task to a dimension does not by itself prove that the task necessarily requires dynamic reasoning. T4D-Bench does not claim that every question requires complete 4D dynamic information. Different tasks probe different capabilities in the six-dimensional framework, as indicated by the colored dimension markers under each task in Figure 5. Relative size evaluates the understanding of real-world 3D object scale rather than temporal reasoning. Egocentric contact is annotated as apparent visual contact or occlusion, rather than physical 3D contact. For perspective matching, the candidates are synchronized views from the same session and are marked in the input. This reduces some scene ambiguity but does not eliminate all appearance or cross-view correspondence cues. The sparse-view control in Q3 directly tests this alternative explanation.

To address whether the benchmark genuinely tests dynamic information, and whether the model can answer from single-frame appearance when temporal cues are absent, we added the temporal controls reported in W1. Single-frame input reduces the score on every task, while both shuffle controls reduce Overall. These results show that multi-frame temporal information benefits performance across the task suite. We treat this as benchmark-level evidence and do not claim that every individual question necessarily requires temporal reasoning.

For D6, we manually identified 100 questions in which the subject is occluded and subsequently reappears, forming an occlusion subset. It contains 32 distance-trend, 26 relative-speed, 21 absolute-speed, and 21 absolute-distance questions. With the four tasks weighted equally, Overall is 50.7 on the occlusion subset, 2.5 percentage points below the mean of the corresponding full-task scores (53.1). This suggests that performance may be slightly lower on questions involving occlusion.

Q1: Dependence on dynamic geometry

Please see W1. When only the 3D features are zeroed while RGB and the 2D features from SpatialTrackerV2 are retained, Overall decreases from 61.0 to 57.1 (−3.9), with particularly clear drops on absolute speed and distance trend. Zeroing the complete tracker stream and temporally shuffling the tracker tokens provide complementary evidence. Together, these results show that both the 3D component and the complete 4D-token stream contribute to stronger reasoning performance, and that T4D exhibits a clear dependence on dynamic geometric information.

AuthorsOpenReview ↗

Response to Reviewer K97B (3/ 3)

Q2: Temporal addressing in PAF

We especially appreciate the reviewer's careful reading of Equation (1) and the distinction between the original query location and the tracked location in the current frame. This is an important implementation detail, and it prompted us to clarify the submitted design and evaluate the suggested alternative.

In the submitted version, (uℓ,vℓ)(u_\ell,v_\ell) is the fixed initial query-grid address of track ℓ\ell. It is repeated across frames to provide a persistent track-slot identifier, while per-frame motion information is carried by the token content. Following the reviewer's suggestion, we also tested an alternative design in which the address at each frame is replaced by the corresponding traj2D coordinate decoded by SpaTrack (SpatialTrackerV2). The complete comparison below has also been added to a new PAF-addressing subsection in the supplementary material.

TaskFixed initial addressPer-frame SpaTrack traj2D address
Absolute distance66.572.0
Absolute speed42.044.6
Distance trend52.048.4
Relative size74.270.4
Relative speed51.956.0
Object count42.839.2
Camera path76.580.0
Camera yaw51.553.0
EgoLife / perspective matching79.275.2
EgoExo4D / egocentric contact73.074.0
Overall (mean over ten tasks)61.061.3

The per-frame addressing design achieves an Overall score of 61.3, a modest improvement of 0.3 over 61.0 with the fixed initial address, although the changes vary across tasks. We thank the reviewer for suggesting this insightful alternative design. We have incorporated both the clarification and the new experiment into the revised appendix, and this discussion has made the PAF description more precise.

Q3: Validation of benchmark capabilities

Please see the single-frame, temporal-shuffle, and occlusion results in W2. After temporal shuffling, relative size, camera path, perspective matching, and egocentric contact remain at 70.2/71.8, 80.0/83.5, 77.8/74.4, and 71.0/69.0, respectively, where each pair reports 4D-token shuffle followed by RGB + token shuffle. Under single-frame input, camera path, egocentric contact, and camera yaw score 54.5, 56.0, and 49.0, respectively. At the same time, single-frame input reduces all ten task scores relative to normal multi-frame input. This indicates that the tasks differ in their sensitivity to temporal information, rather than all depending on temporal cues to the same extent. Overall on the occlusion subset is 50.7, compared with 53.1 for the corresponding full-task scores. This suggests that performance may be slightly lower on questions involving occlusion.

For perspective matching, we also ran the sparse-view control suggested by the reviewer. We uniformly retain one or four frames, together with their corresponding 4D tokens, and compare them with the published full-frame setting:

Input settingCorrect / totalAccuracy
Single frame (middle sampled frame only)202 / 50040.4
Four frames (uniformly sampled, with 4D tokens)299 / 50059.8
Full frames (published setting)396 / 50079.2

Performance decreases from 79.2 with the full sequence to 59.8 with four frames and 40.4 with a single frame. Thus, perspective matching cannot be solved from sparse-frame appearance correspondence alone: even four uniformly sampled views remain 19.4 points below the full input, and the single-frame input is 38.8 points below. These results show that multi-frame information plays an important role in cross-view matching.

We again thank the reviewer for the detailed and thoughtful comments. The suggested controls have helped us provide more direct evidence for dynamic-geometry use and present the benchmark's capabilities more carefully.

Reviewer_JHawOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies 4D spatial intelligence for MLLMs, including a new benchmark (T4D-Bench), and a model (T4D) built on a frozen geometry expert and a video backbone. For training, TAP is introduced as a staged curriculum, which aligns the model pointmaps, tracks, and spatial questions. Empirically, T4D achieves strong gains over other baselines.

Critical strengths
  • The paper is clearly written and easy to follow.

  • This paper studies 4D spatial intelligence, which is an important but not well-studied area for MLLMs. Different from prior works that mainly proposed benchmarks, T4D not only provides a comprehensive benchmark (T4D-Bench), but also trains a T4D network that achieves strong performance on the benchmark.

  • T4D is shown effectiveness to improve spatial understanding ability on T4D-Bench.

Critical weaknesses
  • Novelty of the benchmark is overstated. OSI-Bench [1] is a very relevant benchmark to evaluate the spatial capabilities of MLLMs. It is a large open-world benchmark that captures high-quality egocentric videos with precise geometric ground truth. Most scenes contain dynamic objects, such as pedestrians and cars. Therefore, it is hard to argue that 4D Spatial Intelligence is a "void" area. The authors should rephrase the introduction, cite OSI-Bench [1], and clarify the difference between T4D-Bench and OSI-Bench.

  • The performance of T4D is only tested on T4D-Bench, whose data distribution is similar. It would be helpful to add the performance comparison on other spatial intelligence benchmarks.

  • T4D network relies on an external geometry expert during both training and inference. Although this brings better performance than directly finetuning VLMs, it may not scale to large VLMs or downstream applications.

[1] Wu et al. From Indoor to Open World: Revealing the Spatial Reasoning Gap in MLLMs. CVPR 2026.

Questions for authors

About the leading performance of GPT-6 Astra, would it be possible that it has access to these open-sourced datasets during training already, or other 3D data?

1 comment
AuthorsOpenReview ↗

Response to Reviewer JHaw

We thank the reviewer for the careful evaluation. We appreciate the comments on the positioning of related work, external evaluation, and the use of an external geometry expert. In response, we have narrowed our claims in the paper and added clarifications.

W1: Relationship between T4D-Bench and OSI-Bench

We agree that OSI-Bench has made an important early contribution to evaluating spatial reasoning in MLLMs on open-world videos, including metric 3D annotations and kinematics questions. We have revised the Introduction and Related Work to avoid presenting 4D spatial intelligence as an almost completely unexplored area, and added a citation to OSI-Bench.

T4D-Bench aims to evaluate dynamic spatial reasoning over a more systematic range of tasks. Through a top-down design, we organize 4D spatial intelligence into six dimensions and build a unified benchmark with 10 tasks and 3,250 expert-verified questions. In particular, T4D-Bench includes paired evaluations across first- and third-person videos to test whether models can match the same spatial facts across viewpoints. We have clarified the complementary relationship between the two benchmarks in the paper. We thank the reviewer for helping us define our contribution more accurately.

W2: Lack of evaluation on external benchmarks

We have evaluated T4D on the external benchmark VSI-Bench and report the complete results in Appendix Table 6. Because the paper primarily focuses on constructing and analyzing T4D-Bench, we placed the detailed third-party benchmark results in the appendix and added a reference to that table in the main experimental section.

The complete VSI-Bench results are as follows:

ModelObj. CountAbs. Dist.Obj. SizeRoom SizeRel. Dist.Rel. Dir.Route PlanAppr. OrderAvg.
Spatial-MLLM-4B65.334.863.145.141.346.233.546.347.0
VG-LLM-4B66.037.855.259.244.645.633.536.447.3
VLM-3R-7B70.249.469.267.165.480.545.440.160.9
T4D (4B)71.853.375.969.970.682.945.981.469.0

T4D obtains an Avg. of 69.0, higher than VLM-3R (60.9), VG-LLM (47.3), and Spatial-MLLM (47.0). These results provide complementary evidence of spatial understanding across benchmarks. However, VSI-Bench primarily evaluates static egocentric spatial understanding, so these results should not be interpreted as establishing cross-dataset generalization for dynamic 4D reasoning. The baseline scores are taken from published results and were not rerun under our evaluation setup.

W3: Reliance on an external frozen geometry expert

We agree that using the frozen SpatialTrackerV2 adds inference overhead and means that T4D remains slower overall than a native MLLM without a geometry expert. We use a sparser set of feature-extraction points to balance runtime cost and geometric coverage. This reduces the amount of feature extraction, although the overall system remains slower than a native MLLM. At the same time, a mature geometry expert provides explicit 3D/4D trajectory information, giving the MLLM geometric evidence beyond RGB.

In the comparison with models trained on the same data, T4D obtains an Overall score of 61.0, compared with 46.5 for Qwen3-VL-4B and 49.1 for the best same-data spatial specialist. This shows that the current system design brings practical gains on T4D-Bench. However, we do not attribute the gain to any single module alone, nor do we claim that the external expert has no computational cost or risk of error propagation. Using a frozen expert also provides a modular upgrade path. If a replacement expert uses a different feature interface or distribution, the projection or fusion modules may still need to be adapted.

Question: Did GPT-6 Astra use public benchmarks or other 3D data?

We thank the reviewer for raising this important question. GPT-6 Astra has not disclosed its training data, so we cannot determine whether it was exposed to relevant public benchmarks, source videos, or other 3D-annotated data. We therefore cannot rule out the potential influence of training-data exposure on its performance.

The specific questions and answers in T4D-Bench were newly constructed for this work and had not been publicly released. However, the source videos and associated 3D data come from multiple public datasets, and their use in proprietary-model training is unknown. It is also not possible to reliably infer a black-box model’s training-data composition from its output behavior. We therefore treat GPT-6 Astra’s results as reference performance under the same evaluation protocol, rather than as evidence of cross-dataset generalization under conditions where data exposure can be excluded.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.