- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper, T4D: Comprehending the Dynamic World with MLLMs (under review at ICLR 2027), addresses the limitation that existing multimodal large‑language models excel at static 3D spatial reasoning but struggle with 4D space‑time reasoning for dynamic real‑world scenes. It first establishes a six‑dimensional taxonomy for dynamic spatial intelligence and introduces T4D‑BENCH, a benchmark with 10 tasks and 3,250 expert‑validated QA pairs covering all six dimensions, alongside an automated pipeline generating around 600K training QA pairs from real‑world indoor and outdoor videos. The authors design the T4D network, which fuses explicit instance‑level 3D object trajectories with visual features via Position‑Addressed 4D Fusion (PAF), and propose Trajectory‑Aligned Training (TAP), a multi‑stage training recipe to prevent the model from taking 2D‑appearance shortcuts by aligning visual tokens and trajectory tokens. Evaluations show that the 4B‑parameter T4D model achieves state‑of‑the‑art performance on T4D‑BENCH among comparable‑scale open‑source MLLMs, though it still lags behind top‑tier proprietary models.
Critical strengths
-
The paper builds a 6-dimensional classification framework for dynamic spatial intelligence from a dynamic perspective, points out the gaps in capabilities covered by the existing 4D benchmarks, and clarifies what abilities need to be evaluated for 4D spatial reasoning.
-
The benchmark mentioned in the paper fully covers six major dynamic spatial dimensions and includes 3,250 QA entries manually verified by experts. It features both multiple-choice questions and open-ended physical value answers (like distance and speed), not just qualitative judgments.
-
A lot of ablation experiments were done, confirming all the effects of the proposed method.
Critical weaknesses
-
The model's performance heavily depends on the frozen SpatialTrackerV2 trajectory extractor. If the trajectory estimation is wrong, the error gets passed directly to the MLLM; once the trajectory module fails, the whole 4D inference is greatly affected, and there's no end-to-end joint optimization.
-
The method proposed in the paper requires running 3D trajectory extraction in advance, which is more costly than regular video MLLM inference; during evaluation, it's limited to a maximum of 128 frames, so 4D inference on very long sequence videos hasn't been fully tested.
-
Although T4D is SOTA among open-source models, its overall metrics are far below GPT‑6 Astra, being outperformed in all 10 tasks, which shows that dynamic 4D space reasoning still has a lot of room for improvement. Of course, this isn’t really the main weaknesses of the paper.
Questions for authors
This issue mainly affected my impression of this piece of work: Unfortunately, all key empirical findings are evaluated exclusively against the authors’ proposed T4D‑BENCH. Additional evaluations on out‑of‑distribution external benchmarks would be desirable to better verify the cross‑dataset generalisation capacity of the T4D method. Furthermore, most compared baseline models are trained on out‑of‑distribution data disjoint from T4D, which casts doubt on whether the reported performance improvements represent genuine advances in dynamic spatial reasoning or stem from training‑test distribution mismatches.
4 comments
Response to Reviewer KWkm (2/2)
W3: Gap relative to GPT-6 Astra
We thank the reviewer for the balanced comment and agree that GPT-6 Astra (78.0) substantially outperforms T4D (61.0), showing room for progress. T4D uses a 4B backbone and exceeds GPT-5.6 Sol (59.3) and Gemini 3.8 Flash (58.4). We do not speculate about proprietary models' parameter counts or training data; our claim is not that T4D surpasses them.
Question: External benchmarks and training-data distribution
We thank the reviewer for this important question, and we address it with two independent lines of evidence.
First, on external benchmarks: T4D obtains 69.0 Avg. on VSI-Bench, above VLM-3R (60.9), VG-LLM (47.3), and Spatial-MLLM (47.0); please see Appendix Table 6 for the full results.
| Model | Obj. Count | Abs. Dist. | Obj. Size | Room Size | Rel. Dist. | Rel. Dir. | Route Plan | Appr. Order | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Spatial-MLLM-4B | 65.3 | 34.8 | 63.1 | 45.1 | 41.3 | 46.2 | 33.5 | 46.3 | 47.0 |
| VG-LLM-4B | 66.0 | 37.8 | 55.2 | 59.2 | 44.6 | 45.6 | 33.5 | 36.4 | 47.3 |
| VLM-3R-7B | 70.2 | 49.4 | 69.2 | 67.1 | 65.4 | 80.5 | 45.4 | 40.1 | 60.9 |
| T4D (4B) | 71.8 | 53.3 | 75.9 | 69.9 | 70.6 | 82.9 | 45.9 | 81.4 | 69.0 |
VSI-Bench evaluates static spatial understanding, so we treat it as complementary cross-benchmark evidence, not direct proof of dynamic 4D generalization. Baseline scores are from published results and were not rerun by us.
Second, on the training-distribution concern: for a controlled comparison, Qwen3-VL-4B scores 46.5, the best same-data spatial specialist 49.1, and T4D 61.0—a gap that training–test distribution mismatch alone cannot explain. We cannot verify whether other models' pretraining data overlaps with T4D-Bench and make no assumption about such overlap.
Thank you for the detailed responses. While most of my concerns have been addressed, I still see some limitations in the practical value of this work. I lean to borderline, but will finalize my assessment during the AC-reviewer discussion.
We are glad that most of your concerns have been addressed, and we hope the additional results and revisions will be helpful as you finalize your assessment during the AC discussion. We remain happy to answer any further questions.
Response to Reviewer KWkm (1/2)
We thank the reviewer for the thoughtful review and for recognizing the six-dimension taxonomy, the expert-verified benchmark, and the ablation studies. We address each point below.
W1: Dependence on the frozen SpatialTrackerV2
We thank the reviewer for raising this important point. We acknowledge that a frozen tracker may pass errors downstream and is not jointly optimized with the MLLM. It provides explicit 3D trajectories beyond RGB and remains modular. On the same test set, ADEs are 29.73 m (Qwen3-VL-4B), 10.45 m (T4D), and 8.36 m (SpatialTrackerV2). Qualitatively, the backbone's decoded tracks tend to collapse toward the image center, as illustrated in the revised Figure 8. T4D scores 61.0 versus 46.5 for the same-data Qwen3-VL-4B on T4D-Bench, showing a clear system benefit without ruling out error propagation. Freezing the expert keeps the 4D evidence fixed, which makes the gains in Table 2 attributable to the training recipe. Replacing the tracker requires adapting only the lightweight projection/fusion modules, not the MLLM itself.
W2: Precomputation cost and the 128-frame limit
We appreciate the reviewer pointing this out. External experts add computation, an overhead inherent to pipelines that use an external geometry expert. During training, we pre-extract and cache expert features to avoid repeated tracker runs. At inference, the expert still adds cost, and T4D is slower than a native MLLM. Sparser feature-extraction points balance runtime and geometric coverage. All Table 1 models use 1 FPS and at most 128 frames, ensuring a consistent comparison; this is also in line with frame-sampled evaluation protocols used in spatial video benchmarks such as VSI-Bench and STI-Bench. This does not establish long-video ability: T4D-Bench targets short-clip geometry, kinematics, and cross-view reasoning, and we make no long-video claim.