acceptodds
Under review as a conference paper at ICLR 2027

4DThinker: Thinking with 4D Imagery for Dynamic Spatial Understanding

11 views

Abstract

Dynamic spatial reasoning from monocular video is essential for bridging visual intelligence and the physical world, yet remains challenging for vision-language models (VLMs). Prior approaches either verbalize spatial-temporal reasoning entirely as text, which is inherently verbose and imprecise for complex dynamics, or rely on external geometric modules that increase inference complexity without fostering intrinsic model capability. In this paper, we present 4DThinker, the first framework that enables VLMs to "think with 4D" through dynamic latent mental imagery, i.e., internally simulating how scenes evolve within the continuous hidden space. Specifically, we first introduce a scalable, annotation-free data generation pipeline that synthesizes 4D reasoning data from raw videos. We then propose Dynamic-Imagery Fine-Tuning (DIFT), which jointly supervises textual tokens and 4D latents to ground the model in dynamic visual semantics. Building on this, 4D Reinforcement Learning (4DRL) further tackles complex reasoning tasks via outcome-based rewards, restricting policy gradients to text tokens to ensure stable optimization. Extensive experiments across multiple dynamic spatial reasoning benchmarks demonstrate that 4DThinker consistently outperforms strong baselines and offers a new perspective toward 4D reasoning in VLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent Reasoning, Spatial Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9mPEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes 4DThinker, a framework for dynamic spatial reasoning from monocular video. It introduces a scalable data pipeline for synthesizing 4D reasoning data from raw videos. Dynamic-Imagery Fine-Tuning (DIFT) combines text-token cross-entropy with a cosine-similarity objective that aligns predicted hidden states with pooled visual embeddings of the highlighted frames. 4D Reinforcement Learning (4DRL) improves the reasoning capability using correctness and format rewards. The experiments show that Qwen3-VL-32B/InternVL3.5-38B based 4DThinker outperforming the compared baselines on DSR-Bench and Dyn-Bench.

Critical strengths
  1. Consistent improvements across multiple benchmarks and backbones. The experiments demonstrate performance gains on DSR-Bench and Dyn-Bench using Qwen2.5-VL-3B/7B, Qwen3-VL-8B/32B, and InternVL3.5-8B/38B.

  2. Informative ablation studies. The ablation studies through comparisons across training strategies, loss terms, and reward components, which support the contributions of the major components of 4DThinker.

Critical weaknesses
  1. Limited methodological novelty. DIFT closely resembles the latent visual grounding approach introduced in Machine Mental Imagery. Machine Mental Imagery also explicitly optimizes only text-token probabilities while allowing gradients to propagate through latent states. LVR addresses the absence of discrete token distributions at continuous latent positions through a GRPO variant that computes the policy-gradient objective solely over text-generation tokens, which is similar with 4DRL. The paper should clarify the substantive differences from these methods and explain what makes 4DRL specifically suited to 4D reasoning.

  2. The comparison with DSR Suite-Model does not control in backbone size. Although Table 1 shows that Qwen3-VL-32B based 4DThinker outperforms DSR Suite-Model (62.0 vs 58.9), the latter uses the substantially smaller Qwen2.5-VL-7B backbone. When using the same Qwen2.5-VL-7B backbone, 4DThinker achieves a score of 47.1, substantially below DSR Suite-Model’s 58.9. Therefore, the results do not establish a performance advantage of the proposed method over the baselines.

Questions for authors

Please see the weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9mPE [1/2]

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We address each point below and will revise the manuscript accordingly.

W1: Substantive Differences from Machine Mental Imagery (Mirage) and LVR, and Why 4DRL Suits 4D Reasoning

Thank you for highlighting these two representative prior works. We cite both Mirage (Yang et al., 2025c) and LVR (Li et al., 2025a) in Sec. 2.1, and will add a dedicated comparison paragraph in the revised manuscript clarifying our contributions across five dimensions: problem space, supervision construction, annotation dependency, temporal structure, and RL dynamics:

DimensionMachine Mental Imagery (Mirage)LVR4DThinker (Ours)
Reasoning Target & SpaceStatic single-image spatial reasoningStatic single-image fine-grained perception / countingMonocular video 4D dynamic spatial reasoning (coupling camera ego-motion and 3D object motion)
Physical Meaning of LatentsStatic auxiliary-view / reference-image featuresLocal ROI patch refocusing within the input imageCross-timestamp dynamic trajectory evolution (time-evolving target state sequences supervised beyond raw input frames)
Supervision AcquisitionRequires manually paired auxiliary reference imagesRequires manually annotated bounding boxes (ROIs)Fully automated, annotation-free pipeline (decomposing camera and object motion trajectories directly from raw videos)
Temporal Structure of LatentsSingle static latent block; no temporal notionSingle static latent block; no cross-frame recursionMulti-segment latent chain strictly ordered by timestamp; shuffling temporal order drops accuracy from 31.1 to 23.9 (below base 24.6, Table 6(a))
RL Stage & Latent InteractionNo RL stage designed for multi-step temporal latent trajectoriesSkips text loss at latent positions over single-segment static box featuresStability protection for multi-step recursive 4D trajectories, paired with structural think–imagery–answer rewards for compound-motion exploration

Beyond applying Mirage or LVR to video inputs, 4DThinker resolves two technical challenges that static latent reasoning does not address:

  1. From static local inspection to cross-timestamp 4D state simulation (DIFT): LVR latents zoom into local regions of an already observed single image, while Mirage relies on manually curated static auxiliary views. Neither represents an object accelerating or turning while the camera translates and rotates. In dynamic video reasoning, collapsing supervision to a single static timestamp (Last-frame DIFT in Table 6(a)) yields only 26.7 accuracy, matching text-only CoT SFT (26.8). Introducing temporally ordered, mask-grounded sequences enables the model to relate the camera reference frame (static landmark displacement) to object motion trajectories in latent space, raising accuracy to 31.1. Under the shared Qwen2.5-VL-7B backbone (Table 12), 4DThinker reaches 47.1 / 65.9 on DSR-Bench / Dyn-Bench, outperforming the static latent-reasoning baseline LVR (33.5 / 58.4) by a wide margin.
  2. Why masking policy gradients at latent positions is critical for 4D dynamic reasoning (4DRL): As you noted, computing policy gradients only on text tokens (since continuous latent positions lack a discrete vocabulary distribution) shares the same formal principle as LVR. In 4DThinker, however, latent states are not single-step static feature extractions. They form a directed temporal trajectory generated recursively across multiple timestamps via Eq. (10), and 4DRL must compose camera and object motion reasoning autonomously on compound-motion data (DSR-Train) without intermediate visual annotations. Without masking discrete losses at latent positions (vanilla GRPO in Table 6(c)), high-variance token-level gradients backpropagate along the recursive chain across multi-step latent states. Cosine similarity to ground-truth dynamic overlays then drops from 0.63 (DIFT initialization) to 0.51, weakening cross-video causal grounding. By contrast, 4DRL maintains cosine similarity at 0.61 and outperforms vanilla GRPO on DSR-Bench (34.2 vs. 32.5, and combining RfmtR_{\text{fmt}} to verify the temporally interleaved think–imagery–answer structure also improves over 33.4 without the format reward).

In summary, Mirage and LVR target localized static-image representations with manual annotations, whereas 4DThinker enables annotation-free cross-timestamp simulation of coupled camera-object motion in monocular video. Under the same 7B backbone, 4DThinker improves over LVR by +13.6 / +7.5 points, and our ablations and cosine tracking verify that multi-step temporal ordering and recursive gradient isolation are both necessary in 4D settings.

AuthorsOpenReview ↗

Response to Reviewer 9mPE [2/2]

W2: Controlled Backbone-Scale (7B) and Matched-Dataset Comparison with DSR Suite-Model

Thank you for pointing out this fairness issue. In Sec. 4.1 of the revised manuscript, we will revise the cross-backbone phrasing and combine our additional evaluations from Reviewer dFA5 (W1/Q1) and Reviewer gaQe (W1, W2) to present controlled comparisons under the same Qwen2.5-VL-7B backbone across all five dynamic spatial reasoning benchmarks and their overall mean (Avg.):

Controlled GroupModelBackboneDSR Training DataExternal Geometric Modules at InferenceDSR-BenchDyn-BenchVLM4DDSI-BenchSTI-BenchAvg.
Same 7BDSR Suite-ModelQwen2.5-VL-7BOfficial full 50KGSM (π3\pi^3 + SAM2 + Orient)58.960.848.338.635.248.4
Same 7B4DThinker (main paper)Qwen2.5-VL-7B37K subsetNone47.165.954.246.844.851.8
Same 7B4DThinker (full DSR-Train 50K)Qwen2.5-VL-7BOfficial full 50KNone60.866.855.147.645.755.2

This controlled comparison clarifies the performance of 4DThinker at matched backbone scale:

Under the shared Qwen2.5-VL-7B backbone, 4DThinker already leads in five-benchmark average and surpasses DSR Suite-Model on all five benchmarks once aligned to the full 50K training set: The 7B gap on DSR-Bench in the main paper (47.1 vs. 58.9) stems primarily from differences in training data volume and external modules. DSR Suite-Model trains on the full DSR-Train 50K set and attaches three external vision/3D models (π3\pi^3, Grounded-SAM2, and Orient Anything) at inference to inject explicit geometric tokens. By contrast, 4DThinker-7B in the main paper uses a 37K subset with sparse outcome rewards and zero external modules at inference. Even under the main-paper setting, 4DThinker-7B outperforms DSR Suite-Model by 5.1–9.6 points across all four cross-distribution benchmarks (Dyn-Bench, VLM4D, DSI-Bench, and STI-Bench), reaching a five-benchmark average of 51.8 (above DSR Suite-Model at 48.4). When aligned to the exact same official full DSR-Train 50K set (as in Reviewer dFA5 W1/Q1, using DSR-Train 50K in both stages), module-free 4DThinker-7B reaches 60.8 on DSR-Bench (exceeding DSR Suite-Model at 58.9) and raises the five-benchmark average to 55.2 (+6.8 points higher).

In summary, under a controlled comparison with the same 7B backbone + the same 50K training set and zero external geometric modules, 4DThinker outperforms DSR Suite-Model both on DSR-Bench and in average accuracy across all five dynamic spatial benchmarks (7B: 55.2 vs. 48.4). We will add these controlled comparisons to the revised manuscript, and thank you again for this constructive feedback.

AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer 9mPE

Dear Reviewer 9mPE,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

Reviewer_gaQeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes 4DThinker, a framework for improving dynamic spatial reasoning in VLMs through interleaved textual CoT and latent visual representations. The method introduces an automatic data generation pipeline that constructs motion-centric QA pairs and imagery-based reasoning traces from videos. It then adopts a two-stage training strategy: Dynamic-Imagery Fine-Tuning (DIFT) jointly supervises text and visual latents, while 4D Reinforcement Learning (4DRL) further improves reasoning through outcome-based rewards. Experiments across multiple VLM backbones and benchmarks demonstrate consistent improvements in dynamic spatial understanding.

Critical strengths
  1. The paper extends latent visual reasoning to dynamic spatial understanding by interleaving textual CoT with temporally ordered latent visual representations.

  2. The method achieves substantial improvements across different VLM backbones and dynamic spatial reasoning benchmarks.

  3. The paper provides relatively thorough ablation studies. Table 6 demonstrates the importance of temporal ordering and the benefits of latent supervision over directly providing visual overlays. The causal interventions in Table 9 further suggest that latent states contribute to answer prediction rather than merely serving as auxiliary training signals.

  4. The proposed pipeline constructs motion-centric QA pairs and imagery-based reasoning traces from videos without requiring manual annotations.

Critical weaknesses

major:

  1. Lack of General Capability Evaluation. The paper demonstrates substantial improvements in dynamic spatial reasoning, but does not evaluate whether the specialized DIFT and 4DRL training stages affect the base VLM's general-purpose capabilities. Additional evaluations on general multimodal and video understanding benchmarks would help determine whether the improvements in dynamic spatial reasoning are achieved while preserving the broader capabilities of the original models.

  2. Missing Strong Baseline and Incomplete SOTA Comparison. The paper claims to surpass the previous SOTA on DSR-Bench (62.0 vs. 58.9). However, GeoSR (ECCV 2026), which reports a higher accuracy of 66.1 using Qwen2.5-VL-7B, is neither cited nor compared against. Although GeoSR relies on external geometric features, the paper already includes several geometry-augmented baselines, such as DSR Suite and VG-LLM. The omission of this relevant and competitive method weakens the experimental comparison and calls for a more carefully qualified SOTA claim.

  3. Insufficient Clarity of the 4DRL Formulation. In Eq. (13), the importance ratio is computed against the frozen reference policy rather than the rollout policy typically used in GRPO. The paper does not clarify whether this is a notation error or an intentional modification. Moreover, while Table 6(c) compares 4DRL with vanilla GRPO, it remains unclear how vanilla GRPO handles the continuous latent positions and how its implementation differs from 4DRL. These ambiguities make it difficult to fully assess and reproduce the proposed RL training strategy.

  • Minor
  1. Evaluation Details. Table 11 reports an option-free evaluation on 100 questions using binary correctness judgments. However, some reported accuracies (e.g., 22.5% and 38.5%) are not consistent with a single evaluation of 100 binary outcomes. Were these results averaged over multiple runs, or was a different aggregation procedure used?

  2. Reference Issues.

    • Zhou et al. (2025a) and Zhou et al. (2025b) appear to be duplicate references to the same DSR Suite paper. Furthermore, VideoLoom is incorrectly cited as Zhou et al. (2025b) in Sec. 2.2.

    • Qwen3-VL-8B/32B is cited using the Qwen3 Technical Report rather than the Qwen3-VL Technical Report.

  3. Inconsistent CoT Validation Descriptions. Sec. 3.1 states that the automated CoT validator checks placeholder count, chronological consistency, and answer isolation, whereas Appendix I lists format completeness, placeholder count, and answer validity. The authors should clarify the actual validation procedure, particularly how chronological consistency is verified.

Questions for authors
  1. Have the authors evaluated whether DIFT and 4DRL preserve the base VLM's general multimodal and video understanding capabilities?

  2. Could the authors clarify why GeoSR is neither cited nor included in the comparisons, despite the inclusion of other geometry-augmented baselines?

  3. In Eq. (13), is the use of the frozen reference policy instead of the rollout policy intentional or a notation error? Additionally, how is vanilla GRPO implemented for continuous latent positions in Table 6(c)?

  4. Table 9 evaluates latent interventions only on the DIFT-only model. Would similar interventions on the final DIFT+4DRL model lead to comparable performance drops?

7 comments
AuthorsOpenReview ↗

Response to Reviewer gaQe [1/4]

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We address each point below and will revise the manuscript accordingly.

W1 / Q1: Preservation of General Capabilities After DIFT and 4DRL Training

To test whether specialized 4D training affects the base VLM's general image and video understanding capabilities, we evaluate the original Qwen2.5-VL-7B base model, +DIFT, and the final +DIFT+4DRL model across four standard image benchmarks and three general video understanding benchmarks under their official evaluation protocols:

General BenchmarkModality / DomainBase+DIFT+DIFT+4DRLΔ\Delta vs. Base
MMBench-V1.1-ENGeneral Image82.682.783.0+0.4
MMStarImage Reasoning63.963.964.2+0.3
MMMU (val)Multidisciplinary Multimodal58.658.458.7+0.1
MathVista (testmini)Visual Math68.268.168.4+0.2
Video-MME (w/o sub.)General Short/Long Video65.165.365.6+0.5
MVBenchVideo Motion & Temporal69.669.970.2+0.6
TempCompassFine-Grained Temporal71.772.272.8+1.1

The two-stage training preserves the base model's general multimodal capabilities: (1) Across all four image benchmarks, variations stay within 0.1–0.4 points, leaving general visual-language performance unchanged. (2) Across all three general video benchmarks, accuracy improves consistently by +0.5 to +1.1 points rather than dropping, including a +1.1 gain on TempCompass, which is sensitive to motion direction and temporal transitions. Freezing the visual encoder prevents visual feature drift, while spatiotemporal tracking in latent space transfers positively to general temporal perception in video.

AuthorsOpenReview ↗

Response to Reviewer gaQe [2/4]

W2 / Q2: Comparison with GeoSR and Qualification of SOTA Claims

We will cite and include GeoSR in our comparisons in the revised manuscript, and tighten our SOTA claims throughout the paper as suggested. Using the shared Qwen2.5-VL-7B backbone, we summarize performance across all five dynamic spatial reasoning benchmarks in our paper (DSR-Bench, Dyn-Bench, VLM4D, DSI-Bench, and STI-Bench) together with their overall mean (Avg.):

ModelExternal Geometric Modules at InferenceDSR-BenchDyn-BenchVLM4DDSI-BenchSTI-BenchAvg.
Qwen2.5-VL-7BNone23.554.343.835.734.538.4
GeoSRπ3\pi^3 + Geometry-Guided Fusion (9.23B total params)66.161.649.439.836.650.7
4DThinker (main paper)None47.165.954.246.844.851.8
4DThinker (full DSR-Train 50K)None60.866.855.147.645.755.2

Together with the original GeoSR paper (Sec. 3.4 and Table 6), these results clarify the differences between the two methods and the advantages of 4DThinker in three respects:

  1. External geometric modules add parameter and latency overhead compared with a purely intrinsic architecture: At inference, GeoSR deploys an external frozen 3D geometric encoder (π3\pi^3 / VGGT) followed by three cross-attention query layers and a gated fusion module (Geometry-Guided Fusion). According to GeoSR Table 6, the dynamic branch alone increases total parameters to 9.23B, raises peak GPU memory to 18.95 GB, and adds 10.8% latency per forward pass (0.37s →\rightarrow 0.41s). By contrast, 4DThinker uses zero external modules and zero extra parameters at inference, retaining the lightweight deployment footprint and inference efficiency of the base VLM.
  2. Heavier training data scale and reliance on 3D pretraining priors: As reported in GeoSR Sec. 3.4, its spatial reasoning system builds separate high-cost data pipelines across scenarios. Its static branch relies on large-scale spatial datasets SPAR-7M and LLaVA-Hound (298K supervised samples in total), while its dynamic branch uses the full official DSR-Train (50K samples, vs. the 37K sparse outcome-reward subset in our main paper) alongside 3D priors from external 3D foundation models pretrained on large-scale 3D geometry datasets. In contrast, 4DThinker acquires general 4D spatiotemporal reasoning within a single model using an annotation-free video self-synthesis pipeline and a leaner training budget.
  3. Higher five-benchmark average and clear cross-distribution generalization gains: On the in-distribution fine-grained geometric template benchmark DSR-Bench, GeoSR reaches 66.1 with its explicit 3D geometric branch. Across all four remaining benchmarks that evaluate cross-scene dynamic spatial generalization (Dyn-Bench, VLM4D, DSI-Bench, and STI-Bench), module-free 4DThinker outperforms GeoSR by 4.3–8.2 points. Averaged over all five benchmarks, 4DThinker (DIFT+4DRL) in the main paper already reaches 51.8 with less DSR training data, exceeding GeoSR (50.7). When aligned to the same full DSR-Train 50K set (see the experiment for Reviewer dFA5 W1/Q1, using DSR-Train 50K in both stages), the five-benchmark average rises further to 55.2 (+4.5 points over GeoSR).

Based on these comparisons, we will scope our SOTA claims in the abstract and experiment sections explicitly: Among model-intrinsic methods that do not rely on external 3D or geometric modules at inference, 4DThinker achieves the best performance on DSR-Bench / Dyn-Bench (reaching 47.1 / 65.9 at 7B, well ahead of intrinsic latent-reasoning baselines LVR at 33.5 / 58.4 and 3DThinker at 27.8 / 52.6, and reaching 62.0 / 75.4 at 32B). In overall average performance across all five dynamic spatial benchmarks, 4DThinker surpasses all existing methods, including geometry-augmented approaches such as GeoSR.

AuthorsOpenReview ↗

Response to Reviewer gaQe [3/4]

W3 / Q3: Correction to the Importance Ratio in Eq. (13) and Implementation Difference Between vanilla GRPO and 4DRL

Thank you for catching this detail. Writing πref\pi_{\text{ref}} in the denominator of Eq. (13) was a typographical error, not an intentional algorithmic change. In our supplementary code (built on TRL GRPOTrainer), the importance ratio is always computed relative to the rollout policy πold\pi_{\text{old}} that generated the current group of samples. The frozen DIFT reference policy πref\pi_{\text{ref}} is used solely in the per-step KL regularization term DKL(t)D_{\text{KL}}^{(t)}. The corrected importance ratio is: ρi,t=πθ(xt∣x<t)πold(xt∣x<t).\rho_{i,t} = \frac{\pi_\theta(x_t \mid x_{<t})}{\pi_{\text{old}}(x_t \mid x_{<t})}. Because the underlying training code executes with πold\pi_{\text{old}} throughout, none of the experimental results in the paper are affected by this typo. We will correct Eq. (13) in the revised manuscript and include pseudocode for 4DRL in the appendix to aid reproduction.

Regarding how vanilla GRPO and 4DRL differ in handling continuous latent positions in Table 6(c), both methods share the exact same rollout generation procedure: once <lat_s> is sampled, the model enters latent mode for K=4K=4 consecutive steps, feeds the previous step's last-layer hidden state ht−1h_{t-1} as the next input embedding per Eq. (10), records the placeholder <|latent_pad|> in the discrete sequence completion_ids, and emits <lat_e> to resume standard text sampling. The difference lies in the policy-gradient backward pass:

  • vanilla GRPO: Retains the full-sequence completion_mask (without excluding <lat_s>, <|latent_pad|>, or <lat_e>), computes discrete lm_head vocabulary log-probabilities and policy gradients across all positions including placeholders, and does not block gradients flowing through latent positions. Because latent positions do not perform vocabulary sampling at inference, applying discrete token-level score-function gradients at those positions introduces high-variance interference and degrades the continuous visual representations aligned during DIFT.
  • 4DRL: Keeps all latent blocks intact in the forward attention_mask (so subsequent text tokens condition on the 4D latents), while zeroing out <lat_s>, <|latent_pad|>, and <lat_e> in the loss mask completion_mask so that policy gradients and the KL penalty apply only to genuine text tokens Ttxt(i)T_{\text{txt}}^{(i)}.

Training dynamics directly reflect this design: group mean reward is 0.44 ± 0.09 under vanilla GRPO, compared with 0.51 ± 0.04 under 4DRL. After training, cosine similarity between generated latents and oracle overlays drops from 0.63 (DIFT initialization) to 0.51 under vanilla GRPO, whereas 4DRL preserves it at 0.61 and improves DSR-Bench accuracy from 32.5 to 34.2. We will clarify this implementation comparison in the revised main text, and our supplementary code provides the reproducible RL training implementation.

Q4: Causal Interventions on Latent Representations in the Final DIFT+4DRL Model

Following the exact causal intervention protocol of Table 9 (caching continuous latent blocks from an initial normal pass and modifying only the continuous latent states in a second forward pass while holding the video, question, interleaved text, and answer positions fixed), we evaluate all interventions on the final Qwen2.5-VL-3B (DIFT+4DRL) model:

Causal Intervention SettingDIFTDIFT+4DRL
Normal inference (no intervention)31.134.2
Temporal-block shuffle23.9 (−7.2)26.0 (−8.2)
Cross-video replacement (another video from the same subtask)21.8 (−9.3)24.1 (−10.1)
Preceding latents masked26.9 (−4.2)29.4 (−4.8)
Corpus-mean latents25.7 (−5.4)28.2 (−6.0)
Control: Equal text states masked30.4 (−0.7)33.5 (−0.7)

Rather than weakening, the performance drops under latent interventions on the final DIFT+4DRL model widen by 0.6–1.0 points relative to DIFT (temporal shuffling and cross-video replacement lower accuracy by −8.2 and −10.1 points, respectively). Masking an equal number of preceding text hidden states still shifts accuracy by only −0.7 points. 4DRL post-training does not push the model toward text-only shortcut reasoning. Instead, the final model continues to rely on the temporal ordering and scene-specific motion trajectories encoded in the 4D latents during decision-making. We will merge these results into Table 9 in the revised manuscript.

Reviewer_dFA5OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes 4DThinker, a framework that enables VLMs to perform dynamic spatial reasoning from monocular video by “thinking with 4D” latent mental imagery. The model interleaves textual CoT with continuous latent visual tokens that are supervised by mask-overlay embeddings of salient static/dynamic objects. The authors introduce an annotation-free (no human labels) data-generation pipeline from raw videos, Dynamic-Imagery Fine-Tuning (DIFT) with joint text cross-entropy and latent cosine-similarity losses, and 4D Reinforcement Learning (4DRL), a modified GRPO that restricts policy gradients to text tokens. Experiments on DSR-Bench and Dyn-Bench across multiple VLM backbones show consistent gains, supported by ablations, linear probes, and causal interventions on latent states.

Critical strengths
  1. The paper moves beyond static/2D/3D latent reasoning by supervising latent blocks with temporally ordered, mask-grounded overlays and interleaving them with CoT. This is a meaningful conceptual step for intrinsic dynamic spatial reasoning, especially relevant to autonomous driving and robotics, because inference does not require external geometric modules.

  2. The gains are large across multiple backbones and benchmarks (e.g., Qwen3-VL-32B on DSR-Bench from 28.0 to 62.0; Qwen2.5-VL-7B on Dyn-Bench from 54.3 to 65.9). The ablations distinguish DIFT from CoT SFT and 4DRL from vanilla GRPO, while the probe and intervention analyses provide nontrivial evidence that latent states encode and causally affect dynamic reasoning.

Critical weaknesses
  1. 4DRL is trained on DSR-Train and evaluated on DSR-Bench from the same DSR Suite family. The paper does not clearly establish video-level disjointness between DSR-Train and DSR-Bench, nor whether all compared baselines received comparable training/tuning data. This matters because the largest DSR-Bench gains could partly reflect same-distribution adaptation rather than purely intrinsic 4D reasoning. The Dyn-Bench and non-DSR results mitigate this concern, but a clarification is needed for the central claim.

  2. DIFT aligns latent positions with frozen visual embeddings of overlays, but at inference Eq. 10 feeds the preceding hidden state rather than an overlay embedding. The probe/intervention results are supportive, yet they are measured on the same DIFT checkpoint and largely on DSR-Bench. It would be important to show more directly that latent blocks remain grounded in object/camera motion without overlay inputs and do not merely propagate recursive hidden states. This is a fixable but material interpretability concern.

Questions for authors
  1. Are DSR-Train and DSR-Bench split at the video level with zero overlapping source videos? Please provide the split protocol and any overlap check. Also, were all baselines trained/tuned with comparable data budgets and access to DSR-Train?

  2. At inference, no mask overlay is provided, and Eq. 10 uses the previous hidden state as the latent input. What mechanism prevents latent blocks from drifting away from the intended dynamic visual semantics? Can you provide a quantitative analysis of latent stability/grounding under no-overlay inference?

  3. How sensitive are the results to errors from the data-generation pipeline (Gemini-3-pro labels, SAM3 masks, MegaSaM geometry)? The human audit is useful, but a breakdown of downstream accuracy versus mask/CoT noise would clarify scalability and robustness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer dFA5 [1/2]

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We address each point below and will revise the manuscript accordingly.

W1/Q1: Split Protocol Between DSR-Train and DSR-Bench, and Matched-Training Comparison

Both DSR-Train and DSR-Bench follow the official public protocol of DSR Suite[1]. According to Sec. 3.1 and the appendix of DSR Suite, both splits are sampled from the Koala-36M video pool and partitioned at the video level into 10,000 training videos and 575 evaluation videos. The training set is filtered from captions by DeepSeek-R1, whereas the evaluation set is curated directly from video content by Gemini-2.5-Pro and manually refined question by question. Further data statistics and distribution analyses are reported in the original paper. Training on DSR-Train and evaluating on DSR-Bench follows this established public protocol.

To test whether 4DThinker retains its advantage under the exact same training set, we train 4DThinker on the full DSR-Train 50K set used by DSR Suite-Model (using DSR-Train 50K in both stages via our data generation pipeline) on the shared Qwen2.5-VL-7B backbone:

MethodTraining DataExternal Geometric Modules (infer)DSR-BenchDyn-Bench
DSR Suite-ModelDSR-Train 50KGSM (π3\pi^3 + Grounded-SAM2 + Orient Anything)58.960.8
4DThinkerDSR-Train 50KNone60.866.8

Our experiments follow the official video-level train/evaluation split of DSR Suite. When training data and supervision are aligned to the official DSR-Train 50K set, 4DThinker without external geometric modules outperforms DSR Suite-Model on both DSR-Bench (60.8 vs. 58.9) and Dyn-Bench (66.8 vs. 60.8). These results show that the gains come from intrinsic 4D latent reasoning rather than a training distribution advantage.

W2/Q2: Preventing Semantic Drift in Latent Representations Without Mask-Overlay Inputs at Inference

Four constraints in our training and generation design keep the latent representations grounded in target visual semantics: (1) Direct alignment between training and inference inputs, i.e., LsimL_{\text{sim}} in Eq. (9) trains ht−1h_{t-1} to match the target frame's overlay visual embedding ztz_t, and ht−1h_{t-1} is the exact vector fed back during inference. (2) DIFT freezes the visual encoder fvisf_{\text{vis}} throughout training, keeping the target manifold fixed while cold-starting 4D visual semantics. (3) Each latent block is capped at a short horizon of K=4K=4 steps (terminated by <lat_e> at the step limit), and explicit textual reasoning re-anchors the context between blocks to prevent long-horizon recursive error accumulation. (4) RL rollout sampling in 4DRL runs directly in the no-overlay mode of Eq. (10), so trajectories that drift from motion semantics and produce wrong answers are penalized by outcome-based rewards.

To directly quantify latent quality without any overlay inputs, we run free-form inference on Qwen2.5-VL-3B to generate latent blocks and extract oracle overlay embeddings from the frozen visual encoder for evaluation:

No-Overlay Inference Metric4D LatentsMatched Text StatesVideo-Shuffled Control
Cosine similarity to oracle overlays (DSR-Bench, 500 Qs)0.630.320.26
Cosine similarity to oracle overlays (Dyn-Bench, 500 Qs)0.600.310.25
100-way hard-negative retrieval under no-overlay generation (R@1 / R@5)37.4 / 63.18.4 / 21.71.2 / 5.3
Cross-video linear-probe balanced accuracy on final model (DIFT+4DRL)61.235.117.5

Across recursive positions 1 through 4 within a single latent block, mean cosine similarity to the target embedding is 0.65, 0.64, 0.63, and 0.61 (a mild 0.04 decrease across 4 autoregressive steps, with no divergence). Under free-form no-overlay generation, 100-way retrieval reaches 37.4 / 63.1, closely matching Table 8 (38.6 / 64.2) and remaining well above same-layer text hidden states (8.4 / 21.7). Regarding whether our analyses hold beyond the DIFT stage and DSR-Bench, we also run the motion-attribute linear probe from Table 7 on the final DIFT+4DRL model using held-out SpatialVID videos excluded from training. Even though 4DRL no longer optimizes Lsim\mathcal{L}_{\text{sim}} explicitly, the final model's latents achieve 61.2% macro balanced accuracy (only 2.5 points below DIFT at 63.7%), outperforming matched text hidden states (35.1%) by a wide margin.

Under standard inference with no mask-overlay inputs, the latent blocks retain visual semantics aligned with target object and camera motion after both within-block recursion and 4DRL post-training, rather than degenerating into generic recursive hidden states.

[1] Learning to Reason in 4D: Dynamic Spatial Understanding for Vision Language Models. CVPR 2026. https://arxiv.org/abs/2512.20557

AuthorsOpenReview ↗

Response to Reviewer dFA5 [2/2]

Q3: Sensitivity of Downstream Performance to Errors in the Automated Data Pipeline

Thank you for this suggestion. In our independent human audit (Appendix B, Table 10), the current pipeline achieves 92% accuracy on mask identity and temporal consistency, 98% on motion-label correctness, and 96% on CoT factual validity. To measure downstream sensitivity to potential upstream noise from SAM3, MegaSaM, and Gemini-3-Pro, we inject controlled degradations of varying severity into the 38K DIFT training set, retrain Qwen2.5-VL-3B, and evaluate on DSR-Bench:

Data Degradation or Filtering Setting (Qwen2.5-VL-3B)DSR-BenchΔ\Delta vs. Standard DIFT
Full standard automated pipeline (DIFT)31.1—
SAM3 mask degradation (morphological dilation/erosion & frame shift; mean IoU reduced to 0.85 / 0.70 / 0.55)30.9 / 30.1 / 29.2−0.2 / −1.0 / −1.9
CoT noise injection (replacing 10% / 25% / 50% of samples with unvalidated first-pass raw outputs)30.8 / 30.0 / 28.9−0.3 / −1.1 / −2.2
MegaSaM camera-motion label noise (randomly flipping motion classes in 10% / 25% of samples)30.6 / 29.6−0.5 / −1.5
Retraining DIFT on the high-confidence subset passing all three human-audit criteria (~92%)31.4+0.3
Continuing 4DRL from the "25% CoT noise" DIFT checkpoint33.2+3.2 over this init

Three patterns emerge from this table: (1) Within the empirical noise range of our pipeline (2%–8%, below the 10% tier), downstream accuracy fluctuates by at most 0.3 points. Filtering out ~8% borderline samples to retain only the high-confidence subset raises accuracy slightly to 31.4, indicating that the existing filtering rules already produce clean supervision. (2) Even under severe synthetic degradations well above actual noise levels (e.g., reducing mask IoU to 0.55 or replacing half of the CoTs with unvalidated outputs), accuracy degrades smoothly without collapse (29.2 and 28.9). Both scores remain above the ungrounded-mask ablation (28.0, Table 6(a)) and text-only CoT SFT (26.8, Table 3). (3) Applying 4DRL to the DIFT checkpoint trained with 25% CoT noise still yields a +3.2 net gain (30.0 →\rightarrow 33.2). This matches the gain from the clean checkpoint (+3.1, 31.1 →\rightarrow 34.2) and shows that outcome-rewarded 4DRL corrects residual trajectory noise from the supervised stage.

Downstream reasoning performance stays stable within the actual error range of our automated pipeline, degrades gracefully under amplified synthetic noise, and absorbs residual noise further through 4DRL. These results support scaling 4D training data without manual annotation. We will add this experiment to the revised manuscript.

AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer dFA5

Dear Reviewer dFA5,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.