- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GeoThinker‑F, an efficient multimodal large‑language‑model framework for spatial intelligence. Instead of running an external 3D geometry encoder during inference, GeoThinker‑F learns to endogenously predict current‑frame geometric representations and future‑frame semantics from internal model hidden states via lightweight Latent Frame Prediction (LFP) heads. It adopts a two‑stage training pipeline: first pre‑training LFP heads supervised by an offline 3D expert, then self‑driven tuning with the proposed Spatial‑Grounded Temporal (SGT) Fusion that interleaves geometry and temporal features across transformer layers. Evaluated on multiple spatial‑reasoning benchmarks, GeoThinker‑F achieves competitive spatial‑understanding performance while delivering up to 3.6× inference speedup. Though it suffers from prediction drift on extremely long sequences and subpar metric distance estimation compared with systems using real‑time 3D encoders, it offers a favourable accuracy‑efficiency trade‑off, especially for latency‑sensitive embodied and interactive applications.
Critical strengths
-
The core idea of the paper is that existing VLMs require a 3D encoder to handle spatiotemporal cognition tasks, which takes a lot of time. The paper removes the external 3D geometry encoder during the inference stage and predicts geometry and temporal representations inside the model's latent space, balancing spatial reasoning accuracy and inference efficiency.
-
Standing out on spatial benchmarks like MindCube and ViewSpatial, Qwen3‑VL‑8B scored an average of 69.29 across 6 spatial benchmarks, ranking 2nd on the EASI‑8 list; under controlled conditions with the same training data, its performance can match the GeoThinker baseline with an external encoder.
-
The paper did a lot of ablation studies and feature visualizations, which is great, making the characteristics of the proposed method clearer.
Critical weaknesses
-
The paper tries to use a smaller encoder to estimate complex spatiotemporal changes in real-world scenarios, which isn't realistic. I think this somewhat weakens the paper's transferability.
-
It relies on a two-stage training process. Stage‑1 needs offline 3D experts for supervision, has a complex multi-objective loss function, and the training and tuning are more complicated than regular fine-tuning.
-
It's hard to capture the fine, fleeting details of fast, non-rigid deformations (like fluttering fabric or small animal fur); the internally predicted features are rougher compared to the real 3D encoder outputs.
Questions for authors
This issue mainly affected my impression of this piece of work:
While GeoThinker‑F frames heavy‑runtime external 3D encoders as a widespread bottleneck for spatial‑aware MLLMs, its core motivating assumption risks over‑generalization. A growing line of work, such as MLLM‑4D which produces explicit textual geometric outputs natively and Qwen3.5 that builds hybrid spatio‑temporal feature embeddings, demonstrates that competitive spatial reasoning can be realized without auxiliary 3D encoders at either training or inference time. These models can be fully optimized under a unified SFT+GRPO pipeline, avoiding both runtime geometry overhead and the two‑stage, teacher‑dependent training paradigm adopted in GeoThinker‑F. This means the pain point the paper addresses is not a universal necessity for spatial MLLMs. This weakens the paper’s core motivation: the need to replace external 3D encoders is only critical for a specific subclass of prior dual‑encoder architectures like GeoThinker, rather than the broader field. Consequently, the significance of its endogenous‑prediction solution is partially circumscribed, as alternative architectural avenues already exist to achieve encoder‑free spatial intelligence without offline 3D‑expert supervision.
6 comments
Reply to Reviewer dSPh (Part2)
W1 (What the lightweight prediction heads learn)
“use a smaller encoder to estimate complex spatiotemporal changes in real-world scenarios”
Thank you for raising this question. A lightweight LFP head can make useful predictions because it operates on scene context already encoded by the full VLM and predicts latent features that abstract away many pixel-level changes.
-
Short-horizon prediction in feature space.
Cambrian-S (ICLR 2026) shows that a two-layer MLP can predict next-frame visual features and that its prediction errors can guide video memory management (Cambrian-S, Section 4.1 and Section 4.2). Building on this idea, our LFP heads predict geometry for the observed frame and visual semantics for the next frame. SGT-Fusion uses these predictions as priors for spatial reasoning.
| Dataset (scenes) | Noise AbsRel ↓ | Stage-1 LFP AbsRel ↓ | Stage-2 LFP AbsRel ↓ | VGGT AbsRel ↓ | Stage 2 AbsRel lower than VGGT ↑ |
|---|---|---|---|---|---|
| ScanNet (1,204) | 0.235 | 0.211 | 0.207 | 0.144 | 11.0% (132/1,204) |
| ScanNet++ (848) | 0.287 | 0.280 | 0.272 | 0.247 | 34.9% (296/848) |
| Structured3D (3,089) | 0.882 | 0.850 | 0.827 | 0.755 | 34.6% (1,068/3,089) |
Table A. Mean scale-aligned AbsRel across scenes. The last column gives the fraction of scenes where Stage-2 LFP has lower error than VGGT. The best LFP prediction is shown in monospace.
-
Geometric fidelity of LFP prediction.
As shown in Table A, we probed the geometry retained by LFP on all 5,141 SPAR-7M RGB-D scenes. We use the same frozen VGGT point head for all four feature inputs and report scale-aligned absolute relative error (AbsRel).
Stage-2 LFP achieves lower error than Stage-1 LFP and noise on all three datasets, showing that geometry remains recoverable after task tuning. VGGT yields the lowest mean AbsRel on each of the three datasets and achieves its lowest error on ScanNet, where it also has the largest relative advantage over Stage-2 LFP. This advantage may partly reflect the inclusion of ScanNet in VGGT’s reported training data. At the scene level, Stage-2 LFP yields lower error than VGGT on roughly one-third of ScanNet++ and Structured3D scenes.
W2 (Two-stage training and feature objectives)
“It relies on a two-stage training process. Stage-1 needs offline 3D experts for supervision”
Thank you for the comment. Our two-stage recipe separates learning to predict geometric and temporal priors from learning to use them for spatial reasoning. To assess the value of separating the two stages, we compared them with joint training from the start while keeping the inference architecture fixed.
-
Lightweight pretraining and a standard SFT objective.
Stage 1 updates only the lightweight LFP heads with the full VLM frozen. Its feature losses account for differences in scale and structure between the VGGT geometry features and next-frame visual embeddings. Stage 2 uses standard next-token cross-entropy as its sole objective, with the Stage-1 feature losses turned off.
| Training schedule | VSI-Bench ↑ |
|---|---|
| Joint training from the start | 49.32 |
| Two-stage training (GeoThinker-F) | 50.76 |
Table B. Comparison of training schedules on VSI-Bench in the Vanilla setting (Qwen2.5-VL-7B).
-
Effect of two-stage training.
As shown in Table B, we compared two training schedules for the same Qwen2.5-VL-7B architecture, with both models retaining LFP and SGT-Fusion at inference.
Joint training optimizes LFP, the language backbone, and SGT-Fusion together from the start. Our two-stage recipe first fits the prediction heads, then trains the VLM to use their predictions through SGT-Fusion. This improves VSI-Bench by 1.44 points with the same inference architecture.
W3 (Fast non-rigid motion and geometry fidelity)
“the internally predicted features are rougher compared to the real 3D encoder outputs.”
Thank you for your comment. LFP’s approximate geometry retains enough structure to improve spatial reasoning. In our controlled Vanilla 7B ablation (Table 3 in the paper), adding the predicted geometry branch alone raises VSI-Bench accuracy from 49.36 to 50.17 with the same SFT data. In the Scaled comparison under matched training data, GeoThinker-F also achieves six-benchmark averages comparable to those of GeoThinker, which uses VGGT at inference, on both Qwen backbones.
We also evaluated what the predictions retain. In Appendix Figure 6, LFP activations remain more concentrated around the main person or vehicle in selected snow and driving scenes, where VGGT activations are more diffuse. In the Table A probe, Stage-2 LFP yields lower AbsRel than VGGT on roughly one-third of ScanNet++ and Structured3D scenes.
Thank you once again for taking the time to review our paper and for your thoughtful feedback. We appreciate your questions about how predicted priors complement strong spatial models, what geometric information the lightweight heads can recover, and what the two-stage training schedule contributes. These questions helped us clarify the scope of GeoThinker-F and support our design choices with more direct comparisons.
We sincerely hope that the additional experiments and our response have addressed your concerns. If you have any further questions or suggestions, we would be glad to offer further clarifications. Thank you for your time and consideration!
Reply to Reviewer dSPh (Part1)
Thank you for your detailed and thoughtful review. We appreciate your recognition that GeoThinker-F reduces the inference cost of external geometry encoding while retaining competitive spatial reasoning. We are glad that you highlighted its results on MindCube and ViewSpatial, its six-benchmark average of 69.29 with Qwen3-VL-8B, and its comparable performance to GeoThinker under matched training data. We also appreciate your comments on the ablations and feature visualizations, which help connect the design to its empirical behavior. Your questions helped us clarify where geometry and temporal priors add value, what the lightweight heads can recover, and how the two-stage training recipe supports their use at inference.
Thank you for raising this point about the motivation of our work. We agree that MLLM-4D and Qwen3.5 demonstrate strong spatial reasoning. GeoThinker-F complements these advances by predicting geometric and temporal priors from the VLM’s own hidden states and selectively integrating them to build on its existing capabilities. For completeness, we now cite and discuss both models in Related Work (line 155-157) and Discussion and Scope (line 1843), with the additions highlighted in red.
Geometry priors complement strong VLMs.
SpatialStack (CVPR 2026) shows that adding VGGT improves the VSI-Bench score of fine-tuned Qwen3.5 from 64.76 to 67.52. SpaceMind (CVPR 2026) reports a similar gain when adding VGGT features to fine-tuned InternVL3-8B, from 63.07 to 66.77. SpatialStack, Table 2 · SpaceMind, Table 4
We tested whether internally predicted priors can also help an already spatially trained backbone. On SenseNova-SI-1.1-Qwen3-VL-8B, adding GeoThinker-F to SFT on the same Scaled training data raises the six-benchmark average from 68.31 to 69.07, with modest gains on all six benchmarks.
VGGT is also used in robotic manipulation and autonomous driving.
VGGDrive (CVPR 2026) integrates frozen VGGT features for driving tasks including trajectory planning. R3DP (ECCV 2026, oral) runs VGGT on sparse keyframes during policy inference to provide 3D features for robotic manipulation. VGGDrive · R3DP
Our evaluations measure the performance and runtime of generating the priors internally in two downstream settings.
Temporal prediction also improves spatial reasoning.
GeoThinker-F also predicts next-frame visual semantics to help the VLM reason about how a scene evolves over time. In our Vanilla 7B ablation, adding the temporal branch alone raises Appearance Order from 56.6 to 59.7 and overall VSI-Bench from 49.36 to 50.54. This shows that the prediction and fusion design can improve reasoning using temporal priors alone.