acceptodds
Under review as a conference paper at ICLR 2027

GeoThinker-: Endogenous Predictive World Modeling for Fast Spatial Intelligence

Abstract

Existing geometry-aware Multimodal Large Language Models (MLLMs) increasingly rely on external 3D encoders to provide structural priors, but executing these additional encoder paths at inference introduces substantial latency and computational overhead. We propose GeoThinker-, an efficient framework that internalizes spatiotemporal perception by transforming causal MLLM hidden states into predictive latent world representations. Our key insight is that intermediate semantic representations contain sufficient information to predict useful representations of current geometry and future semantics, without requiring an exact reproduction of features from a dedicated 3D encoder. To realize this idea, GeoThinker- adopts a decoupled two-stage training strategy. In the first stage, the MLLM backbone is frozen while lightweight Latent Frame Prediction (LFP) heads learn from an offline 3D geometry expert and next-frame semantic targets. In the second stage, the external expert is removed, and the model is jointly optimized to generate and selectively integrate its self-predicted priors through a specialized Spatial-Grounded Temporal (SGT) Fusion module. Extensive evaluations show that GeoThinker- eliminates runtime dependence on external 3D geometry encoders while maintaining competitive performance across diverse spatial intelligence benchmarks. It achieves up to a inference speedup, reducing latency from to while retaining strong spatial reasoning performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
MLLMs, Spatial intelligence

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dSPhOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GeoThinker‑F, an efficient multimodal large‑language‑model framework for spatial intelligence. Instead of running an external 3D geometry encoder during inference, GeoThinker‑F learns to endogenously predict current‑frame geometric representations and future‑frame semantics from internal model hidden states via lightweight Latent Frame Prediction (LFP) heads. It adopts a two‑stage training pipeline: first pre‑training LFP heads supervised by an offline 3D expert, then self‑driven tuning with the proposed Spatial‑Grounded Temporal (SGT) Fusion that interleaves geometry and temporal features across transformer layers. Evaluated on multiple spatial‑reasoning benchmarks, GeoThinker‑F achieves competitive spatial‑understanding performance while delivering up to 3.6× inference speedup. Though it suffers from prediction drift on extremely long sequences and subpar metric distance estimation compared with systems using real‑time 3D encoders, it offers a favourable accuracy‑efficiency trade‑off, especially for latency‑sensitive embodied and interactive applications.

Critical strengths
  1. The core idea of the paper is that existing VLMs require a 3D encoder to handle spatiotemporal cognition tasks, which takes a lot of time. The paper removes the external 3D geometry encoder during the inference stage and predicts geometry and temporal representations inside the model's latent space, balancing spatial reasoning accuracy and inference efficiency.

  2. Standing out on spatial benchmarks like MindCube and ViewSpatial, Qwen3‑VL‑8B scored an average of 69.29 across 6 spatial benchmarks, ranking 2nd on the EASI‑8 list; under controlled conditions with the same training data, its performance can match the GeoThinker baseline with an external encoder.

  3. The paper did a lot of ablation studies and feature visualizations, which is great, making the characteristics of the proposed method clearer.

Critical weaknesses
  1. The paper tries to use a smaller encoder to estimate complex spatiotemporal changes in real-world scenarios, which isn't realistic. I think this somewhat weakens the paper's transferability.

  2. It relies on a two-stage training process. Stage‑1 needs offline 3D experts for supervision, has a complex multi-objective loss function, and the training and tuning are more complicated than regular fine-tuning.

  3. It's hard to capture the fine, fleeting details of fast, non-rigid deformations (like fluttering fabric or small animal fur); the internally predicted features are rougher compared to the real 3D encoder outputs.

Questions for authors

This issue mainly affected my impression of this piece of work:

While GeoThinker‑F frames heavy‑runtime external 3D encoders as a widespread bottleneck for spatial‑aware MLLMs, its core motivating assumption risks over‑generalization. A growing line of work, such as MLLM‑4D which produces explicit textual geometric outputs natively and Qwen3.5 that builds hybrid spatio‑temporal feature embeddings, demonstrates that competitive spatial reasoning can be realized without auxiliary 3D encoders at either training or inference time. These models can be fully optimized under a unified SFT+GRPO pipeline, avoiding both runtime geometry overhead and the two‑stage, teacher‑dependent training paradigm adopted in GeoThinker‑F. This means the pain point the paper addresses is not a universal necessity for spatial MLLMs. This weakens the paper’s core motivation: the need to replace external 3D encoders is only critical for a specific subclass of prior dual‑encoder architectures like GeoThinker, rather than the broader field. Consequently, the significance of its endogenous‑prediction solution is partially circumscribed, as alternative architectural avenues already exist to achieve encoder‑free spatial intelligence without offline 3D‑expert supervision.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer dSPh (Part1)

Thank you for your detailed and thoughtful review. We appreciate your recognition that GeoThinker-F reduces the inference cost of external geometry encoding while retaining competitive spatial reasoning. We are glad that you highlighted its results on MindCube and ViewSpatial, its six-benchmark average of 69.29 with Qwen3-VL-8B, and its comparable performance to GeoThinker under matched training data. We also appreciate your comments on the ablations and feature visualizations, which help connect the design to its empirical behavior. Your questions helped us clarify where geometry and temporal priors add value, what the lightweight heads can recover, and how the two-stage training recipe supports their use at inference.

Q1 (Scope of predictive world representations)
“competitive spatial reasoning can be realized without auxiliary 3D encoders at either training or inference time.”

Thank you for raising this point about the motivation of our work. We agree that MLLM-4D and Qwen3.5 demonstrate strong spatial reasoning. GeoThinker-F complements these advances by predicting geometric and temporal priors from the VLM’s own hidden states and selectively integrating them to build on its existing capabilities. For completeness, we now cite and discuss both models in Related Work (line 155-157) and Discussion and Scope (line 1843), with the additions highlighted in red.

  1. Geometry priors complement strong VLMs.

    SpatialStack (CVPR 2026) shows that adding VGGT improves the VSI-Bench score of fine-tuned Qwen3.5 from 64.76 to 67.52. SpaceMind (CVPR 2026) reports a similar gain when adding VGGT features to fine-tuned InternVL3-8B, from 63.07 to 66.77. SpatialStack, Table 2 · SpaceMind, Table 4

    We tested whether internally predicted priors can also help an already spatially trained backbone. On SenseNova-SI-1.1-Qwen3-VL-8B, adding GeoThinker-F to SFT on the same Scaled training data raises the six-benchmark average from 68.31 to 69.07, with modest gains on all six benchmarks.

  2. VGGT is also used in robotic manipulation and autonomous driving.

    VGGDrive (CVPR 2026) integrates frozen VGGT features for driving tasks including trajectory planning. R3DP (ECCV 2026, oral) runs VGGT on sparse keyframes during policy inference to provide 3D features for robotic manipulation. VGGDrive · R3DP

    Our evaluations measure the performance and runtime of generating the priors internally in two downstream settings.

    • LIBERO (Table 5). At 30K training steps, GeoThinker-F achieves 97.1% mean success, compared with 96.6% for StarVLA-OFT and 96.7% for GeoThinker. Relative to GeoThinker, it reduces inference latency from 254.8 to 104.2 ms.
    • NAVSIM (Tables 4 and 7). GeoThinker-F reaches 83.3 PDMS, compared with 81.9 for ReCogDrive and 84.4 for GeoThinker. Relative to GeoThinker, it reduces amortized latency from 42.62 to 37.08 ms per planning token.
  3. Temporal prediction also improves spatial reasoning.

    GeoThinker-F also predicts next-frame visual semantics to help the VLM reason about how a scene evolves over time. In our Vanilla 7B ablation, adding the temporal branch alone raises Appearance Order from 56.6 to 59.7 and overall VSI-Bench from 49.36 to 50.54. This shows that the prediction and fusion design can improve reasoning using temporal priors alone.

AuthorsOpenReview ↗

Reply to Reviewer dSPh (Part2)

W1 (What the lightweight prediction heads learn)
“use a smaller encoder to estimate complex spatiotemporal changes in real-world scenarios”

Thank you for raising this question. A lightweight LFP head can make useful predictions because it operates on scene context already encoded by the full VLM and predicts latent features that abstract away many pixel-level changes.

  1. Short-horizon prediction in feature space.

    Cambrian-S (ICLR 2026) shows that a two-layer MLP can predict next-frame visual features and that its prediction errors can guide video memory management (Cambrian-S, Section 4.1 and Section 4.2). Building on this idea, our LFP heads predict geometry for the observed frame and visual semantics for the next frame. SGT-Fusion uses these predictions as priors for spatial reasoning.

Dataset (scenes)Noise AbsRel ↓Stage-1 LFP AbsRel ↓Stage-2 LFP AbsRel ↓VGGT AbsRel ↓Stage 2 AbsRel lower than VGGT ↑
ScanNet (1,204)0.2350.2110.2070.14411.0% (132/1,204)
ScanNet++ (848)0.2870.2800.2720.24734.9% (296/848)
Structured3D (3,089)0.8820.8500.8270.75534.6% (1,068/3,089)

Table A. Mean scale-aligned AbsRel across scenes. The last column gives the fraction of scenes where Stage-2 LFP has lower error than VGGT. The best LFP prediction is shown in monospace.

  1. Geometric fidelity of LFP prediction.

    As shown in Table A, we probed the geometry retained by LFP on all 5,141 SPAR-7M RGB-D scenes. We use the same frozen VGGT point head for all four feature inputs and report scale-aligned absolute relative error (AbsRel).

    Stage-2 LFP achieves lower error than Stage-1 LFP and noise on all three datasets, showing that geometry remains recoverable after task tuning. VGGT yields the lowest mean AbsRel on each of the three datasets and achieves its lowest error on ScanNet, where it also has the largest relative advantage over Stage-2 LFP. This advantage may partly reflect the inclusion of ScanNet in VGGT’s reported training data. At the scene level, Stage-2 LFP yields lower error than VGGT on roughly one-third of ScanNet++ and Structured3D scenes.

W2 (Two-stage training and feature objectives)
“It relies on a two-stage training process. Stage-1 needs offline 3D experts for supervision”

Thank you for the comment. Our two-stage recipe separates learning to predict geometric and temporal priors from learning to use them for spatial reasoning. To assess the value of separating the two stages, we compared them with joint training from the start while keeping the inference architecture fixed.

  1. Lightweight pretraining and a standard SFT objective.

    Stage 1 updates only the lightweight LFP heads with the full VLM frozen. Its feature losses account for differences in scale and structure between the VGGT geometry features and next-frame visual embeddings. Stage 2 uses standard next-token cross-entropy as its sole objective, with the Stage-1 feature losses turned off.

Training scheduleVSI-Bench ↑
Joint training from the start49.32
Two-stage training (GeoThinker-F)50.76

Table B. Comparison of training schedules on VSI-Bench in the Vanilla setting (Qwen2.5-VL-7B).

  1. Effect of two-stage training.

    As shown in Table B, we compared two training schedules for the same Qwen2.5-VL-7B architecture, with both models retaining LFP and SGT-Fusion at inference.

    Joint training optimizes LFP, the language backbone, and SGT-Fusion together from the start. Our two-stage recipe first fits the prediction heads, then trains the VLM to use their predictions through SGT-Fusion. This improves VSI-Bench by 1.44 points with the same inference architecture.

W3 (Fast non-rigid motion and geometry fidelity)
“the internally predicted features are rougher compared to the real 3D encoder outputs.”

Thank you for your comment. LFP’s approximate geometry retains enough structure to improve spatial reasoning. In our controlled Vanilla 7B ablation (Table 3 in the paper), adding the predicted geometry branch alone raises VSI-Bench accuracy from 49.36 to 50.17 with the same SFT data. In the Scaled comparison under matched training data, GeoThinker-F also achieves six-benchmark averages comparable to those of GeoThinker, which uses VGGT at inference, on both Qwen backbones.

We also evaluated what the predictions retain. In Appendix Figure 6, LFP activations remain more concentrated around the main person or vehicle in selected snow and driving scenes, where VGGT activations are more diffuse. In the Table A probe, Stage-2 LFP yields lower AbsRel than VGGT on roughly one-third of ScanNet++ and Structured3D scenes.

Thank you once again for taking the time to review our paper and for your thoughtful feedback. We appreciate your questions about how predicted priors complement strong spatial models, what geometric information the lightweight heads can recover, and what the two-stage training schedule contributes. These questions helped us clarify the scope of GeoThinker-F and support our design choices with more direct comparisons.

We sincerely hope that the additional experiments and our response have addressed your concerns. If you have any further questions or suggestions, we would be glad to offer further clarifications. Thank you for your time and consideration!

Reviewer_4SemOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GeoThinker-F, a framework that replaces runtime external geometry encoding with internally predicted features. Lightweight Latent Frame Prediction (LFP) heads are first trained with a frozen VLM to predict current geometry and next-frame semantics. These features are then integrated through Spatial-Grounded Temporal Fusion during spatial instruction tuning. Experiments with Qwen2.5-VL-7B and Qwen3-VL-8B report competitive performance with reduced inference cost across VSI-Bench, MMSI-Bench, MindCube, ViewSpatial, SITE, and CV-Bench.

Critical strengths
  • The paper extends GeoThinker by reducing the computational overhead of external geometry encoders while maintaining competitive spatial reasoning performance.
  • The paper is well organized, and the evaluations cover multiple spatial benchmarks and downstream applications.
Critical weaknesses
  • Incomplete training details. The complete Stage-2 objective, trainable components, and continued use of feature-prediction losses are insufficiently specified. The temporal supervision targets and their alignment with intermediate VLM states also need clarification.

  • Missing recent comparisons. Relevant CVPR and ICML 2026 methods—including SpatioLM [1], SpatialStack [2], SpaceMind [3], Beyond 3D VQAs [4], HiSpatial [5], and EgoMind [6]—are absent from the comparisons. SpatioLM is particularly relevant because it also removes runtime dependence on an external spatial encoder; the methodological and empirical differences need clarification. The 3DThinker baseline should also be checked against its final CVPR version [7]. Comparisons under matched settings would better establish the framework’s advantages.

  • Insufficient analysis of experimental results. Table 1 shows regressions relative to GeoThinker in room-size estimation (62.2→58.1) and relative direction (46.2→41.8). With the same Qwen2.5-VL-7B backbone, MMSI-Bench performance also trails 3DThinker (31.2 versus 43.3). These results warrant further analysis of the predicted priors and their task-specific benefits. VGGT’s established geometric capabilities do not guarantee that lightweight LFP heads recover equally useful information from VLM states. Heatmaps and final QA scores provide limited validation of geometric fidelity, particularly after Stage-2 adaptation.

[1] SpatioLM: Towards General Physical Spatial Intelligence in Vision-Language Models. ICML 2026.

[2] SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning. CVPR 2026.

[3] SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models. CVPR 2026.

[4] Beyond 3D VQAs: Injecting 3D Spatial Priors into Vision-Language Models for Enhanced Geometric Reasoning. CVPR 2026.

[5] HiSpatial: Taming Hierarchical 3D Spatial Understanding in Vision-Language Models. CVPR 2026.

[6] EgoMind: Activating Spatial Cognition through Linguistic Reasoning in MLLMs. CVPR 2026.

[7] Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Views. CVPR 2026.

Questions for authors
  • Can the authors quantitatively validate the geometry retained by LFP outputs, including after Stage-2 tuning, using geometric probes or cross-view consistency measurements?
  • How do the authors explain the task-level regressions relative to GeoThinker in Table 1? Are they associated with identifiable errors in the predicted geometric features?
  • Can the framework generalize to stronger backbones such as SenseNova-SI-8B and alternative geometry-integration designs under controlled training settings?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer 4Sem (Part1)

Thank you for your detailed and constructive review. We appreciate your recognition of the main contribution of GeoThinker-F: it reduces the inference overhead of external geometry encoding while maintaining competitive spatial reasoning. We are also grateful for your positive comments on the organization of the paper and the breadth of our evaluation across six spatial reasoning benchmarks and downstream applications in embodied manipulation and autonomous driving. Your questions motivated us to explain the training recipe more precisely, evaluate recent methods on our benchmarks, and directly test what geometric information LFP retains after task tuning. We also tested whether changing the Stage-1 data mixture improves the two VSI-Bench categories you identified and evaluated transfer to the SenseNova-SI backbone you suggested.

W1 (Stage-2 objective and temporal supervision)
“The complete Stage-2 objective, trainable components, and continued use of feature-prediction losses are insufficiently specified.”

Thank you for your comment. Stage 2 uses next-token cross-entropy alone. We train the language backbone and SGT-Fusion while keeping the vision tower and visual projector frozen. LFP remains frozen in Vanilla and is jointly tuned through the language loss in the default Scaled setting. Geometry and temporal feature-prediction losses are used in Stage 1, where the VLM is frozen. For temporal supervision, the selected language layers predict from the visual-token states of frame ii. We average their predictions and align them by spatial token index to the frozen visual encoder's post-merger embeddings of frame i+1i+1.

We clarified the Stage-2 language objective and trainable components in Section 3.2 (starting at line 302) and highlighted the LFP freeze/unfreeze settings in Appendix B.1. Appendix K.3 describes the temporal supervision. We also added a discussion of the teacher's multi-view context and the student's causal inputs in Appendix K.4 (line 1662).

W2 (Recent comparisons)
“SpatioLM is particularly relevant because it also removes runtime dependence on an external spatial encoder”

Thank you for suggesting these comparisons. We evaluated SpatioLM, SpatialStack, and EgoMind on the six benchmarks and added the results to Table 2. The table below shows the overall average and six benchmark scores.

MethodSix-benchmark avg. ↑VSI-Bench ↑MMSI-Bench ↑MindCube ↑ViewSpatial ↑SITE ↑CV-Bench ↑
SpatioLM63.4171.641.585.342.657.581.7
SpatialStack53.6967.528.135.846.958.285.5
EgoMind46.0850.122.330.539.158.076.2
GeoThinker-F (Qwen3-VL-8B)69.2970.430.893.282.653.785.0

Table A. Selected results from the revised Table 2.

SpatioLM and GeoThinker-F both avoid running an external geometry encoder at inference. SpatioLM learns a geometry-supervised Spatio-Vision module alongside a frozen VLM. GeoThinker-F predicts current-frame geometry and next-frame semantics from VLM states and integrates those predictions through SGT-Fusion. Across the six benchmarks, GeoThinker-F averages 69.29 vs. 63.41 for SpatioLM. The largest advantage in this comparison is on ViewSpatial (82.6 vs. 42.6), with a further gain on MindCube (93.2 vs. 85.3). SpatioLM leads on VSI-Bench and MMSI-Bench.

We also evaluated the single-image HiSpatial model on CV-Bench. For SpaceMind and GASP, we report the results from their papers, as their model weights were not publicly available at the time of our evaluation. Following the final CVPR version of 3DThinker, we report its MindCube-only configuration (76.0 on MindCube-Tiny) and its large-scale S1+S2 configuration (43.3 on MMSI-Bench) on separate rows.

To make the comparison more comprehensive, we cite the suggested papers and include their benchmark results in Table 2, with the added results highlighted in red.

AuthorsOpenReview ↗

Reply to Reviewer 4Sem (Part2)

W3, Q1 (Geometry retained after Stage 2)
“Can the authors quantitatively validate the geometry retained by LFP outputs, including after Stage-2 tuning, using geometric probes or cross-view consistency measurements?”

Thank you for suggesting a quantitative probe. Across all 5,141 SPAR-7M RGB-D scenes, Stage-2 LFP yields lower mean error than Stage-1 LFP and the noise control on all three datasets. We use the same frozen VGGT point head for all four feature inputs and report scale-aligned absolute relative error (AbsRel).

Dataset (scenes)Noise AbsRel ↓Stage-1 LFP AbsRel ↓Stage-2 LFP AbsRel ↓VGGT AbsRel ↓Stage 2 AbsRel lower than VGGT ↑
ScanNet (1,204)0.2350.2110.2070.14411.0% (132/1,204)
ScanNet++ (848)0.2870.2800.2720.24734.9% (296/848)
Structured3D (3,089)0.8820.8500.8270.75534.6% (1,068/3,089)

Table B. Mean scale-aligned AbsRel across scenes. The last column gives the fraction of scenes where Stage-2 LFP has lower error than VGGT. The best LFP prediction is shown in monospace.

Geometry remains recoverable from LFP features after task tuning. VGGT yields the lowest mean AbsRel on each of the three datasets and achieves its lowest error on ScanNet, where it also has the largest relative advantage over Stage-2 LFP. This advantage may partly reflect the inclusion of ScanNet in VGGT’s reported training data. At the scene level, Stage-2 LFP yields lower error than VGGT on roughly one-third of ScanNet++ and Structured3D scenes.

W3, Q2 (Task-level differences and MMSI-Bench)
“How do the authors explain the task-level regressions relative to GeoThinker in Table 1? Are they associated with identifiable errors in the predicted geometric features?”

Thank you for highlighting these task-level differences. Both categories are sensitive to the Stage-1 pretraining mixture. Adding SPAR to LLaVA-Hound roughly halves both gaps to GeoThinker while maintaining a comparable overall VSI-Bench score.

Vanilla Qwen2.5-VL-7BVSI-Bench avg. ↑Room Size ↑Relative Direction ↑Appearance Order ↑
GeoThinker50.5062.246.252.6
GeoThinker-F (LLaVA-Hound)50.7658.141.863.6
GeoThinker-F (LLaVA-Hound + SPAR)50.6060.244.160.0

Table C. Vanilla VSI-Bench scores for the two Stage-1 pretraining mixtures and the GeoThinker reference.

Adding SPAR improves Room Size by 2.1 points and Relative Direction by 2.3 points, roughly halving both gaps to GeoThinker. The overall score remains comparable (50.76 vs. 50.60), and Appearance Order remains 7.4 points above GeoThinker. The result shows that a more spatially focused Stage-1 data mixture improves the balance across these tasks.

In the Scaled comparison on Qwen2.5-VL-7B, 3DThinker leads on MMSI-Bench, while GeoThinker-F leads by 3.6 points on VSI-Bench and 14.6 points on ViewSpatial. Thank you for highlighting this comparison. It motivates us to explore 3DThinker’s reinforcement learning approach to improve GeoThinker-F on MMSI-Bench.

Q3 (Transfer to a stronger backbone)
“Can the framework generalize to stronger backbones such as SenseNova-SI-8B and alternative geometry integration designs under controlled training settings?”

Thank you for suggesting SenseNova-SI-8B. We compared standard SFT with SFT using GeoThinker-F on the same Scaled training data and from the same SenseNova-SI-1.1-Qwen3-VL-8B checkpoint. Adding our modules improves all six benchmarks and raises the average from 68.31 to 69.07.

SenseNova-SI-1.1-Qwen3-VL-8BAvg.VSI-BenchMMSI-BenchMindCube-TinyViewSpatialSITECV-Bench
Original60.3864.838.173.851.249.684.78
+ Standard SFT68.3168.7829.6092.5782.7952.4483.69
+ SFT and GeoThinker-F modules69.0769.4630.5193.3383.6753.4084.06

Table D. SenseNova-SI-1.1 before adaptation and after Scaled SFT with and without GeoThinker-F modules.

Standard SFT raises the original checkpoint’s average from 60.38 to 68.31. GeoThinker-F adds a further 0.76 points on average, with gains ranging from 0.37 points on CV-Bench to 0.96 points on SITE, including 0.91 points on MMSI-Bench. The modest gains across all six benchmarks show that the prediction and fusion modules transfer to an already spatially trained backbone. On Qwen2.5-VL-7B, Table 3 also compares geometry-only, temporal-only, and joint fusion.

Thank you once again for taking the time to review our paper and for your constructive feedback. We appreciate your suggestions to compare with recent methods, quantify the geometry retained by LFP after task tuning, and test transfer to a stronger backbone. Your comments helped us make the evaluation more comprehensive and clarify the training procedure.

We sincerely hope that the additional experiments and our response have addressed your concerns. If you have any further questions or suggestions, we would be glad to offer further clarifications. Thank you for your time and consideration!

Reviewer_bkaWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduce a method called GeoThinker-F, it aims to reduce the inference cost of geometry-aware multimodal models by replacing an external geometry encoder with lightweight internal prediction heads. In the first training stage, the VLM is frozen while these heads learn to predict current-frame geometry features supervised by an offline VGGT teacher, together with next-frame semantic features. In the second stage, the external teacher is removed, and the predicted representations are integrated through SGT-Fusion during instruction tuning, with geometry and temporal features injected into alternating layers. At inference, the model retains the prediction and fusion modules, while additional spatial token compression further reduces computational cost. Experiments examine the resulting accuracy–efficiency trade-off across spatial reasoning benchmarks and downstream tasks.

Critical strengths
  • The paper addresses a well-defined efficiency problem and demonstrates meaningful architectural savings.
  • Controlled-data comparisons and pretraining/component ablations provide useful evidence for the proposed approach.
  • Methods remain valid cross different backbone (Qwen2.5-VL and Qwen3-VL)
  • Removes the external geometry encoder at inference while maintaining competitive spatial reasoning performance.
Critical weaknesses
  • Table 6 omits accuracy for the variants without further token compression, leaving their accuracy–latency trade-off incomplete.
  • The headline scaled-regime gains partly reflect additional OmniSpatial training data; matched-data results show average parity rather than consistent improvements across benchmarks.
  • The benefit of retaining LFP and SGT-Fusion over training-only geometric supervision remains unclear.
  • The related work should discuss 3D-IDE: 3D Implicit Depth Emergent (CVPR 2026), which removes its auxiliary geometry modules for RGB-only inference.
Questions for authors
  1. How does GeoThinker-F compare with a training-only auxiliary-supervision baseline that removes its auxiliary heads at inference, under comparable backbone, teacher, data, geometry/temporal supervision, and input budgets?
  2. Could you report the missing accuracy for the uncompressed variants in Table 6, together with their input resolutions and token counts, to clarify the accuracy–latency trade-off?
  3. How are the teacher’s and student’s information scopes aligned? The appendix notes that frame-aligned VGGT targets are generated via joint multi-view encoding, whereas the student predicts from causal hidden states. While privileged teacher context does not inherently constitute information leakage, these architectural distinctions should be made explicit.

I will raise my rating if the authors address my concerns.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer bkaW (Part1)

Thank you for your constructive and encouraging review. We appreciate your recognition that runtime geometry extraction is a meaningful efficiency bottleneck and that GeoThinker-F removes the external 3D encoder while retaining competitive spatial reasoning. We are glad that you found the efficiency gains across both Qwen2.5-VL and Qwen3-VL convincing, and that the controlled-data comparisons and the pretraining and component ablations helped support the design. Your questions helped us sharpen two central comparisons: the accuracy–latency trade-off with further token compression and the value of retaining the prediction and fusion modules at inference. We also discuss the connection to 3D-IDE and clarify how the teacher and student are aligned during LFP pretraining.

W1, Q2 (Accuracy and latency with token compression)
“Could you report the missing accuracy for the uncompressed variants in Table 6, together with their input resolutions and token counts, to clarify the accuracy–latency trade-off?”

Thank you for your suggestion. We evaluated all four 32-frame configurations with further token compression disabled. Enabling compression lowers FLOPs and latency and also improves VSI-Bench accuracy in all four comparisons.

BackboneModelFurther compressionFLOPs (T) ↓Latency (s) ↓VSI-Bench ↑
Qwen2.5-VL-7BGeoThinkerX229.741.6968.1
GeoThinker✓135.051.4568.5
GeoThinker-FX173.321.0465.6
GeoThinker-F (Ours)✓55.330.4867.3
Qwen3-VL-8BGeoThinkerX201.881.6670.7
GeoThinker✓113.771.4572.6
GeoThinker-FX142.590.9165.4
GeoThinker-F (Ours)✓43.010.4070.4

Table A. Accuracy and efficiency on 32-frame VSI-Bench inputs. ✓ enables further token compression. X uses base tokenization.

For GeoThinker-F, further compression roughly halves latency on both backbones, from 1.04 to 0.48 s on Qwen2.5-VL-7B and from 0.91 to 0.40 s on Qwen3-VL-8B. The Scaled GeoThinker-F checkpoints were trained with further compression enabled for long inputs. Keeping it enabled at evaluation therefore matches their training configuration. In these runs, that configuration gives the better accuracy–latency trade-off on both backbones.

All runs use the same 32 VSI-Bench frames per example, with an original resolution of 640 × 480. Qwen2.5-VL-7B resizes each frame to 504 × 392 and uses 8,064 visual tokens before further compression and 2,016 after. Qwen3-VL-8B resizes each frame to 512 × 384 and uses 6,144 visual tokens before further compression and 1,536 after. Within each backbone and compression setting, GeoThinker and GeoThinker-F use the same input resolution and visual-token count.

We added the four scores to Appendix Table 6 (p. 20) and highlighted them in red.

W2 (Matched-data accuracy and inference efficiency)
“matched-data results show average parity rather than consistent improvements across benchmarks.”

Thank you for emphasizing the matched-data comparison. We report accuracy and end-to-end latency side by side. With the same training data, GeoThinker-F maintains comparable average accuracy across six benchmarks on both backbones while delivering 3.0× and 3.6× inference speedups.

BackboneGeoThinker avg.GeoThinker-F avg.GeoThinker latency (s)GeoThinker-F latency (s)Speedup
Qwen2.5-VL-7B60.4360.561.450.483.0×
Qwen3-VL-8B62.2362.211.450.403.6×

Table B. Six-benchmark averages with matched training data (Table 16) and 32-frame end-to-end latency (Table 6). For each model and backbone, the matched-data checkpoint uses the same inference architecture and token-compression settings as the checkpoint timed in Table 6.

This is the operating point GeoThinker-F was designed to improve. GeoThinker executes a separate geometry encoder at inference to provide spatial priors. GeoThinker-F predicts and fuses its priors internally. The matched-data results show that this change preserves overall spatial reasoning performance across the six benchmarks while substantially reducing inference cost on both backbones.

AuthorsOpenReview ↗

Reply to Reviewer bkaW (Part2)

W3, Q1 (Comparison with training-only auxiliary supervision)
“How does GeoThinker-F compare with a training-only auxiliary-supervision baseline that removes its auxiliary heads at inference, under comparable backbone, teacher, data, geometry/temporal supervision, and input budgets?”

MethodLFP + SGT-Fusion at inferenceVSI-Bench ↑
GeoThinker-F (Ours)✓50.76
Training-only distillation (pretrained, frozen LFP)X45.98
Standard SFTX49.36

Table C. Main comparison on Vanilla VSI-Bench with Qwen2.5-VL-7B.

Thank you for suggesting this comparison. We trained a baseline that uses LFP feature supervision during training and runs only the VLM at inference. GeoThinker-F improves Vanilla VSI-Bench from 45.98 to 50.76 (+4.78 points) in the closest comparison on Qwen2.5-VL-7B.

GeoThinker-F and the training-only distillation baseline start Stage 2 from the same Stage-1 checkpoint and share the same teacher targets, SFT data, and input budget. Both keep the pretrained LFP frozen in Stage 2. In the training-only baseline, the LFP feature loss updates the VLM alongside the SFT loss in Stage 2. At inference, the resulting VLM answers directly. In GeoThinker-F, Stage 2 trains the VLM and SGT-Fusion with the task loss, and SGT-Fusion continues to use LFP predictions at inference. Standard SFT provides a reference for training the VLM with the task loss alone.

GeoThinker-F also improves over standard SFT by 1.40 points. Its separate geometry and temporal predictions let SGT-Fusion selectively integrate task-relevant priors at the token level during inference. We also tested whether changing the distillation schedule would improve the training-only baseline. Table D shows the three additional variants and a joint-training control that retains the same inference modules as GeoThinker-F.

Additional controlStage 1Stage 2LFP + SGT-Fusion at inferenceVSI-Bench ↑
Joint training with fusion—Jointly train VLM, LFP, and SGT-Fusion✓49.32
Pretrain LFP, then distill jointlyTrain LFP onlySFT + distillation, update VLM and LFPX47.34
Distill jointly from the start—SFT + distillation, update VLM and LFPX46.14
Distill before SFTDistill into VLM and LFPStandard SFTX46.88

Table D. Additional training controls under the same Vanilla setting.

Allowing LFP to update during Stage 2 gives the strongest training-only result (47.34). GeoThinker-F remains 3.42 points higher. The joint-training control answers a separate question. It keeps the same prediction and fusion modules at inference and scores 49.32, compared with 50.76 for our two-stage recipe. Together, these comparisons show the benefits of first fitting the prediction heads and then learning to use their outputs during task tuning.

W4 (Relation to 3D-IDE)
“The related work should discuss 3D-IDE: 3D Implicit Depth Emergent”

Thank you for pointing us to 3D-IDE. Both methods use training-time 3D supervision to support RGB-only inference without executing an external geometry encoder. 3D-IDE learns geometry-aware visual tokens through auxiliary depth prediction, cross-view consistency, and scene-level alignment with VGGT features. At inference, it discards the auxiliary depth head and passes the resulting visual tokens to the language model. GeoThinker-F uses its lightweight prediction heads at inference to produce current-frame geometry and next-frame semantic features. SGT-Fusion integrates these priors selectively at alternating intermediate layers of the language model.

We now cite and discuss 3D-IDE in Section 2.2 (lines 178–182), with the new passage highlighted in red.

AuthorsOpenReview ↗

Reply to Reviewer bkaW (Part3)

Q3 (Teacher and student information scope)
“How are the teacher’s and student’s information scopes aligned?”

Thank you for the comment. The geometry prediction and target are aligned on frame ii's spatial token grid, while the target can contain information from views beyond the student's causal context. Both LFP heads predict from the VLM's causal hidden states at the visual-token positions of frame ii, using the observations available up to those positions.

For geometry, VGGT jointly encodes the sampled clip. The target is anchored to frame ii and can incorporate evidence from later views. This supervision trains the geometry head to estimate, from the observations available so far, the scene geometry that VGGT derives from multiple viewpoints. For temporal prediction, the supervision comes from the visual features of frame i+1i+1. Predicting those features from frame-ii states trains the temporal head to anticipate how the visual representation evolves.

The multi-view geometry targets and next-frame visual features are used as supervision in Stage 1. At inference, LFP generates both priors from the VLM's causal states, and SGT-Fusion uses the predictions for reasoning. We added a short discussion in Appendix K.4 (line 1662) explaining how the teacher’s additional context is used during Stage 1 and what the student can observe at inference. The corresponding prediction targets are highlighted in red in Section 3.1.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response — Thanks to All Reviewers for Their Constructive Feedback

We would like to thank all reviewers for their careful reading and constructive feedback. We are encouraged by your recognition of GeoThinker-F’s ability to reduce inference costs while maintaining competitive spatial reasoning.

Reviewer bkaW recognized the importance of reducing the inference overhead of external geometry encoding and highlighted the architectural savings across Qwen2.5-VL and Qwen3-VL. We also appreciate the reviewer’s recognition of the controlled-data comparisons and the pretraining and component ablations. Reviewer 4Sem highlighted that GeoThinker-F reduces the computational overhead of GeoThinker while maintaining competitive spatial reasoning. We are grateful for the reviewer’s positive comments on the organization of the paper and the breadth of the evaluation, including both spatial reasoning benchmarks and downstream applications. Reviewer dSPh highlighted the balance between spatial reasoning accuracy and inference efficiency, the strong results on MindCube and ViewSpatial, and the comparable performance to GeoThinker under matched training data. We also appreciate the reviewer’s recognition that the ablations and feature visualizations help explain the behavior of the method.

Your feedback guided the following additions. Our individual replies provide the full experimental results and explain how we addressed each reviewer’s questions.

  1. Token compression and matched-data efficiency (Reviewer bkaW). We evaluated the uncompressed variants on both Qwen backbones and reported their accuracy, FLOPs, latency, input resolutions, and visual-token counts. We also summarized the accuracy and latency comparison with GeoThinker under matched training data.
  2. Training-only supervision and two-stage training (Reviewers bkaW and dSPh). We tested several training-only supervision baselines and compared our two-stage recipe with joint training under the same inference architecture.
  3. Geometry retained after task tuning (Reviewers 4Sem and dSPh). We ran a geometric probe on all 5,141 SPAR-7M RGB-D scenes to compare Stage-1 and Stage-2 LFP features with VGGT and a noise control.
  4. Stage-1 data and task-level performance (Reviewer 4Sem). We tested whether adding SPAR to Stage-1 pretraining narrows the gaps on Room Size and Relative Direction.
  5. Transfer to a stronger backbone (Reviewers 4Sem and dSPh). We compared standard SFT with SFT using GeoThinker-F on SenseNova-SI-1.1-Qwen3-VL-8B under matched training data.
  6. Comparisons with recent work (Reviewer 4Sem). We expanded Table 2 with the suggested baselines and reported the two 3DThinker training configurations separately using the final CVPR results.

We also clarified the Stage-2 training objective and the teacher–student information scope, and added discussions of 3D-IDE, MLLM-4D, and Qwen3.5. These changes are highlighted in red in the revised manuscript. We thank the reviewers and the Area Chair for their time and consideration.

Dear ICLR reviewers,

Could you please check the rebuttal from the authors and see if the concerns have been addressed? Thank you!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.