- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GeoWorld-VLM, which uses features from a frozen, camera-conditioned world model to improve a VLM’s spatial understanding. The teacher is used only during training, leaving the original inference architecture unchanged.
Critical strengths
The paper is clearly structured and easy to read. The experiments cover multiple benchmarks, including What’sUp, VSR, and EmbSpatial-Bench, and two VLM families. The paper provides extensive ablations and reports training costs, generalization results, and limitations.
Critical weaknesses
-
Limited methodological novelty. There are already some distillation methods, like VGGT and DiNO, listed in the paper. And using such foundation models to supervise VLMs is not a novel idea.
-
Mismatch with claim and motivation. The author says that they would use a world model (basically a generative objective) to supervise VLMs for better spatial understanding. However, there is no clear evidence to show why a generative objective can help with understanding spatial relationships (Only citing some video generative papers is limited to support this). Table 3 shows that predicted frames even defaced the elements we really care about, like the lamp and vehicle. The author should explain those.
-
Training cost and parameter sensitivity. Appendix D shows that performance depends on the extraction layer and denoising stage. Given the additional training cost, the benefit over simpler approaches needs clearer justification, especially compared with lightweight methods such as AdaptVis(ICML 2025) and SpatialRGPT(NeurIPS 2024).
Questions for authors
- The comparison in Figure 3 does not explicitly explain why a generated model or objective is useful for understanding location. I expected the authors to provide more direct evidence that world-model supervision improves spatial understanding or depth representations.
- How are the visual feature response maps computed? Are they feature norms, attention weights, or something else?
- Do the improvements justify the additional training cost compared with simpler methods?
7 comments
Rebuttal by Authors (2/3)
W2 & Q1: Why Generative World-Model Supervision Benefits Spatial Understanding
A2: We appreciate the reviewer's request for a clearer explanation of why generative world-model supervision can benefit spatial understanding. We agree that high-quality video generation does not necessarily imply accurate spatial reasoning, and that qualitative generation examples alone are insufficient to establish this connection. However, we would like to clarify an important distinction: GeoWorld-VLM transfers intermediate predictive representations from a generative world model, rather than supervising the VLM with generated pixels or video reconstruction objectives.
(1) Why can generative representations provide useful spatial supervision? Camera-conditioned video prediction requires modeling how a scene changes under viewpoint transformations while maintaining its underlying spatial configuration. This objective encourages the world model to encode spatial cues such as relative object placement, depth, and occlusion, which are relevant to spatial relational understanding. Our method specifically elicits such predictive representations using sampled camera trajectories and a scene-stability prompt, and transfers them into the VLM's visual pathway through feature alignment. We do not assume that generative objectives guarantee geometrically accurate representations; rather, we investigate whether these representations provide useful supervision for downstream spatial reasoning.
(2) Quantitative evidence of predictive representation transfer. Our existing ablations provide evidence beyond qualitative video generation. Table 3 shows that jointly incorporating camera and prompt conditioning improves spatial QA accuracy from 64.01% (I2V-only) to 65.45%, with the same world-model architecture and student training configuration. Furthermore, Section 4.9 (Table 8) demonstrates that aggregating predictive features across nine frames achieves 65.45%, compared with 61.92% and 63.30% when using only the first or last frame, respectively. These results suggest that the predictive trajectory provides useful information beyond an individual frame representation.
Importantly, our evaluation also measures spatial reasoning directly. With the language model frozen, GeoWorld-VLM improves over matched task-only fine-tuning by 1.53–3.52 percentage points across four backbone–benchmark settings. On Gemma4/EmbSpatial, the improvements on close and far relations reach +3.99 and +4.10 points, respectively. These controlled comparisons demonstrate that the transferred representations are more effective for the evaluated spatial reasoning tasks, although they do not independently establish metric depth accuracy.
(3) Clarifying viewpoint-dependent changes in generated frames. We have revised the Figure 3 caption to identify the vehicle example's camera motion as a backward movement with a left turn. Viewpoint changes can alter visibility, so the vehicle's disappearance alone does not establish a spatial-consistency failure. For the lamp example, the displayed montage alone does not determine whether the apparent change is a viewpoint effect or a synthesis error. We therefore do not use these decoded examples as evidence of geometrically accurate generation. The revised caption distinguishes illustrative generated frames from the intermediate predictive features transferred during training; the conditioning and temporal-feature ablations, together with matched spatial QA evaluations, provide the quantitative evidence relevant to our claim.
Q2: Computation of Visual Feature Response Maps
A3: We thank the reviewer for requesting this clarification. The visual response maps in Figure 4 represent channel-averaged activations of intermediate vision-encoder features, rather than attention weights or feature norms.
Specifically, for each spatial location, we average the corresponding feature vector across channels, apply per-map min-max normalization, map the normalized values using the YlOrRd colormap, and bilinearly resize the resulting map to the input image resolution.
Importantly, the colors represent relative feature activations, not physical depth or attention probabilities. Since each map is independently normalized, absolute color intensities should not be directly compared across models. These visualizations are intended to illustrate changes in visual representations, rather than independently establish improved geometric understanding.
We have explicitly documented this computation and its interpretation in the revised Figure 4 caption and accompanying text. The figure now also includes manually drawn question-relevant regions and representative dataset questions with ground-truth labels; these annotations provide context rather than per-model prediction results.
Rebuttal by Authors (3/3)
W3 & Q3: Training Cost, Parameter Sensitivity, and Comparison with Lightweight Methods
A4: We thank the reviewer for raising this question regarding computational efficiency. We agree that the training cost of world-model supervision must be justified by its empirical benefits. Our paper provides controlled feature-distillation comparisons, training-cost analysis, and comparison with inference-time world-model reasoning. To further address this concern, we conducted a Gemma4 port/transfer evaluation of AdaptVis (ICML 2025), including its combination with GeoWorld-VLM.
(1) Existing comparisons and computational trade-offs. Tables 1 and 2 compare GeoWorld-VLM against task-only fine-tuning and four feature teachers (DINOv3, SigLIP2, RADIO, and VGGT) under matched configurations. GeoWorld-VLM achieves the highest overall mean accuracy across all four backbone–benchmark settings. Table 7 compares our method with MindJourney, an inference-time world-model reasoning approach. On matched 200-question subsets, GeoWorld-VLM outperforms FT+MindJourney by 12.0–21.5 percentage points, requiring 3–5 seconds per query versus 97–155 seconds for MindJourney.
We acknowledge the additional training computation. Appendix D.7 (Table 16) reports 5.3 GPU-hours for GeoWorld-VLM, versus 0.6 GPU-hours for task-only FT and 1.4 GPU-hours for DINOv3 under the evaluated EmbSpatial schedule. However, this cost is incurred only during training: the frozen teacher and alignment heads are removed afterward, preserving the original VLM inference architecture without additional inference-time computation.
(2) AdaptVis port/transfer evaluation on Gemma4. We added this comparison to Section 4.8 and documented calibration in Appendix D.8 (Tables 18 and 19). Hyperparameters are selected per model and dataset using validation data only. The appendix specifies candidate grids, accuracy/F1 selection rules, confidence gate, and final seed-42 settings. AdaptVis accuracies below are means over three calibration seeds; Table 19 reports only the seed-42 configuration. Evaluation uses the fixed 3,091-example legacy test partition without test-set tuning.
| Model | Standard (%) | + AdaptVis (%) | AdaptVis Gain |
|---|---|---|---|
| Base Gemma4 | 61.70 | 62.04 | +0.34 |
| GeoWorld-VLM | 65.45 | 66.39 | +0.94 |
GeoWorld-VLM + AdaptVis achieves 66.39%, outperforming Base Gemma4 + AdaptVis by 4.35 percentage points. AdaptVis also provides a larger incremental gain on GeoWorld-VLM (+0.94) than on Base Gemma4 (+0.34). These results show that our representation enhancement remains effective with inference-time attention adaptation, suggesting complementary benefits.
AdaptVis uses inference-time attention intervention, whereas GeoWorld-VLM enhances representations during training without changing inference. The methods can therefore be combined, with the combined setting retaining AdaptVis's inference-time intervention.
(3) Parameter sensitivity and comparison with SpatialRGPT. Appendix D.1-D.2 reports extraction-layer and denoising-step sensitivity. Layer 24 and two denoising steps were informed by prior observations, fixed before benchmark evaluation, and retained across both backbones and benchmarks. Subsequent ablations assess sensitivity rather than select the main configuration. These settings are teacher-specific, not universally optimal.
SpatialRGPT incorporates depth- and region-aware components through a different training and inference pipeline. Comparing its reported scores directly with ours would confound backbone architectures, auxiliary inputs, and supervision. GeoWorld-VLM instead retains the original single-image VLM inference interface without additional deployment modules. We do not report a matched SpatialRGPT experiment or claim superiority over it. Our controlled comparisons are the matched feature-teacher evaluations and Gemma4 AdaptVis evaluation in Appendix D.8.
Overall, we do not claim that world-model distillation is the cheapest adaptation strategy. Our results show that a one-time training investment improves spatial reasoning under the original inference architecture, outperforms matched feature-distillation baselines, and provides additional benefits alongside lightweight inference-time adaptation such as AdaptVis.
Rebuttal by Authors (1/3)
Dear Reviewer ttTK, We sincerely thank you for your careful evaluation and constructive feedback. We appreciate your thoughtful concerns regarding the methodological novelty of GeoWorld-VLM, the connection between generative world-model representations and spatial reasoning, and the computational trade-offs of our approach. We recognize that these aspects deserve clearer explanations and stronger empirical justification. In response, we have refined the positioning of our contributions, clarified the interpretation of our qualitative results, and provided additional experimental evidence, including a new comparison with AdaptVis. We hope these clarifications and results will address your main concerns and help you reassess the contributions and empirical significance of our work.
W1: Methodological Novelty Beyond Conventional Feature Distillation
A1: We thank the reviewer for raising this concern. We agree that feature distillation from foundation models is an established technique, and we do not claim novelty in feature matching itself. However, our contribution goes beyond replacing one feature teacher with another. As Reviewer mHKi independently recognized, the idea of eliciting spatial information from world models to enhance VLM spatial reasoning is "surprising and interesting." We would like to clarify the distinctive contributions of our approach.
(1) Geometry-oriented feature elicitation from world models. Unlike distillation from teachers without camera-conditioned video prediction (e.g., DINOv3 or VGGT), GeoWorld-VLM introduces a teacher-side conditioning strategy that combines sampled camera trajectories with a scene-stability prompt. This allows us to elicit viewpoint-conditioned predictive representations from a frozen generative world model, rather than directly distilling static image features. We then transfer these representations through post-projector alignment and interface preservation, while keeping the LLM frozen and introducing no additional inference-time computation.
(2) Empirical evidence beyond teacher replacement. Our controlled experiments demonstrate that both the choice of supervision source and the way its representations are elicited matter. Under matched training configurations, Tables 1 and 2 show that GeoWorld-VLM achieves the highest overall mean accuracy across all four backbone–benchmark settings compared with task-only fine-tuning and alternative feature teachers, including DINOv3, SigLIP2, RADIO, and VGGT. Furthermore, Table 3 shows that joint camera and prompt conditioning improves accuracy from 64.01% (I2V-only) to 65.45%, while keeping the teacher architecture and student training recipe unchanged. This demonstrates that our contribution is not merely the selection of a different teacher, but also a specific strategy for eliciting and transferring its predictive representations.
We have sharpened this positioning in the Introduction and Section 3, emphasizing the combination of camera- and prompt-conditioned predictive feature elicitation, post-projector transfer, and interface preservation, rather than presenting feature matching itself as a new technique.