acceptodds
Under review as a conference paper at ICLR 2027

GeoWorld-VLM: Geometry from World Models for Vision-Language Models

Abstract

Modern Vision-Language Models (VLMs) achieve strong semantic recognition, yet remain brittle on elementary spatial relations such as left of, on, behind, and between. We introduce GeoWorld-VLM, which transfers predictive representations from a frozen camera-conditioned video world model into a VLM's existing visual pathway to improve single-image spatial reasoning. From each static training image, a sampled camera trajectory and fixed scene-stability prompt produce viewpoint-conditioned teacher features without real multi-view observations. Post-projector feature alignment transfers this supervision, while a preservation anchor constrains changes to the visual interface consumed by the frozen language model. Only the image encoder, multimodal projector, and student alignment head are updated; the teacher and alignment heads are removed after training. Across two VLM backbones on What'sUp+VSR and EmbSpatial, GeoWorld-VLM improves over matched task-only fine-tuning by – percentage points and achieves the highest mean accuracy among the evaluated feature teachers. These results demonstrate that predictive world-model representations can improve single-image spatial reasoning at no extra inference cost.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, spatial reasoning, world models, knowledge distillation, visual representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ttTkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GeoWorld-VLM, which uses features from a frozen, camera-conditioned world model to improve a VLM’s spatial understanding. The teacher is used only during training, leaving the original inference architecture unchanged.

Critical strengths

The paper is clearly structured and easy to read. The experiments cover multiple benchmarks, including What’sUp, VSR, and EmbSpatial-Bench, and two VLM families. The paper provides extensive ablations and reports training costs, generalization results, and limitations.

Critical weaknesses
  1. Limited methodological novelty. There are already some distillation methods, like VGGT and DiNO, listed in the paper. And using such foundation models to supervise VLMs is not a novel idea.

  2. Mismatch with claim and motivation. The author says that they would use a world model (basically a generative objective) to supervise VLMs for better spatial understanding. However, there is no clear evidence to show why a generative objective can help with understanding spatial relationships (Only citing some video generative papers is limited to support this). Table 3 shows that predicted frames even defaced the elements we really care about, like the lamp and vehicle. The author should explain those.

  3. Training cost and parameter sensitivity. Appendix D shows that performance depends on the extraction layer and denoising stage. Given the additional training cost, the benefit over simpler approaches needs clearer justification, especially compared with lightweight methods such as AdaptVis(ICML 2025) and SpatialRGPT(NeurIPS 2024).

Questions for authors
  1. The comparison in Figure 3 does not explicitly explain why a generated model or objective is useful for understanding location. I expected the authors to provide more direct evidence that world-model supervision improves spatial understanding or depth representations.
  2. How are the visual feature response maps computed? Are they feature norms, attention weights, or something else?
  3. Do the improvements justify the additional training cost compared with simpler methods?
7 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Dear Reviewer ttTK, We sincerely thank you for your careful evaluation and constructive feedback. We appreciate your thoughtful concerns regarding the methodological novelty of GeoWorld-VLM, the connection between generative world-model representations and spatial reasoning, and the computational trade-offs of our approach. We recognize that these aspects deserve clearer explanations and stronger empirical justification. In response, we have refined the positioning of our contributions, clarified the interpretation of our qualitative results, and provided additional experimental evidence, including a new comparison with AdaptVis. We hope these clarifications and results will address your main concerns and help you reassess the contributions and empirical significance of our work.


W1: Methodological Novelty Beyond Conventional Feature Distillation

A1: We thank the reviewer for raising this concern. We agree that feature distillation from foundation models is an established technique, and we do not claim novelty in feature matching itself. However, our contribution goes beyond replacing one feature teacher with another. As Reviewer mHKi independently recognized, the idea of eliciting spatial information from world models to enhance VLM spatial reasoning is "surprising and interesting." We would like to clarify the distinctive contributions of our approach.

(1) Geometry-oriented feature elicitation from world models. Unlike distillation from teachers without camera-conditioned video prediction (e.g., DINOv3 or VGGT), GeoWorld-VLM introduces a teacher-side conditioning strategy that combines sampled camera trajectories with a scene-stability prompt. This allows us to elicit viewpoint-conditioned predictive representations from a frozen generative world model, rather than directly distilling static image features. We then transfer these representations through post-projector alignment and interface preservation, while keeping the LLM frozen and introducing no additional inference-time computation.

(2) Empirical evidence beyond teacher replacement. Our controlled experiments demonstrate that both the choice of supervision source and the way its representations are elicited matter. Under matched training configurations, Tables 1 and 2 show that GeoWorld-VLM achieves the highest overall mean accuracy across all four backbone–benchmark settings compared with task-only fine-tuning and alternative feature teachers, including DINOv3, SigLIP2, RADIO, and VGGT. Furthermore, Table 3 shows that joint camera and prompt conditioning improves accuracy from 64.01% (I2V-only) to 65.45%, while keeping the teacher architecture and student training recipe unchanged. This demonstrates that our contribution is not merely the selection of a different teacher, but also a specific strategy for eliciting and transferring its predictive representations.

We have sharpened this positioning in the Introduction and Section 3, emphasizing the combination of camera- and prompt-conditioned predictive feature elicitation, post-projector transfer, and interface preservation, rather than presenting feature matching itself as a new technique.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W2 & Q1: Why Generative World-Model Supervision Benefits Spatial Understanding

A2: We appreciate the reviewer's request for a clearer explanation of why generative world-model supervision can benefit spatial understanding. We agree that high-quality video generation does not necessarily imply accurate spatial reasoning, and that qualitative generation examples alone are insufficient to establish this connection. However, we would like to clarify an important distinction: GeoWorld-VLM transfers intermediate predictive representations from a generative world model, rather than supervising the VLM with generated pixels or video reconstruction objectives.

(1) Why can generative representations provide useful spatial supervision? Camera-conditioned video prediction requires modeling how a scene changes under viewpoint transformations while maintaining its underlying spatial configuration. This objective encourages the world model to encode spatial cues such as relative object placement, depth, and occlusion, which are relevant to spatial relational understanding. Our method specifically elicits such predictive representations using sampled camera trajectories and a scene-stability prompt, and transfers them into the VLM's visual pathway through feature alignment. We do not assume that generative objectives guarantee geometrically accurate representations; rather, we investigate whether these representations provide useful supervision for downstream spatial reasoning.

(2) Quantitative evidence of predictive representation transfer. Our existing ablations provide evidence beyond qualitative video generation. Table 3 shows that jointly incorporating camera and prompt conditioning improves spatial QA accuracy from 64.01% (I2V-only) to 65.45%, with the same world-model architecture and student training configuration. Furthermore, Section 4.9 (Table 8) demonstrates that aggregating predictive features across nine frames achieves 65.45%, compared with 61.92% and 63.30% when using only the first or last frame, respectively. These results suggest that the predictive trajectory provides useful information beyond an individual frame representation.

Importantly, our evaluation also measures spatial reasoning directly. With the language model frozen, GeoWorld-VLM improves over matched task-only fine-tuning by 1.53–3.52 percentage points across four backbone–benchmark settings. On Gemma4/EmbSpatial, the improvements on close and far relations reach +3.99 and +4.10 points, respectively. These controlled comparisons demonstrate that the transferred representations are more effective for the evaluated spatial reasoning tasks, although they do not independently establish metric depth accuracy.

(3) Clarifying viewpoint-dependent changes in generated frames. We have revised the Figure 3 caption to identify the vehicle example's camera motion as a backward movement with a left turn. Viewpoint changes can alter visibility, so the vehicle's disappearance alone does not establish a spatial-consistency failure. For the lamp example, the displayed montage alone does not determine whether the apparent change is a viewpoint effect or a synthesis error. We therefore do not use these decoded examples as evidence of geometrically accurate generation. The revised caption distinguishes illustrative generated frames from the intermediate predictive features transferred during training; the conditioning and temporal-feature ablations, together with matched spatial QA evaluations, provide the quantitative evidence relevant to our claim.


Q2: Computation of Visual Feature Response Maps

A3: We thank the reviewer for requesting this clarification. The visual response maps in Figure 4 represent channel-averaged activations of intermediate vision-encoder features, rather than attention weights or feature norms.

Specifically, for each spatial location, we average the corresponding feature vector across channels, apply per-map min-max normalization, map the normalized values using the YlOrRd colormap, and bilinearly resize the resulting map to the input image resolution.

Importantly, the colors represent relative feature activations, not physical depth or attention probabilities. Since each map is independently normalized, absolute color intensities should not be directly compared across models. These visualizations are intended to illustrate changes in visual representations, rather than independently establish improved geometric understanding.

We have explicitly documented this computation and its interpretation in the revised Figure 4 caption and accompanying text. The figure now also includes manually drawn question-relevant regions and representative dataset questions with ground-truth labels; these annotations provide context rather than per-model prediction results.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W3 & Q3: Training Cost, Parameter Sensitivity, and Comparison with Lightweight Methods

A4: We thank the reviewer for raising this question regarding computational efficiency. We agree that the training cost of world-model supervision must be justified by its empirical benefits. Our paper provides controlled feature-distillation comparisons, training-cost analysis, and comparison with inference-time world-model reasoning. To further address this concern, we conducted a Gemma4 port/transfer evaluation of AdaptVis (ICML 2025), including its combination with GeoWorld-VLM.

(1) Existing comparisons and computational trade-offs. Tables 1 and 2 compare GeoWorld-VLM against task-only fine-tuning and four feature teachers (DINOv3, SigLIP2, RADIO, and VGGT) under matched configurations. GeoWorld-VLM achieves the highest overall mean accuracy across all four backbone–benchmark settings. Table 7 compares our method with MindJourney, an inference-time world-model reasoning approach. On matched 200-question subsets, GeoWorld-VLM outperforms FT+MindJourney by 12.0–21.5 percentage points, requiring 3–5 seconds per query versus 97–155 seconds for MindJourney.

We acknowledge the additional training computation. Appendix D.7 (Table 16) reports 5.3 GPU-hours for GeoWorld-VLM, versus 0.6 GPU-hours for task-only FT and 1.4 GPU-hours for DINOv3 under the evaluated EmbSpatial schedule. However, this cost is incurred only during training: the frozen teacher and alignment heads are removed afterward, preserving the original VLM inference architecture without additional inference-time computation.

(2) AdaptVis port/transfer evaluation on Gemma4. We added this comparison to Section 4.8 and documented calibration in Appendix D.8 (Tables 18 and 19). Hyperparameters are selected per model and dataset using validation data only. The appendix specifies candidate grids, accuracy/F1 selection rules, confidence gate, and final seed-42 settings. AdaptVis accuracies below are means over three calibration seeds; Table 19 reports only the seed-42 configuration. Evaluation uses the fixed 3,091-example legacy test partition without test-set tuning.

ModelStandard (%)+ AdaptVis (%)AdaptVis Gain
Base Gemma461.7062.04+0.34
GeoWorld-VLM65.4566.39+0.94

GeoWorld-VLM + AdaptVis achieves 66.39%, outperforming Base Gemma4 + AdaptVis by 4.35 percentage points. AdaptVis also provides a larger incremental gain on GeoWorld-VLM (+0.94) than on Base Gemma4 (+0.34). These results show that our representation enhancement remains effective with inference-time attention adaptation, suggesting complementary benefits.

AdaptVis uses inference-time attention intervention, whereas GeoWorld-VLM enhances representations during training without changing inference. The methods can therefore be combined, with the combined setting retaining AdaptVis's inference-time intervention.

(3) Parameter sensitivity and comparison with SpatialRGPT. Appendix D.1-D.2 reports extraction-layer and denoising-step sensitivity. Layer 24 and two denoising steps were informed by prior observations, fixed before benchmark evaluation, and retained across both backbones and benchmarks. Subsequent ablations assess sensitivity rather than select the main configuration. These settings are teacher-specific, not universally optimal.

SpatialRGPT incorporates depth- and region-aware components through a different training and inference pipeline. Comparing its reported scores directly with ours would confound backbone architectures, auxiliary inputs, and supervision. GeoWorld-VLM instead retains the original single-image VLM inference interface without additional deployment modules. We do not report a matched SpatialRGPT experiment or claim superiority over it. Our controlled comparisons are the matched feature-teacher evaluations and Gemma4 AdaptVis evaluation in Appendix D.8.

Overall, we do not claim that world-model distillation is the cheapest adaptation strategy. Our results show that a one-time training investment improves spatial reasoning under the original inference architecture, outperforms matched feature-distillation baselines, and provides additional benefits alongside lightweight inference-time adaptation such as AdaptVis.

Reviewer_mHKiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to enhance the spatial understanding of vision-language models (VLMs) by distilling representations from world models into the visual representations in VLMs. To handle this task, the paper proposes the method called GeoWorld-VLM which aligns the features from world models with the visual features in VLMs. Specifically, the world model features are elicited by sampled camera trajectory and text prompts. The VLMs learn the additional spatial information feature alignment from world models during finetuning, so it adds no additional overhead for inference. Experimental results demonstrate that the proposed method achieves superior performance and the ablation study shows the effectiveness of the parts in the proposed solution.

Critical strengths
  • The paper is clearly presented, and the method is a simple but effective solution. I find it easy to understand and can be generally applicable to the more advanced VLMs and world models in the future. Also, the idea of eliciting spatial information from world models to enhance the spatial reasoning capability of VLMs is surprising and interesting to me.

  • The ablation studies and analysis are comprehensive and can serve as valuable guidelines for users or future work for setting adoption of leveraging world models to inject spatial information into VLMs.

Critical weaknesses
  • The preparation of the appendix seems to be careless. On Page 19, Section F, there are two completely empty sections F.1 and F.2. On Page 15, the appendix in Section C "STD of 3 seeds" also looks like a very rough draft. I acknowledge the authors for having abundant details and ablations in the appendix to better support the soundness, reproducibility and provide helpful analysis for the method, but the paper cannot be published without fixing these obvious issues for the paper rigorousness.

  • The performance of the proposed method (in Tables 1 and 2) are not consistently better than the other feature alignment/distillation strategies (e.g., using SigLIP2, RADIO, VGGT), although it indeed outperforms the base VLM models before any finetuning on distillation. Also, in the visualizations of the visual features, the paper does not really point the audience to where to focus on for the comparison between the visual features. For example, in Figure 4, it is unexplained why the proposed method produces better features and where to look at.

The above two points are the critical weaknesses. The other minor weaknesses include that it is kind of unjustified why in the world-to-VLM alignment process, the teacher model would need a fixed and randomly initialized teacher head, instead of directly projecting the student representation to the teacher representation space (also asked in the following Questions part).

Questions for authors
  • Why do you think the feature map gets improved from the visualization in Figure 4? From the visualization, it seems that the feature has been enhanced with more depth-relevant information. However, in some cases, it seems that the closer parts have lighter visualization, while some are darker. Is it a visualization issue, or indeed the value of close/far regions can either be lighter or darker? Basically, how should we read these visualizations to know which ones are better?

  • In the world-to-VLM alignment design, why do we need to have the fixed randomly initialized teacher head to project the teacher model (the world model feature) to another feature space, instead of simply having a projector to map the VLM model representation to the world model feature space (just like the projector in VLMs that directly maps the visual features to the language token space)?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

Dear Reviewer mHKi, We sincerely thank you for your thoughtful and constructive feedback. We greatly appreciate your recognition of our interesting and effective approach, clear presentation, and comprehensive ablation studies, as well as the potential applicability of our method to future VLMs and world models. We hope the following clarifications can adequately address your valuable concerns.


W1: Appendix Formatting and Completeness

A1: Thank you for carefully pointing out these presentation issues. We would like to clarify that the qualitative results for Sections F.1 and F.2 were already included in the original submission. However, LaTeX float placement caused these tables to appear away from their subsection headings in the PDF compiled in Overleaf, creating the misleading appearance of empty sections. We acknowledge that this layout made the appendix difficult to follow. We have now corrected the placement so that each subsection heading is immediately followed by the opening rows of its corresponding case-study table (Tables 21 and 22 in the revised manuscript). The tables can continue across pages with repeated column headers, and images, questions, and predictions are now top-aligned within each row. We have rewritten Appendix C, now titled "Variation across Three Independent Seeds," to explain the reporting protocol and replace the draft-like wording. Table 6 in the main paper reports mean +/- standard deviation over three independent training seeds, and Table 10 in Appendix C reports the corresponding standard deviations in percentage points. We have clarified that these summarize variation across training runs and do not by themselves establish statistical significance.


W2 & Q1: Feature Teacher Comparison and Visualization A2: We sincerely appreciate the reviewer's careful examination of our results and, particularly, your insightful observation regarding the apparent depth-related patterns in Figure 4. We agree that our original presentation did not sufficiently explain how these visualizations should be interpreted.

(1) Clarifying the feature visualization. The heatmaps show channel-mean activations of intermediate visual features, not depth estimates or attention weights. At each spatial location, we average features across channels, apply per-map min-max normalization and a yellow-to-red (YlOrRd) colormap, then bilinearly resize to the input resolution. We have documented this procedure in the revised Figure 4 caption. The colors represent relative activations, not physical depth: nearby and distant regions can appear lighter or darker because activation intensity has no predefined monotonic relationship with distance. Since each map is normalized independently, absolute intensities are not comparable across models. Visual differences alone do not establish better representations; the boxes guide inspection, while matched spatial QA accuracy measures downstream usefulness.

(2) Interpreting representation improvements. We agree that visual differences alone cannot establish superior geometric understanding. In the revised Figure 4, manually drawn blue boxes identify (a) the baby, (b) the motorcycle, (c) the man, and (d) the background benches. Each row includes a representative dataset question and ground-truth answer (right, bottom, right, and top, respectively). In (d), the relevant benches are above the child. Appendix F.3 lists source image IDs and associated questions. These annotations guide spatial comparisons across models; the answers are dataset labels, not model predictions. Figure 4 illustrates representation changes qualitatively, while Tables 1 and 2 provide quantitative evidence of downstream effectiveness.

(3) Comparison with alternative teachers. We acknowledge that our method does not consistently outperform every teacher on every individual relation or subset. Nevertheless, Tables 1 and 2 show that GeoWorld-VLM achieves the highest overall mean accuracy in all four backbone–benchmark settings under the matched training recipe. In particular, on Gemma4/What'sUp+VSR, our method achieves 65.45%, compared with 64.40% for DINOv3, 63.88% for RADIO, and 63.65% for VGGT. We interpret these results as evidence of the effectiveness of predictive world-model supervision at the aggregate level, rather than uniform superiority across all spatial relations. We thank the reviewer again for highlighting this ambiguity, which has helped us identify a concrete improvement to the presentation and interpretation of our results.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

W3 & Q2: Necessity of the Fixed Random Teacher Head

A3: We thank the reviewer for this insightful question. We have moved the existing projection-head ablation from Appendix D.4 into Section 4.6 (Table 5) of the main paper to make this design choice easier to assess. It compares our two-sided design (trainable student head + fixed random teacher head) with the student-only variant that removes the teacher projection.

Backbone / BenchmarkStudent-only ProjectionOurs (Two-sided Projection)Improvement
Gemma4 / What'sUp+VSR63.1665.45+2.29
Gemma4 / EmbSpatial76.8378.78+1.95
InternVL3.5-2B / What'sUp+VSR59.1861.66+2.48
InternVL3.5-2B / EmbSpatial78.6079.70+1.10

As shown above, our two-sided projection consistently outperforms the student-only alternative across both VLM backbones and both benchmarks, with improvements of 1.10–2.48 percentage points.

The necessity of the teacher-side projection relative to the evaluated student-only alternative is directly supported by Table 5: removing it reduces accuracy by 1.10–2.48 percentage points in all four backbone–benchmark settings. The fixed, randomly initialized teacher MLP maps high-dimensional world-model features into a shared 512-dimensional alignment space, allowing the trainable student head to learn against a consistent projected target rather than directly matching the raw teacher space. Thus, the ablation provides empirical justification for our two-sided projection design; it does not imply that random initialization is uniquely optimal among all possible teacher mappings.

We have made this ablation and its motivation more prominent in Sections 3.3 and 4.6. Both alternatives use frozen teacher features, so a stationary target is not unique to the two-sided design. The experiment supports the evaluated projection design without establishing that random initialization is uniquely optimal.

AuthorsOpenReview ↗

Thanks to Reviewer mHKi

Thanks to Reviewer mHKi

Dear Reviewer mHKi,

We sincerely thank you again for your thoughtful review and constructive feedback. We particularly appreciate your recognition of the interesting motivation, effective approach, clear presentation, and comprehensive ablation studies.

Your suggestions have helped us improve the manuscript substantially. We have addressed the appendix formatting issues, clarified Figure 4, moved the projection-head ablation into the main paper, and added quantitative depth and surface-normal probes.

We sincerely hope our responses and additional experiments have addressed your concerns. Please kindly let us know if any concerns remain or if further clarification would be helpful. We would greatly appreciate your feedback!

Best regards,

GeoWorld-VLM Authors

Reviewer_FhaoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GeoWorld-VLM distills features from a camera-conditioned video world model into a VLM’s visual pathway using feature alignment and a preservation loss, while keeping the language model frozen. It improves spatial QA on two backbones without additional inference-time computation.

Critical strengths
  1. The method is practical and retains the original inference architecture.
  2. Comparisons against multiple feature teachers, including VGGT, under a shared training recipe and across three seeds provide useful evidence of effectiveness.
Critical weaknesses
  1. Limited technical novelty and mechanistic evidence. The contribution appears incremental, primarily involving teacher selection and adaptation design within an established feature-distillation framework. The distinctive geometric value of world-model supervision remains insufficiently demonstrated. Figure 4 offers limited support: the response-map construction and normalization are insufficiently explained, and the examples lack corresponding questions and predictions. These visual differences do not establish improved geometric understanding; quantitative geometric probes would be more informative.

  2. Incomplete attribution of the gains. Table 4 reports 62.71 for task-only fine-tuning, 63.20 with alignment alone, and 65.45 for the full method, but omits preservation without alignment. This missing control prevents isolating the contribution of world-model supervision beyond preservation and their interaction.

  3. Uneven generalization and capability retention. Improvements vary across transfer settings, performance falls below task-only fine-tuning on SAT ego-motion questions, and general multimodal accuracy can degrade substantially after adaptation. These results limit the evidence for broadly transferable spatial reasoning.

Questions for authors
  1. What evidence shows that the gains arise from distinctive geometric information in the world-model features? Could you clarify how the response maps in Figure 4 are computed and normalized, and provide quantitative geometric probes to support this interpretation?

  2. Could you add a preservation-only baseline without teacher alignment, completing the 2×2 ablation? How much improvement does world-model alignment provide over this baseline across both backbones?

  3. What explains the regression on SAT ego-motion questions and the dataset-dependent degradation on MMStar? Can these failures be mitigated without sacrificing the main spatial QA improvements?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

Dear Reviewer Fhao, We sincerely thank you for your careful review and constructive feedback. We appreciate your recognition of our practical design, which introduces no additional inference-time computation, as well as the rigorous comparisons with multiple feature teachers across three random seeds. We understand your concerns regarding the novelty and geometric evidence of our approach, the attribution of performance gains, and the generalization of spatial capabilities. We would like to clarify several important aspects of our contribution and provide additional analyses to address these concerns.


W1 & Q1: Technical Novelty and Evidence of Spatial Representation Transfer

A1: We appreciate the request for stronger evidence about the role of world-model representations. Our contribution goes beyond selecting a world model as a feature teacher. As Reviewer mHKi noted, eliciting spatial information from world models is “surprising and interesting”; our key contribution is how we elicit spatially useful predictive representations and transfer them into a VLM’s existing visual pathway. We condition the teacher on sampled camera trajectories and a scene-stability prompt, encouraging predictions under viewpoint changes while preserving scene layout. We then transfer these features through post-projector alignment and interface preservation, without modifying the frozen LLM or adding inference-time computation.

(1) Evidence that feature elicitation matters. Table 3 controls the teacher architecture and student training recipe while varying teacher conditioning: I2V-only achieves 64.01%, I2V+camera 63.16%, I2V+prompt 62.61%, and joint camera+prompt conditioning 65.45%. The joint setting improves by 1.44 points over I2V-only, showing that world-model supervision depends on how features are elicited, not teacher selection alone. This ablation isolates conditioning effects under one fixed training setup; it does not independently measure geometry. Table 8 further shows that aggregating predictive features across a trajectory matters: nine-frame mean pooling reaches 65.45%, versus 61.92% using only the first frame and 63.30% using only the last. This temporal ablation tests predictive aggregation in the evaluated setting.

(2) Quantitative evidence from spatial reasoning. Our goal is to improve spatial relational reasoning, not explicitly reconstruct depth maps or camera poses. We use the frozen LLM as a fixed, task-grounded functional readout of the adapted visual tokens. Because it is the VLM’s original language module and remains unchanged, this tests whether those tokens support better spatial judgments without training a separate probing head. Rather than introducing a readout trained for this evaluation, we assess whether the original VLM can use the adapted representations through its existing inference interface. With task supervision and student configurations matched, GeoWorld-VLM improves over task-only fine-tuning by 1.53–3.52 points across two VLM backbones and two spatial benchmarks. On Gemma4/EmbSpatial, gains for close and far relations are +3.99 and +4.10 points. These are quantitative, task-relevant measures of spatial information usable by the original LLM, rather than visual impressions from feature maps. We clarified in Section 4.2 that this fixed readout measures usefulness for spatial QA, not independent geometric accuracy. A separately trained depth or camera-pose probe could provide complementary evidence, but we did not conduct one. Thus, our results do not establish metric-depth accuracy, explicit 3D reconstruction, or a uniquely identified geometric transfer mechanism. Our supported conclusion is that the elicited predictive representations improve spatial QA through the existing visual-language interface.

(3) Figure 4 clarification. We agree that Figure 4 needed a clearer explanation. Its response maps are qualitative illustrations of visual feature changes, not independent proof of improved geometric understanding. For each spatial location, we average intermediate vision-encoder features across channels, then apply per-map min-max normalization, map values with the YlOrRd colormap, and bilinearly resize to input resolution. Each map is normalized independently, so colors show relative activations, not physical depth or attention probabilities, and should not be compared as absolute intensities across models. We clarified the computation and interpretation in the revised caption and added manually drawn blue boxes, representative dataset questions, and ground-truth answers for the baby, motorcycle, man, and background benches. Appendix F.3 lists the source image IDs and associated annotations. The displayed answers are dataset labels, not recovered model predictions. The additions direct readers to relevant regions while keeping the qualitative visualization distinct from the quantitative spatial-QA evidence.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

W2 & Q2: Disentangling the Contributions of Alignment and Preservation

A2: We thank the reviewer for identifying the missing preservation-only baseline. We have now completed this control with three random seeds on Gemma4 / What'sUp+VSR, enabling a full 2×2 ablation in this setting and measuring the contribution of world-model alignment beyond preservation.

(1) Existing evidence. As reported in Table 4, on Gemma4 / What'sUp+VSR, task-only FT achieves 62.71%, adding world-model alignment without preservation improves accuracy to 63.20% (+0.49), and the complete GeoWorld-VLM achieves 65.45% (+2.74 over FT). Notably, comparing the latter two configurations already demonstrates that adding preservation in the presence of world-model alignment provides a further +2.25-point improvement.

(2) Completing the missing control. We additionally evaluated task supervision + preservation loss (without alignment) under the same training configuration. Averaged over three random seeds, this preservation-only baseline achieves 63.37%:

ConfigurationTask LossAlignmentPreservationAccuracy (%)
Task-only FT✓✗✗62.71
Alignment only✓✓✗63.20
Preservation only (New)✓✗✓63.37
GeoWorld-VLM (Full)✓✓✓65.45

This completes the factorial comparison for Gemma4 / What'sUp+VSR. Preservation alone improves task-only FT from 62.71% to 63.37% (+0.66), whereas adding world-model alignment on top of preservation further improves accuracy from 63.37% to 65.45% (+2.08). Conversely, alignment alone gives +0.49 over FT, while adding preservation on top of alignment gives +2.25. Thus, preservation contributes to the gain but cannot alone explain it. The difference of differences is +1.59 points, indicating a positive interaction in this mean-level comparison; this is not a statistical interaction test or evidence across backbones. The new preservation-only control is currently limited to this Gemma4 setting. The four-setting matched-teacher comparisons do not substitute for this missing factorial control. We have incorporated the completed ablation into Section 4.5 (Table 4) and clarified the complementary roles of the objectives: preservation constrains drift of the pretrained visual interface, while world-model alignment supplies predictive representation supervision. Their combination achieves the strongest mean accuracy in this controlled setting.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

W3 & Q3: Cross-Benchmark Generalization, Ego-Motion Reasoning, and Capability Retention

A3: We thank the reviewer for noting varied performance across spatial and general multimodal evaluations. GeoWorld-VLM leads the compared feature-distillation baselines in both transfer directions, while its gains vary by capability.

(1) Cross-benchmark generalization. While the magnitude of improvement differs across transfer directions, GeoWorld-VLM achieves the highest accuracy among all compared methods in both settings, as reported in the bidirectional transfer table that we have moved into the main paper (Table 9; protocol in Appendix D.5):

Training → EvaluationFTDINOv3VGGTGeoWorld-VLM
What'sUp+VSR → EmbSpatial56.0458.3560.3061.77
EmbSpatial → What'sUp+VSR62.1962.3260.1662.42

From What'sUp+VSR to EmbSpatial, GeoWorld-VLM exceeds FT, DINOv3, and VGGT by +5.73, +3.42, and +1.47 points. In reverse, it leads DINOv3 by only +0.10. The asymmetry is in gain magnitude, not baseline ranking: GeoWorld-VLM also has the highest mean across the four backbone–benchmark settings (Table 6). These results support the framework but do not imply large or statistically significant gains in every setting.

(2) SAT ego-motion regression. We acknowledge that GeoWorld-VLM underperforms both FT and DINOv3 on SAT ego-motion questions (52.43 vs. 56.22 and 63.78, respectively), despite achieving the highest mean accuracy on the non-ego-motion split (58.24 vs. 58.03 for FT and 57.41 for DINOv3).

We believe this distinction is closely related to the different requirements of static spatial-layout reasoning and dynamic ego-motion reasoning. Our current approach aggregates predictive world-model features through temporal mean pooling, which encourages the extraction of stable scene-layout representations but may suppress temporal ordering and motion-direction information essential for ego-motion reasoning.

Importantly, our method primarily targets spatial relational reasoning from a single image, rather than explicit dynamic motion understanding. We have clarified this single-image spatial-reasoning scope and the ego-motion limitation in Section 4.9 and Appendix E. The temporal-pooling explanation remains a hypothesis rather than an experimentally isolated causal conclusion.

(3) MMStar degradation and capability retention. We agree that general multimodal performance can degrade after spatial adaptation. However, trade-offs between task-specific adaptation and previously acquired capabilities are a well-recognized challenge in modern LLM/VLM post-training, commonly studied as capability interference or catastrophic forgetting. Importantly, our MMStar results are not uniformly negative. As reported in Appendix D.6:

BackboneSpatial TrainingBaseFTGeoWorld-VLM
Gemma4What'sUp+VSR50.6150.6947.66
Gemma4EmbSpatial50.6150.8752.17
InternVL3.5-2BWhat'sUp+VSR55.3751.8248.01
InternVL3.5-2BEmbSpatial55.3750.5255.03

In particular, after EmbSpatial training, GeoWorld-VLM improves MMStar over task-only FT by +1.30 points on Gemma4 and +4.51 points on InternVL3.5-2B, indicating that degradation is not an inevitable consequence of our world-model supervision. Rather, the observed retention behavior depends substantially on the adaptation dataset.

The existing experiments do not establish a mitigation that eliminates the SAT ego-motion regression or the MMStar degradation while retaining the main spatial QA gains. We have stated the capability-retention limitation explicitly. The MMStar results characterize dataset-dependent retention under the evaluated adaptation recipe; they do not constitute a controlled mitigation study.

We emphasize that our preservation objective is designed to constrain visual-interface drift, rather than guarantee the complete preservation of all pretrained multimodal capabilities. We have made this distinction explicit in Section 3.4 and in the MMStar discussion in Section 4.9 and Appendix D.6. Overall, our results demonstrate consistent improvements over matched feature teachers in the evaluated spatial reasoning and cross-benchmark settings, while identifying ego-motion reasoning and dataset-dependent multimodal retention as meaningful boundaries of the current approach.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.