acceptodds
Under review as a conference paper at ICLR 2027

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

10 views

Abstract

Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models recover this structure from RGB, but existing methods either fuse the estimated features with visual tokens and learn to use them from question answering alone, or align VLM latents with geometry features. Capturing geometry, however, does not ensure its use: latents aligned with geometry features can be zeroed without lowering accuracy. We introduce georeader, a framework that learns geometric representations for the VLM that reads them. First, we train lightweight Geometric Readers to produce continuous input tokens, supervising them with geometry descriptions decoded by a frozen VLM, so that the Readers express geometry in a form the target model can already read. We then freeze the Readers to preserve this interface while fine-tuning the VLM on spatial QA, with geometry injection conditioned on the question. Geometry descriptions serve only as training targets: at inference, the VLM answers directly from the continuous geometric tokens without generating an intermediate geometry chain of thought. With Qwen3-VL-4B and geometry predicted from RGB images, georeader reaches on MindCube-Tiny and, with a multi-source QA mixture, an average of across six additional spatial benchmarks. These results suggest that the most useful geometric representation is the one the receiving model can use, not necessarily the one that reconstructs geometry most accurately.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vlm, spatial understanding, latent geometry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UXxgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how geometry foundation models should be connected to VLMs for spatial reasoning. The authors argue that existing routes—QA-only fusion of geometry features and alignment of VLM latents to geometry features—can capture geometry without ensuring that the answering VLM actually uses it. They provide a diagnostic showing that zeroing 3DThinker latents leaves MindCube accuracy essentially unchanged and that answer tokens barely attend to them. At inference, the VLM answers directly from continuous geometric tokens without generating geometry CoT. Experiments across seven spatial benchmarks and three VLM families, plus ablations, geometry interventions, attention analysis, and failure cases, support the design.

Critical strengths
  1. The paper moves the objective from “reconstruct/align geometry accurately” to “learn a geometric interface the receiving VLM can actually read and use.” The diagnostic that aligned latents can be zeroed without accuracy loss is a strong motivation, and the two-stage separation—adapt geometry to the VLM, then freeze the interface and learn to answer—is conceptually clean and practically relevant for embodied/autonomous systems.

  2. GeoReader reaches 94.3 on MindCube-Tiny and 70.6 average across six benchmarks, with gains across Qwen2.5-VL, Qwen3-VL, and Qwen3.5 families. The ablations isolate readability pretraining, pair supervision, Reader freezing, and question-conditioned injection, while geometry interventions and attention analysis provide evidence that the model actually uses the supplied geometry rather than ignoring it.

Critical weaknesses
  1. Stage 2 uses a multi-source QA collection derived from VSI-style, SpatialLadder, SPAR, ViewSpatial, MMSI, and CV-Bench-style data, while evaluation includes these benchmarks. The authors state that official held-out test questions are excluded and training scenes do not overlap with evaluation scenes, which is reassuring, but the paper should more explicitly document question-generation/split boundaries and whether all compared baselines received comparable QA data and tuning budgets. This matters for interpreting the six-benchmark averages, though the controlled MindCube ablations and Table 3 mitigate the concern.

  2. The 3DThinker comparison partly relies on a released checkpoint and cited numbers. Table 4 Panel B includes adapted VLM-3R/SpaceMind-style controls, which is good, but a same-backbone, same-QA-data, same-tuning-budget feature-alignment control would make the central “readability vs. alignment” claim more directly supported.

Questions for authors
  1. Please provide the exact split protocol between the multi-source Stage 2 QA collection and the six evaluation benchmarks, including whether any training questions are generated from the same benchmark scenes/templates as the test questions. Also clarify whether all baselines were trained/tuned with comparable QA data and budgets.

  2. Stage 1 trains Readers with annotation-derived geometry, while main results use DA3-predicted geometry. How large is the domain gap, and can you quantify Reader sensitivity to pose/depth estimation errors beyond the aggregate results in Table 15?

  3. Each view contributes one geometric token. Is there an ablation on the number of geometric tokens per view or a discussion of the information bottleneck? Also, after Stage 2 fine-tunes the VLM, how stable is the frozen Reader interface?

4 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer UXxg,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 4793

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer UXxg,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 4793

Reviewer_UXxgOpenReview ↗

partially addressed my concerns

Thanks for the authors' responses and additional experiments. They have partially addressed my concerns and improve the experimental completeness of the paper.

AuthorsOpenReview ↗

Thank You for Your Response

Dear Reviewer UXxg,

Thank you for taking the time to read our rebuttal and assess the additional experiments. We appreciate your positive assessment of GeoReader and your rating in favor of acceptance.

We are glad that the new experiments addressed some of your concerns and made the evaluation more complete. If any concerns remain, we would be happy to discuss them here.

Thank you again for your time and feedback!

Warm regards,

The Authors of Submission 4793

Reviewer_h6rROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new paradigm for injecting geometric information from pretrained models into VLM for better spatial reasoning. Previous methods leverage geometric features as additional modality for VLM or by aligning the latent features, which cannot be directly used by VLM so that the performance will be limited. The authors propose a two-stage training, where the first stage trains a Geometric Reader to enable VLM to describe the geometric properties with extracted tokens and the second stage trains VLM to leverage these tokens to answer spatial reasoning questions. The performance of Qwen-VL trained in this two-stage manner achieves the highest on different benchmarks.

Critical strengths
  • Different from previous paradigms, the proposed training framework makes the extracted geometric tokens readable by VLMs and performance on different benchmarks also prove its effectiveness.

  • The experiments are comprehensive, including ablations about different backbone model, loss design, stage-wise effectiveness, which helps validate the design of different components.

Critical weaknesses
  • The geometric signals in the first stage is heuristic, where the designed descriptions are only camera poses and depth. However, the types of geometry is numerous and different questions may need different geometric features. Therefore, the scaling of the model can be limited.

  • I wonder how will the model perform on general understanding benchmarks such as Video-MME. It is not desired If the general understanding ability of the model degrades.

  • How will the model perform when using different geometry models?

Questions for authors

Please refer to weaknesses.

5 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer h6rR,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 4793

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer h6rR,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 4793

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer h6rR,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 4793

Reviewer_GoEqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I think GeoReader asks how to give a VLM geometry in a form it can use. Lightweight Geometric Readers (camera and depth) turn per-view geometry into continuous tokens, trained in Stage 1 so that the frozen target VLM can decode reference-target geometry descriptions from them. In Stage 2 the Readers are frozen and the VLM plus a question-conditioned injector are fine-tuned on spatial QA, with no geometry text generated at inference. With DA3-predicted geometry on Qwen3-VL-4B, it reports 94.3 on MindCube-Tiny and a 70.6 average over six spatial benchmarks, plus MindCube ablations and geometry interventions.

Critical strengths

I think the best evidence is Tab. 8. On a fixed checkpoint, zeroing geometry drops MindCube-Tiny from 89.2 to 77.2, and shuffling poses across views drops it to 63.1, below no-geometry SFT (68.6). That's a causal check this area rarely bothers with, and it convinces me the model really reads its geometry. It's also practical: with DA3 geometry on Qwen2.5-VL-3B, GeoReader gets 81.1 vs 68.6 for vanilla SFT on the same 10K QA (Tab. 5, 14), and it never generates geometry text (0.504 s vs 2.544 s for Text-CoT, Tab. 5).

Critical weaknesses

My main concern is that the core idea, readability through the frozen VLM, is never isolated. No run changes only the Stage-1 objective, and both Tab. 4A alternatives sit below no-geometry SFT (54.7 and 62.7 vs 68.6), so beating them says little. I also wonder how much of the Stage-1 layout survives a zero-initialized, trainable 2048x2048 adapter (A.2.3), and nobody checks what the frozen VLM actually decodes from Z. Tab. 4B has the same problem. GeoReader (89.2) gets annotation-derived poses (A.1.3) while the VLM-3R-style (86.1) and SpaceMind-style (81.1) rows get VGGT features from RGB (A.4.6), and with DA3 geometry GeoReader drops to 81.1 (Tab. 14), 5.0 below VLM-3R-style.

Second, I don't think the headline numbers can be credited to geometry. The only geometry vs no-geometry control on the six benchmarks (3B, Tab. 3) is worth about +1.3 points pooled, against +25.8 over the base in Tab. 2, and the 135K mix has training questions styled after all six benchmarks (A.1.2). At 7B GeoReader beats 3DThinker by only 0.3 Avg and loses VSI by 8.8, which the text doesn't mention. On MindCube only Qwen2.5-VL-3B has a vanilla-SFT row, and that SFT (68.6) already beats the raw-QA SFT in the 3DThinker paper (52.3), so I suspect much of the 5.9 to 18.5 point margin in Tab. 1 is recipe. Everything is also a single run, and fixed scaling, identical to direct injection at init, ends 10 points lower (68.5 vs 78.6, Tab. 7), so I wouldn't trust any 3-point gap here.

Questions for authors

Here are my questions:

  1. Could you put a frozen Reader pretrained by pose/depth regression through the identical Stage 2? That's the run that would most change my score.
  2. How does GeoReader do against VLM-3R-style fusion with the same geometry source, and what does vanilla SFT get with Qwen3-VL-4B on MindCube-Tiny?
  3. I'm curious about the zero condition in Tab. 8. At 77.2 it still keeps about 42% of the gain over vanilla SFT. Is that the slot tokens, the read pass, or geometry acting as a training-time aid?
  4. Correct me if I'm wrong, but SSR (Liu et al., NeurIPS 2025) and GASP (Yeh et al., CVPR 2026), which are not cited, already trains latents so a frozen LM reconstructs depth-derived rationales, then answers without them. Could contribution 1 be scoped to what's actually new?
  5. A.1.2 lists "training questions from ... MMSI", but I thought MMSI-Bench was evaluation-only. Where do these come from, and how was overlap checked, given the +13.5 on MMSI camera motion?
5 comments
AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer GoEq,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Best,

The Authors of Submission 4793

AuthorsOpenReview ↗

Follow-Up to Your Review (Second Reminder)

Dear Reviewer GoEq,

Thank you again for the time and care you put into reviewing our work!

We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.

If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.

Thanks again for your support!

Sincerely yours,

The Authors of Submission 4793

AuthorsOpenReview ↗

Follow-Up to Your Review (Third Reminder)

Dear Reviewer GoEq,

We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.

As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.

Thank you again for your time and effort devoted in this review!

Best,

The Authors of Submission 4793

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all our reviewers for the time and thoughtful feedback!

Your questions prompted us to isolate the effect of Stage 1 supervision in GeoReader, compare methods using the same geometry source, and document the data and training settings more clearly.


We are glad that our reviewers recognize the key contributions of this work, including:

  • Reviewer GoEq highlighted the fixed-checkpoint geometry interventions, especially the drop when poses are shuffled across views, and noted the inference speed of answering directly from continuous tokens.
  • Reviewer h6rR highlighted that the training makes geometry tokens readable by the VLM and valued the ablations across backbones, objectives, and training stages.
  • Reviewer UXxg highlighted the distinction between encoding geometry and getting the VLM to use it, the separation of Reader pretraining from QA adaptation, and the gains across three VLM families.

We would like to highlight again the key motivation and value of this work:

  • Learning geometry the receiving VLM can read. GeoReader trains lightweight Camera and Depth Readers through description supervision with the target VLM frozen. The Camera Reader learns from reference–target descriptions that are independent of the QA prompt.
  • Adapting the use of geometry to the question. We freeze the Readers during QA training and learn a question-conditioned injector. At inference, the VLM answers directly from continuous geometry tokens without generating a geometry description.
  • Testing whether predictions depend on geometry. We evaluate GeoReader on seven spatial reasoning benchmarks across three VLM families. With the MindCube-Tiny checkpoint fixed, zeroing the geometric residual reduces accuracy from 89.2% to 77.2%, and shuffling poses across views reduces it to 63.1%.

Additional Results & Experiments

Based on your suggestions, we have made efforts in revision, including point-by-point responses, as well as the revised manuscript with changes highlighted in blue colors. Specifically, we summarize the changes made as follows:

  • Stage 1 objective and frozen-VLM predictions (GoEq). Under matched Stage 2 settings, readability supervision improves over pose/depth regression from 68.48% to 78.57% with direct injection and from 85.81% to 89.24% with the question-conditioned injector. On eight image pairs, removing geometry increases the frozen VLM's masked cross-entropy at geometric-content positions from 0.708 to 2.078.
  • Same-source fusion and matched training controls (GoEq, UXxg). With DA3 fixed, GeoReader scores 81.14%, compared with 77.71% for VLM-3R-style fusion and 79.24% for SpaceMind-style fusion. GeoReader is 12.57 points above the strongest of three matched 3DThinker-style feature-alignment controls. We separately report the original-recipe Stage 1 reproduction and the Stage 2 LoRA and full fine-tuning runs. For Qwen3-VL-4B, matched Vanilla SFT scores 81.33% and GeoReader with DA3 scores 94.29%.
  • General understanding and geometry prediction (h6rR). GeoReader scores 78.70% on MMBench-EN Test and 57.81% on Video-MME without subtitles, 0.39 and 0.97 points below the locally evaluated base model. With VGGT, GeoReader reaches 79.71% on MindCube-Tiny, compared with 81.14% with DA3 and 68.57% for Vanilla SFT.
  • Repeatability and token capacity (GoEq, UXxg). Accuracy varies by at most one percentage point across three repeat runs. Increasing the Camera Reader from one to eight tokens per view raises MindCube-Tiny accuracy from 89.24% to 91.33%.

Other Revisions & Modifications

  • We have documented the Stage 2 QA sources, shared question templates, and overlap checks. We will correct “MMSI” to “MMSI-style” in Appendix A.1.2 and distinguish local matched controls from published comparisons. Table and figure numbers refer to the submitted manuscript.
  • We have clarified how GeoReader builds on SSR and GASP and discuss the mixed Qwen2.5-VL-7B results. GeoReader and 3DThinker have similar six-benchmark averages (65.0% and 64.7%), while GeoReader scores 8.8 points lower on VSI-Bench.
AuthorsOpenReview ↗

Thank You for Your Response

Dear Reviewer GoEq,

Thank you for taking the time to read our rebuttal and follow up!

We are glad that our responses addressed your concerns, and we appreciate your decision to raise your rating towards acceptance.

Your feedback helped us strengthen the manuscript. Thanks again!

Warm regards,

The Authors of Submission 4793

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.