acceptodds
Under review as a conference paper at ICLR 2027

AskGeo: Learning to Query Geometry Models for Spatial Reasoning

Abstract

Spatial understanding requires vision-language models (VLMs) to identify the geometric evidence relevant to a given question, across both static and dynamic scenes. Despite their strong semantic capabilities, existing VLMs often struggle to recover 3D structure and reason about spatial relations directly from visual observations. Meanwhile, pretrained feed-forward geometry models encode rich geometric knowledge, but existing methods transfer it by extracting geometric features independently of the question and only afterwards selecting or fusing them, so the geometry model performs the same computation regardless of what is asked. We observe that query-based geometry models such as D4RT already provide a retrieval mechanism: their decoder retrieves geometric information for arbitrary spatiotemporal queries. Building on this, we introduce AskGeo, a framework that lets a VLM learn to query a pretrained geometry model for the evidence a question requires. AskGeo projects question-conditioned VLM features into the native query space of the geometry decoder, which retrieves geometric evidence tailored to the question and returns it to the VLM as a few geometry tokens. Because the queries are formed from the VLM's own representation of the question, the retrieved tokens are aligned with how the VLM interprets the question, and what to ask is learned end to end from question-answer supervision. Experiments on the static VSI-Bench and dynamic DSR-Bench show that AskGeo achieves state-of-the-art performance on both benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D vision, 3D scene understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Gw9yOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose to use query-based pretrained geometry model to extract geometric cues for spatial reasoning. Different from previous methods that extract fixed features of geometry models first and obtain question-relevant clues then, the proposed AskGeo framework directly queries the decoder of pretrained geometry model to extract question-relevant geometric features, making the resultant features contain more important information. On both static and dynamic spatial reasoning benchmarks, the proposed framework achieves the best performance.

Critical strengths

To prove the effectiveness of the proposed framework, the authors conduct comprehensive experiments. First, they compare with different baselines across both static and dynamic spatial reasoning benchmarks, showing the effectiveness of extracting geometric features directly in a query-based approach rather than based on fixed geometric features. Second, they conduct numerous ablations to validate the design of different components.

Critical weaknesses
  1. I think there should be an additional ablation study to evaluate whether the general understanding ability of the final model degrades on benchmarks such as Video-MME.

  2. I wonder whether the model trained on DSR-Train could improve performance also on similar benchmarks such as VLM4D and STI-Bench.

Questions for authors

Please refer to the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Gw9y

We thank the reviewer for the positive assessment and the useful suggestions. Changes in the revised PDF are shown in teal.

W1 & W2: General video understanding and generalization to other spatio-temporal benchmarks.

Following the suggestions, we evaluate the models trained on DSR-Train zero-shot, without any further training, on Video-MME (general video understanding) and on VLM4D and STI-Bench (spatio-temporal reasoning). All models are evaluated on the official test sets with the official evaluation protocols, using the same evaluation setting. The results have been added to App. A.5 (Table 11).

ModelVideo-MMEVLM4DSTI-Bench
GSM (DSR Suite)61.450.933.1
GeoSR55.045.032.6
AskGeo (Ours)63.758.940.6

On Video-MME, AskGeo scores 63.7, compared with 61.4 for GSM and 55.0 for GeoSR, indicating that it preserves general video understanding. On VLM4D and STI-Bench, AskGeo improves over GSM by 8.0 and 7.5 points and over GeoSR by 13.9 and 8.0 points, respectively, suggesting that the gains from training on DSR-Train carry over to other spatio-temporal benchmarks.

We thank the reviewer again and hope that our responses have addressed the reviewer’s concerns.

The authors address part of my concerns. I keep my rating for weak acceptance.

Reviewer_6bpPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents AskGeo, a framework for improving spatial reasoning in vision-language models through question-conditioned retrieval from pretrained geometry models. Instead of extracting generic geometric features independently of the question, AskGeo maps question-conditioned language-model representations into the query space of a frozen geometry decoder. The retrieved features are fused across decoder layers and projected into compact geometry tokens, which are combined with the original visual and language tokens for answer generation. The query interface is learned through answer-token supervision using a two-stage strategy: first training the query and projection modules with frozen backbones, then adapting the language model with LoRA while restoring its visual inputs. Experiments on DSR-Bench and VSI-Bench report improved spatial reasoning performance, with ablations supporting the usefulness of geometry tokens and question-conditioned retrieval.

Critical strengths
  1. A coherent approach to reusing pretrained geometric knowledge. AskGeo introduces question conditioning directly into a pretrained geometry decoder through queries derived from the VLM’s own question representations. The interface is learned from answer supervision without requiring additional query annotations or updating the geometry backbone. This provides a practical way to connect language understanding with existing geometric representations. The strong Stage 1 result—65.7 on DSR-Bench with a frozen language model and without its original visual tokens—supports the usefulness of the retrieved features.

  2. Meaningful empirical support beyond ordinary task-specific fine-tuning. On DSR-Bench, AskGeo achieves 68.8 compared with 56.8 for VLM-only SFT. More importantly, the retrieval-design ablations use the same geometry and VLM backbones: AskGeo outperforms Q-Former selection by 7.9 points and dense-grid queries by 2.0 points. These comparisons provide evidence that the proposed retrieval design contributes beyond backbone choice and question-answer fine-tuning, strengthening the case for its methodological contribution.

Critical weaknesses
  1. Limited validation of the geometric content retrieved by the learned queries. The queries are optimized solely through answer supervision, and the current analysis primarily establishes question dependence rather than grounding in the relevant geometric evidence. In particular, Table 4 shows that changing the query-construction question reduces accuracy, but question-conditioned tokens can also carry linguistic information. Targeted interventions—such as replacing the geometry-branch video while keeping the question fixed, or selectively masking relevant versus irrelevant objects and temporal segments—would help establish whether the gains depend on retrieving the required geometric evidence. This distinction is consequential because question-relevant geometry retrieval is the paper’s central methodological claim.

  2. Unclear generalization beyond structured question formulations. Although AskGeo improves overall DSR-Bench performance, its Non-Template score is 51.1, compared with 52.9 for GeoSR. The overall gains therefore do not establish an advantage on questions with more flexible linguistic expressions. Evaluation using answer-preserving paraphrases or held-out question formulations would help determine whether the learned query interface generalizes beyond familiar question patterns. This matters for assessing the broader applicability of a method whose retrieval mechanism is directly conditioned on language.

Questions for authors
  1. Could the authors provide a controlled experiment validating that the retrieved tokens capture question-relevant geometric evidence? For example, selectively masking relevant versus irrelevant objects or temporal segments in the geometry branch, while keeping the question and original VLM visual inputs fixed, would help establish whether the gains depend on the required scene evidence.

  2. Could the authors evaluate robustness to meaning-preserving paraphrases of existing test questions, comparing AskGeo with a retrieval baseline under the same perturbations? Given the weaker result on the Non-Template subset, this experiment would help determine whether the benefits of question-conditioned retrieval persist across different formulations of the same spatial question.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6bpP

We thank the reviewer for the thoughtful review and the concrete experimental suggestions, which we followed. Changes in the revised PDF are shown in orange.

1. W1 / Q1: Do the retrieved tokens capture question-relevant geometric evidence?

Following the suggestion, we perturb the video input of the geometry branch, and evaluate the final model on DSR-Bench (App. A.3, Table 8, Fig. 5):

Geometry-branch inputAcc.Δ
Original video68.8–
Referent masked (queried objects masked in every frame)61.3−7.5
Random-position masked (same mask sizes and motion, minimal overlap with referents)67.9−0.9
Temporal segment masked (frames overlapping the queried window)66.9−1.9

Masking the referred objects costs 7.5 points, more than eight times the drop from masks of the same size elsewhere. Moreover, about 48% of the DSR-Bench test questions specify a query object by its bounding box, so this benchmark-wide drop comes from roughly half of the questions. This shows that the geometry tokens depend on the evidence of the queried objects rather than on generic scene content. Masking a temporal segment causes a smaller drop of 1.9 points, likely because the remaining frames still cover part of the queried window. Since the VLM's own visual tokens remain intact in all cases, these drops are lower bounds on how much the answer relies on the masked evidence in the geometry branch.

On the question side, we also added a controlled type-swapped question (Table 5): it keeps the same video, objects, and time window and changes only what is asked, and it still reduces accuracy by 11.6 points. In addition, App. A.2 visualizes which space-time locations the learned queries retrieve; they concentrate on the object referred to by the question and follow it over time.

2. W2 / Q2: Non-Template subset and robustness to paraphrasing.

Non-Template gap. In Table 1, AskGeo adapts the VLM with LoRA, whereas GeoSR fine-tunes the full VLM. With full fine-tuning, as in GeoSR, AskGeo reaches 54.8 on the Non-Template subset, surpassing GeoSR (52.9), and 69.3 overall (App. A.4, Table 10, with per-category results). The gap on this subset in Table 1 therefore comes from the fine-tuning strategy rather than from the question-conditioned retrieval. We state this in Sec. 4.2 of the revision.

Paraphrase robustness. We paraphrased all DSR-Bench eval questions in three styles with increasing deviation from the original wording: structural reordering, synonym substitution, and free rewording (word overlap with the original: 1.00 / 0.62 / 0.52). Options and answers are unchanged. The paraphrases were generated by an LLM (GPT-6 Astra) from a different model family than our VLM, then filtered by rule-based checks (bounding boxes, timestamps, object names, and reference frame preserved) and a blind LLM judge. Example:

  • Original: Between 41.0s and 57.1s, from the perspective of camera at 36.0s, how does the distance between camera and car with initial bounding box coordinates [421,179,441,193] change?
  • Free rewording: With the camera's viewpoint at 36.0s as a fixed reference, how does the gap between the camera and the car (initial bounding box coordinates [421,179,441,193]) change from 41.0s to 57.1s?
Question styleOverlapAskGeo
Original–68.8
Reordering1.0068.6 (−0.2)
Synonym substitution0.6268.5 (−0.3)
Free rewording0.5268.5 (−0.3)

Accuracy changes by at most 0.3 points even for free rewording, so the learned query interface does not depend on the exact template wording. Details are in App. A.4 (Table 9).

We thank the reviewer again and hope that our responses have addressed the reviewer’s concerns.

The authors’ response has addressed most of my concerns. I will maintain my original rating.

Reviewer_8PsbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AskGeo, a framework that turns a vision-language model's question representation into queries for a frozen pretrained geometry decoder. Instead of explicitly reconstructing geometry or selecting from question-independent geometry tokens, the method replaces the decoder's coordinate queries with learned question-conditioned queries, aggregates features from several decoder layers, and passes the resulting compact geometry tokens to the VLM. Training has two stages: the query and projection modules are first learned without the VLM's original visual tokens, after which those tokens are restored and the language model is adapted with LoRA. AskGeo reaches 68.8 average accuracy on DSR-Bench, 2.7 points above GeoSR and 7.9 points above a reproduced Q-Former selection baseline. VSI-Bench results and targeted ablations further suggest that question-conditioned retrieval through the decoder is useful.

Critical strengths
  1. The approach is well motivated. Using a pretrained geometry decoder's native query interface provides a direct route to retrieving relevant geometric evidence before it is compressed into a fixed, question-independent representation.

  2. Modularity is evident in the frozen geometry encoder and decoder, the video-length-independent number of added tokens, and the Point4D experiment. These choices suggest that the design is not restricted to one geometry backbone.

  3. On DSR-Bench, the 68.8 average accuracy is a strong result. The largest improvements occur in several relative-motion categories, which fits the motivation for question-conditioned spatiotemporal geometry.

  4. Several ablations examine meaningful alternatives, including Q-Former selection, dense-grid queries, direct question-token embeddings, question-free queries, and mismatched questions. Together, they support both decoder-level retrieval and conditioning on the question.

Critical weaknesses
  1. Coverage across the two claimed settings is uneven. DSR-Bench receives a full category-level comparison, whereas VSI-Bench is supported only by aggregate values in an ablation table, without the same category-level comparison or dataset and evaluation details. The state-of-the-art claim therefore cannot be assessed with the same confidence on both benchmarks.

  2. Semantic-specificity evidence remains suggestive rather than conclusive. Removing the question or substituting an unspecified different question changes the input distribution and may also misalign the geometry query with the answer prompt. The accuracy loss does not by itself isolate whether the retrieved tokens encode the particular evidence requested.

  3. Cost reporting does not yet substantiate the efficiency claim. Although AskGeo adds a fixed number of tokens to the VLM, it still runs a pretrained geometry encoder and multiple decoder layers; training cost, inference latency, memory use, and matched-compute comparisons with token-selection baselines are not provided.

Questions for authors
  1. How is the other-question condition in Table 4 constructed? Is the replacement sampled from the same video, matched by question type and length, or chosen randomly? Could controls distinguish semantic mismatch from broader distribution shift?

  2. Can the authors include complete per-category VSI-Bench comparisons in the main paper, together with the precise baselines and evaluation protocol behind the state-of-the-art claim?

  3. What are the end-to-end latency, memory footprint, and training costs relative to the reproduced Q-Former selection baseline and VLM-only supervised fine-tuning? In practice, how much additional cost comes from using four decoder layers despite the compact VLM input?

  4. Why does a decoder pretrained with coordinate-based queries respond meaningfully to projected language-derived queries, and how sensitive is this transfer to the geometry model's original query parameterization? An analysis of the learned queries or retrieved tokens would help validate the proposed mechanism.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8Psb (1/2)

We thank the reviewer for the detailed and constructive review. Changes in the revised PDF are shown in blue.

1. W1 / Q2: Uneven coverage of VSI-Bench.

We moved the full per-category VSI-Bench comparison into the main paper (Table 2), with the training data and evaluation protocol in Sec. 4.1, which follows GeoSR. In the original submission, our VSI-Bench model used roughly half of GeoSR's input resolution. In the revision, we use the same training and evaluation settings as GeoSR, including the training data (SPAR-7M and LLaVA-Hound) and the input resolution. Following the official protocol, we report accuracy for multiple-choice subtasks, mean relative accuracy for numerical subtasks, and the average over all eight subtasks. AskGeo achieves 58.1, 6.2 points above GeoSR.

2. W2 / Q1: How is the "other question" constructed, and is the drop caused by semantic mismatch or by distribution shift?

In the original Table 4 (now Table 5), the "other question" was randomly sampled from the questions of other videos, including non-template questions. As the reviewer points out, this also changes the referred objects, time windows, and phrasing, so its drop mixes the two effects. To separate them, we add a controlled type-swapped question. It keeps the same video, template, bounding boxes, and time window, and changes only the words that specify the question type (e.g., "distance" → "direction"). The answer prompt to the LLM is unchanged.

Question used to build geometry queriesAcc.Δ
Original question68.8–
Type-swapped (same video, objects, time window)57.2−11.6
Random question from another video49.4−19.4
No question45.0−23.8

Matching the input distribution recovers 7.8 of the 19.4 points, so part of the original drop is indeed caused by distribution shift. The remaining 11.6-point drop appears when only the requested quantity differs, which we attribute to semantic mismatch. On the concern about misalignment with the answer prompt: in the type-swapped condition, the geometry query and the answer prompt refer to the same objects and time window and differ only in what is asked. The misalignment therefore amounts to the requested evidence being missing, which is the effect we intend to measure.

We also added an intervention on the geometry input (App. A.3, suggested by Reviewer 6bpP). Masking the queried objects only in the geometry branch, with the question and the VLM's visual input unchanged, lowers accuracy by 7.5 points, whereas masks of the same size at random positions lower it by only 0.9. Since about 48% of the DSR-Bench test questions specify a query object by its bounding box, this drop over the whole benchmark comes from roughly half of the questions. Together, the two controls indicate that the tokens carry evidence for what the question asks about and for the objects it refers to. We also changed "confirm" to "indicate" in Sec. 4.3 and revised the Conclusion accordingly.

AuthorsOpenReview ↗

Response to Reviewer 8Psb (2/2)

3. W3 / Q3: Cost.

We added App. A.6 (Table 12), which compares the training and inference cost of AskGeo with VLM-only SFT and the reproduced Q-Former selection on the same geometry model and VLM.

MethodTraining (GPU-h)Latency (s)Peak mem. (GB)Acc.
VLM-only SFT15.10.8817.756.8
Q-Former selection21.73.2824.660.9
AskGeo21.41.3522.268.8

Compared with Q-Former selection, AskGeo has a similar training cost (21.4 vs. 21.7 GPU-h), lower latency (1.35 vs. 3.28 s), and lower peak memory (22.2 vs. 24.6 GB), while improving accuracy by 7.9 points. Both methods run the geometry encoder once per video; AskGeo then decodes only M = 32 queries, whereas Q-Former selection decodes a dense set of question-independent features before selecting from them. Compared with VLM-only SFT, AskGeo adds 0.47 s of latency and 4.5 GB of peak memory, and improves accuracy by 12.0 points.

4. Q4: Why does a decoder pretrained with coordinate queries respond to language-derived queries?

The D4RT decoder retrieves information through cross-attention: each query vector attends to the encoded scene tokens according to feature similarity. Because ϕin\phi_{\mathrm{in}} is trained with gradients that pass through the frozen decoder, it learns to produce query vectors that the decoder can interpret.

To check what the learned queries retrieve (App. A.2, Fig. 4), we pass a dense grid of native coordinate queries (16×16 per frame, 48 frames) through the same frozen decoder and highlight the 1% whose decoder states are most similar to our geometry tokens. This footprint concentrates on the object referred to by the question and follows it over time, rather than spreading over the background or other moving objects. In other words, the learned queries behave like native queries placed on the question-relevant object.

On sensitivity to the query parameterization: AskGeo performs comparably with two query-based geometry models, D4RT and Point4D (Table 6). The learned queries also outperform a dense grid of native coordinate queries by 2.0 points (Table 4), showing that they retrieve more useful evidence than uniformly placed native queries.

We thank the reviewer again and hope that our responses have addressed the reviewer’s concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (to all reviewers)

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF; changes are shown in color (blue: Reviewer 8Psb, orange: Reviewer 6bpP, teal: Reviewer Gw9y, purple: shared). Main changes:

  1. VSI-Bench in the main paper (Table 2, Sec. 4.1–4.2): full per-category comparison, training data, and protocol. With input resolution aligned to GeoSR, AskGeo reaches 58.1 (+6.2 over GeoSR).
  2. Controlled tests of question-specific retrieval (Table 5, App. A.3): a type-swapped question that keeps the video, objects, and time window but changes only what is asked (−11.6), and masking the queried objects only in the geometry branch (−7.5 vs. −0.9 for random masks of the same size).
  3. Robustness and generalization (Sec. 4.3, App. A.4–A.5): accuracy changes by at most 0.3 under three styles of meaning-preserving paraphrases; zero-shot results on Video-MME, VLM4D, and STI-Bench.
  4. Mechanism and cost (App. A.2, A.6): an analysis of what the learned queries retrieve, and training/inference cost against VLM-only SFT and Q-Former selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.