acceptodds
Under review as a conference paper at ICLR 2027

Test-time reasoning with computational evidence for spatial question answering

Abstract

Vision-language models (VLMs) have made substantial progress, while the growing interest in embodied AI underscores the need for reliable spatial reasoning. Existing approaches have attempted to improve spatial reasoning by enhancing visual reasoning capabilities or augmenting VLMs with external visual tools. However, visual reasoning can be unreliable when the VLM lacks sufficient spatial sensing capability. Although tool augmentation can potentially address this limitation, invoking a large set of tools or reconstructing an entire 3D scene is often computationally expensive, inefficient, and unnecessary. Our analysis reveals a capability imbalance: VLMs show strong object-level perception and reasoning but struggle with spatial sensing, suggesting that explicit geometric evidence can help bridge this gap. We propose Test-Time Spatial Reasoning (TTSR), a training-free framework that supports spatial question answering through computational evidence. TTSR constructs a question-specific geometric plan, then grounds the entities and acquires the spatial information using the VLM and minimal auxiliary tools. It performs reference-frame transformations and compositional spatial computations, producing explicit evidence that the VLM interprets using its own reasoning capabilities to get the answer. Experiments on recently proposed challenging spatial reasoning benchmarks demonstrate that TTSR substantially outperforms existing state-of-the-art methods and generalizes consistently across diverse spatial reasoning settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Test time spatial reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_4qFwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

To address the limited spatial perception of vision-language models in complex spatial question answering, particularly in multi-hop reasoning and viewpoint changes, this paper proposes TTSR, a training-free test-time spatial reasoning framework. It decomposes each question into entities and spatial constraints, uses a VLM to localize objects, and obtains geometric information through depth estimation. Coordinate transformations and explicit spatial calculations then produce evidence to support answer selection. Experiments show improvements on three spatial reasoning benchmarks, with MultiHop Spatial accuracy rising from 36.4% to 60.3%.

Critical strengths
  1. TTSR organizes and processes geometric information, converting it into numerical evidence that can directly evaluate candidate answers.
  2. The framework improves performance on MultiHop Spatial, ViewSpatial, and SPHERE, which emphasize different aspects of spatial reasoning.
Critical weaknesses
  1. TTSR shares substantial conceptual similarities with existing frameworks, such as Geometrically-Constrained Agent (GCA) [1]. Prior work has also explored training-free approaches that enhance VLM spatial reasoning through external geometry tools and explicit geometric computation.
  2. Figures such as Figure 2 do not clearly communicate the method’s technical details, making it difficult to identify TTSR’s key innovations and distinguish it from prior approaches.

[1] Chen, Z., Lu, X., Zheng, Z., Li, P., He, L., Zhou, Y., Shao, J., Zhuang, B., & Sheng, L. (2025). Geometrically-Constrained Agent for Spatial Reasoning. arXiv preprint arXiv:2511.22659.

Questions for authors
  1. How does TTSR differ from existing spatial reasoning frameworks, particularly Geometrically-Constrained Agent (GCA; Chen et al., 2025) [1]?
  2. For object localization, why does TTSR use bounding boxes predicted directly by a VLM rather than established segmentation or detection models, such as those in the SAM and DINO families? Also, given the use of Depth Anything V2 for depth estimation, why was Depth Anything V3 not considered?

[1] Chen, Z., Lu, X., Zheng, Z., Li, P., He, L., Zhou, Y., Shao, J., Zhuang, B., & Sheng, L. (2025). Geometrically-Constrained Agent for Spatial Reasoning. arXiv preprint arXiv:2511.22659.

3 comments

We sincerely thank you for your constructive review and for your questions about our comparison with GCA and our choice of perception modules. We address your concerns below.

W1 + Q1 (difference from GCA). Please see the General Response.

W2 (Figure 2). The authors thank you for your constructive suggestions and we will revise the figure to the technical details.

Q2 (VLM boxes vs. SAM/DINO). We first discuss the localization. In Section 3, we discuss the capability boundary of the QWen 3 8b VL, and we observe that the VLM's grounding is already on par with a dedicated detector: On COCO 2017, Qwen3-VL-8B reaches 79.9 F1@IoU>0.5, compared with 79.2 for Faster R-CNN.

Open-vocabulary detectors such as Grounding DINO are tuned mainly for category phrases, whereas the VLM already parses these expressions. Using the VLM's own localization and grounding keeps the streamlined design pipeline to one VLM, align our goal of adding tools only where the VLM strongly lacks a capability, such as depth estimation from 2D image.

SAM is a complementary model to perform fine-grained segmentation. Our Limitations (Appendix I) note that a box center and median box depth can be wrong under occlusion or for irregular shapes. SAM masks from the VLM's boxes could help address exactly this, and we will add this as a direction.

To further validate our claim, we independently integrate SAM and Depth Anything 3 into the TTSR pipeline to investigate whether incorporating additional specialized tools can further improve spatial reasoning performance. Given the limited time available during the rebuttal period, we conduct these experiments on MultiHop-Spatial benchmark.

(Depth Anything V2 vs. V3). TTSR uses depth only to obtain ordering and approximate metric depth at object locations, and the depth module can be applied off the shelf models without any constraint. In our ablation study, we test this directly: replacing Depth Anything V2 with MoGe-2 changes MultiHop accuracy by only +0.5 points. Depth Anything V2 also already reaches 98.0% relative-depth accuracy under our diagnostic protocol. The choice of estimator therefore has limited benefits, and the main error sources lie in planning and viewpoint construction (Table 6). Per our reviewer's suggestion, we conduct experiment that replace Depth anything V2 with V3 to test the performance change on the Multihop spatial

Table: Ablation study of integrating different external tools into the TTSR reasoning pipeline.

RunAccuracyCorrect answer and box (IoU ≥ 0.5)
Ours with (Depth Anything V2)60.3350.22
Ours with (Depth Anything V3)60.8050.49
Ours with (Depth Anything V3 + SAM 2.1)61.0450.89

As shown in the table above, we can clearly observe that incorporating more advanced tools yields only marginal improvements in final prediction accuracy. Specifically, upgrading from Depth Anything V2 to V3 improves performance by only 0.5 percentage, while incorporating SAM provides a 0.24 percentage gain. These findings are consistent with our motivation that simply adding more powerful tools does not necessarily lead to substantial performance improvements. Instead, the effectiveness of tool integration depends on whether the tools provide the specific evidence required by the reasoning task, highlighting the importance of question-driven tool selection in TTSR.

I thank the authors for their thoughtful response. I found it constructive and believe it has substantially addressed several of my concerns. Although the modifications relative to GCA are relatively simple, they target its weaknesses effectively and yield notable improvements. The authors have also demonstrated that a more powerful model is not necessarily needed, which helps validate some of my own intuitions. I will therefore raise my score.However, I sincerely hope that the promised revisions will be incorporated into the paper promptly and clearly highlighted, as this may affect my final decision.

Dear Reviewer 4qFw,

We sincerely thank you for your engagement in the discussion. We are currently revising the paper based on the reviewers' constructive feedback and will post the revised version before the end of the response phase. Thank you once again for your valuable feedback!

Best regards, Authors

Reviewer_MBnZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TTSR, a training-free framework targeting spatial VLM reasoning. Instead of relying on full 3D reconstruction, it turns questions into executable plans with explicit reference frames, constraints, and dependencies. It pairs a frozen VLM for 2D grounding with an off-the-shelf metric depth estimator, lifting boxes into 2.5D space to calculate exact geometric evidence. The core win comes from its dependency-aware execution and relation-specific math. TTSR improves Qwen3-VL-8B's accuracy from 36.4% to 60.3% on MultiHop Spatial, 40.3% to 48.9% on ViewSpatial, and 63.3% to 68.6% on SPHERE.

Critical strengths
  1. TTSR improves overall accuracy on MultiHop Spatial from 36.4% to 60.3%, with 3-hop accuracy increasing from 26.4% to 45.6%. These results show that structured planning and geometric computation can significantly improve multi-hop spatial reasoning. The gains on ViewSpatial and SPHERE further support its effectiveness.

  2. The explicit representation of entities, reference frames, and geometric evidence makes the reasoning process easier to understand and debug. The framework also uses open-source models without additional training, making it a practical approach.

Critical weaknesses
  1. Prior works like APC, GCA, and MSSR already touch on perspective changes, geometric math, and curated evidence, overlapping heavily with TTSR’s core building blocks. The authors need to cut through the terminology and clearly spell out how TTSR’s dependency-aware planning and deterministic geometric executor fundamentally depart from these setups, rather than simply repackaging existing ideas.

  2. The experimental setup feels a bit murky in places. First off, key implementation details for baselines like APC, GCA, and MSSR aren't spelled out clearly, so it's tough to tell if the playing field is actually level. Missing comparisons with frontier models (e.g. GPT-6, Gemini 4, and Qwen3.8-27B) also leave a gap. More importantly, the authors claim better visual perception amplifies TTSR's gains, but they only test stronger backbones on a single benchmark. To back that up, they need to run TTSR with beefier backbones across all three benchmarks and check if the advantage still holds.

Questions for authors
  1. Prior works like APC, GCA, and MSSR have already touched on perspective shifts, geometric math, and targeted evidence selection. What is the core conceptual departure here? How do the dependency-aware planner and deterministic executor fundamentally differ from existing agentic or tool-augmented pipelines?

  2. The authors should clarify the exact evaluation setups for APC, GCA, and MSSR. Which baseline numbers were rerun locally versus lifted directly from the original papers, what prompts and backbones were plugged in, and how did the authors guarantee an apples-to-apples comparison?

  3. How does TTSR stack up against frontier models like GPT-6 and Gemini 4, or larger open weights like Qwen3.8-27B? Would the paper's core claims and performance margins still hold once stronger baselines enter the picture?

  4. Could the authors evaluate TTSR with a stronger backbone (e.g., Qwen3.8-27B) across all three benchmarks, not just MultiHop Spatial? This would show whether the performance gains hold up and truly back up the claim that better visual perception leads to bigger improvements.

1 comment

We sincerely thank you for your constructive review and for recognizing the gains on compositional questions and the interpretability of our method. We address your concerns below.

W1 + Q1 (conceptual departure). Please see the General Response. To answer your question about the planner and executor specifically: in agentic or tool-augmented pipelines such as GCA and MSSR, the VLM decides at inference time which computations to run and how to order them. In TTSR, the plan generates a dependency structure first (for example, ground the reference before measuring distances, and filter by attributes before ranking), and the executor enforces that order. A lexical pass checks the plan against the question text and restores conditions the planner omits. Errors such as ranking all options before the eligibility filter are prevented whenever the condition reaches the plan. Removing this filtering reduces 5.7 points overall and 10.8 points on 2-hop questions (Table 4).

W2 + Q2 (baseline setups). We run APC, GCA, and MSSR from their official github releases. Every VLM used the same checkpoint as our method (Qwen3-VL-8B-Instruct). For scoring, all methods use the same scoring rule as ours. For ViewSpatial: first standalone A-D on the final line. For SPHERE, we use its official parser. On multihop spatial, each method's option is read from their response.

Imported MSSR on ViewSpatial is taken from Table 3 of the MSSR paper. GPT-4o on ViewSpatial (ViewSpatial-Bench paper); GPT-4o, Gemini 2.0 Flash, and Gemini 2.5 Flash on SPHERE (SPHERE paper); Claude Opus 4.5 on MultiHop Spatial (MultiHop Spatial paper).

The other baselines including SpatialVLM, SpaceOM, LVR, TreeVGR use their released checkpoints on their original backbones, and Tables 2–3 report gains relative to each method's own backbone.

W2 + Q3 + Q4 (frontier models and stronger backbones). In our experiment section, we include closed-source models in Tables 2–3. With an 8B backbone, TTSR exceeds Gemini 2.5 Flash, Claude Opus 4.5, and GPT-4o on MultiHop Spatial and exceeds Gemini 2.5 Flash and GPT-4o on ViewSpatial. We also test the Backbone change on MultiHop Spatial: +23.9 points (Qwen3-VL-8B), +9.8 (Qwen2.5-VL-7B), and +11.0 (InternVL3.5-4B). To address your request directly:

ModelMultiHop-SpatialViewSpatialSPHERE
Qwen3.8-27B, baseline46.2249.5375.28
Qwen3.8-27B + TTSR71.78 (+25.56)59.23 (+9.70)75.87 (+0.59)

We can clearly observe that TTSR generalizes effectively to the larger Qwen3.8-27B model. TTSR improves the larger Qwen3.8-27B by +9.7 points on ViewSpatial, +25.56 points on MultiHop-Spatial and +0.6 on SPHERE, where the baseline is already at 75.3%.

We sincerely apologize that we were unable to conduct a full benchmark evaluation using GPT-6 due to the substantial API costs. Nevertheless, to provide additional experimental evidence within our available budget, we evaluated GPT-6.1 Sol through API calls on a subset of 100 MultiHop-Spatial questions, comprising 28 one-hop, 40 two-hop, and 32 three-hop questions.

Model1 Hop2-hop3-hopOverall
GPT 6.1 Sol85.757.571.970.0
Ours Qwen 3-8b VL64.362.546.958.0

GPT-6.1 Sol achieves 70.0% overall accuracy, compared with 58.0% for TTSR using Qwen3-VL-8B. We acknowledge that GPT-6.1 Sol demonstrates stronger overall performance, particularly on one-hop and three-hop questions compared with our 8B backbone.

Reviewer_BYD6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a novel framework, TTSR, which is a training-free pipeline for spatial question answering. It plans question-dependent operations, grounds entities with a frozen VLM, estimates depth, transforms coordinates into the requested reference frame, and executes spatial constraints to support the answer. The main contribution is this integration of geometric evidence with compositional execution. Experiments report gains over Qwen3-VL-8B on MultihopSpatial, ViewSpatial, and SPHERE, with additional backbone and component ablations.

Critical strengths
  1. The evidence representation is a useful design insight.
  • Table 4 reports 42.0 accuracy when raw geometric quantities replace the relation-specific evidence, versus 60.3 for the full pipeline. Following the description in Section 5.2, this supports explicitly evaluating spatial relations instead of merely supplying coordinates. The gain gives the paper value beyond combining perception tools.
  1. The compositional results are substantial.
  • Table 2 reports 36.4 → 60.3 on MultihopSpatial, including 45.6 on 3-hop questions versus Gemini 2.5 Flash's 38.7. Table 4 also reports gains of 9.8 and 11.0 points with Qwen2.5-VL and InternVL. These results support the usefulness of the pipeline across the tested backbones.
Critical weaknesses
  1. Section 4.1 describes a VLM-generated plan, while Appendix E.1 adds a frozen lexical parser without explaining how its output constrains that plan. Section 4.3 presents evidence for the VLM, but the implementation also permits the executor to answer directly. I accept the stated exclusion of metadata and annotations. The unresolved issue is which component performs each reasoning step. BTW, the executable visual-query decomposition already appears in ViperGPT (Surís et al., ICCV 2023).

  2. The comparisons is not clear. Sec. 5 says methods use official or reproduced implementations, while explicitly importing MSSR's ViewSpatial result. Tables 2–3 do not identify the provenance and scoring conditions of individual rows. This matters especially on SPHERE: the tool-free Auto-Research row scores 71.5, compared with TTSR's 68.6 and the backbone's 63.3. The paper attributes that alternative to constrained decoding. Without knowing which questions and answer-scoring procedures these rows share, I cannot determine what the reported gains establish about spatial reasoning beyond answer selection.

  3. The diagnosis is stronger than the evidence supports. Table 1 establishes a depth-ordering gap under its diagnostic protocol, but GSM8K and text-only logical deduction do not establish reliable reasoning over grounded spatial constraints. Table 4 supports the value of explicit execution; it does not separate sensing limitations from composition failures. Likewise, the three small negative SPHERE category deltas in Table 3 qualify the claim of consistent category improvements (p. 8, lines 427–428). The narrower takeaway in Section 3 and the SPHERE tradeoff acknowledged on p. 9 are more convincing than the abstract's general framing. The practical insight may be about making reasoning executable even when perception is available.

Questions for authors
  1. Could you walk through one existing example from question parsing to final answer, identifying what the parser fixes, what the VLM decides, and when the executor determines the answer?

  2. Which headline comparisons use the same evaluated questions and scoring procedure, particularly the backbone, TTSR, and Auto-Research rows on SPHERE? Please distinguish imported results from matched runs.

  3. How should the depth-sensing diagnosis in Table 1 be interpreted alongside the execution gain in Table 4 and the uneven perspective-task gains? Which conclusions about the source of errors do the existing results support?

3 comments

We sincerely thank you for your constructive review and for recognizing the gains from explicit relation execution. We address your concerns below.

W1 && Q1: Which component performs each reasoning step. The responsibility of each module is as follows:

  • VLM call: proposes the plan πq\pi_q (entities, bindings to answer options, reference frames, constraints).
  • Lexical parser: extracts attribute, position, and ranking expressions from exact spans of the question. A compiler reconciles these with the VLM plan. Recognized attributes and resolvable positional constraints replace the planner's versions. Ranking requirements are checked against the question. Shared constraints are instantiated for every answer option.
  • VLM, vision calls: localizes entities, estimates orientation, and verifies appearance attributes from crops. Depth Anything supplies depth.
  • Executor: back-projects entities, transforms reference frames, evaluates spatial predicates, filters, and ranks.
  • Final selection: If the top margin is greater than a threshold, the executor answers. Otherwise the VLM chooses among the survivors, given the evidence.

Relation to ViperGPT. ViperGPT (Surís et al., ICCV 2023) established executable decomposition of visual queries: a code model writes a Python program against a vision API, and an interpreter runs it. The difference lies in what the model is asked to write and in what the executor guarantees.

Unlike ViperGPT, which generates free-form Python programs, TTSR produces structured, question-conditioned reasoning plans that explicitly specify entities, candidate answers, spatial relationships, and reference frames. Importantly, TTSR introduces a deterministic constraint-verification mechanism that extracts attribute, positional, and ranking requirements from the original question and restores constraints omitted by the generated plan. This reduces reliance on the VLM to correctly preserve every requirement during planning.

ViperGPT provides object coordinates and relative depth estimates through its vision API, but does not explicitly implement 3D back-projection or entity-centric reference-frame transformations. In contrast, TTSR reconstructs approximate 3D entity coordinates using estimated depth and assumed camera intrinsics, then evaluates spatial relationships in the question-specified reference frame. This allows TTSR to explicitly reason about spatial relationships from both camera-centric and entity-centric perspectives.

TTSR explicitly enforces eligibility constraints before spatial ranking, ensuring that candidates violating the question requirements are excluded from consideration. This is particularly important for multi-constraint spatial reasoning, where satisfying a spatial relationship alone does not guarantee a valid answer. Our ablation supports the importance of this design: removing eligibility control reduces MultiHop-Spatial accuracy from 60.3% to 54.6%, with a more pronounced decrease from 61.3% to 50.5% on two-hop questions.

Q1 Walk-through an example: We pick a multihop spatial reasoning example to illustrate our method pipeline:

"From the perspective of the person wearing jeans, which square or rectangular object is closest to them?"

  1. Parser: extracts "square or rectangular" as an attribute constraint and "closest" as a nearest-distance operation.
  2. VLM plan: sets the person as the reference, binds the four options to candidate entities, and proposes nearest-distance ranking, but omits the shape requirement. The compiler adds the parser's shape constraint to every candidate.
  3. VLM perception: localizes all five entities and judges shape from crops. Depth Anything V2 supplies depth, and code builds coordinates.
  4. Executor: filters, then ranks by s_i (Eq. 7).
CandidateVLM verdictNormalized distance to person
A: air ventpass0.6264
B: silver basket with bottlefail (cylindrical)excluded before ranking
C: envelope on bedpass0.4370
D: picture framepass0.5828
  1. Answer: Selects C, which leads the runner-up D by 0.146, which the selection policy treats as decisive.

In this example, the parser repairs a missing requirement, the VLM identifies objects and judges appearance, and the executor decides eligibility, comparison, and the final answer.

W2 && Q2 Comparison is not clear Thank you for this question.

Matched runs. Every row not listed under Imported cells was run by us on the full test sets (MultiHop Spatial 1500, ViewSpatial 5,712; SPHERE 2285 questions × 5 option orders = 11425)

Imported cells. The following cells are imported from published results:

  • MSSR on ViewSpatial (MSSR paper).
  • GPT-4o on ViewSpatial (ViewSpatial-Bench paper).
  • GPT-4o, Gemini 2.0 Flash, and Gemini 2.5 Flash on SPHERE (SPHERE paper).
  • Claude Opus 4.5 on MultiHop Spatial (MultiHop Spatial paper).

Scoring. Baselines and TTSR SPHERE predictions are scored with SPHERE's released answer parser and strict matching.

Auto Research uses constrained decoding, restricting the first token to the valid answer labels, or digits 0–9 for counting (Appendix G). This changes how predictions are generated. Two observations bear on your question of what TTSR's gains show beyond answer selection. First, AR isolates the answer-selection effect on this backbone, and it gives only +1.4 points on MultiHop Spatial and +0.7 on ViewSpatial, compared with +23.9 and +8.6 for TTSR. Second, AR selected SPHERE as its development benchmark and iterated on it (Appendix B), whereas TTSR uses no benchmark-specific tuning. With the same backbone, AR leads on SPHERE, while TTSR leads on MultiHop Spatial (60.3 vs. 37.8) and ViewSpatial (48.9 vs. 41.0). Section 5.1 discusses why geometric execution helps compositional questions but can add failure modes on heterogeneous SPHERE questions.

W3 + Q3: Scope of the diagnosis. Thank you for pointing this out. We totally agree with your framing: the practical insight is that making spatial reasoning executable helps even when object-level perception is available. The existing results support the following conclusions:

(1) The VLM's own depth sensing is weak. Table 1 shows a relative-depth gap under its diagnostic protocol (Qwen3-VL 80.8% vs. 98.0% for Depth Anything V2).

(2) Once a specialized estimator is used, the choice of depth model matters little. Replacing Depth Anything V2 with MoGe-2 changes MultiHop accuracy by only +0.5 points (Table 4).

(3) Composition matters. The "w/ Direct evidence" variant uses the same grounding and depth as the full method and differs only in how the evidence is composed. It only reaches 42.0% compared with 60.3%.

(4) The remaining errors are mostly semantic and viewpoint errors, not depth errors. Planning/semantic errors account for 63.3% of TTSR's errors on SPHERE and 57.4% on ViewSpatial. Orientation and viewpoint construction account for 16.7% on ViewSpatial (Table 6), and zero-shot orientation accuracy is 66.7% on PASCAL3D+ (Appendix I). This matches the uneven perspective gains: +27.8 points on person-relative object orientation, but +1.3 on person-relative direction and +1.8 on scene simulation.

Thank you for the response and the revised PDF; I have read both. The redrawn Fig. 2 and the Qwen3.8-27B result (46.2 → 71.8) are useful additions. I keep my rating at 2, because the revision leaves the three issues I raised in place.

1. The method text still does not match the method. Your response says the parser's constraints replace the planner's, and that the executor answers when the margin is decisive. Fig. 2 now shows a "lexical repair" step, but Section 4.1 is unchanged, the parser is still one sentence in Appendix F.1, and the abstract still says the VLM interprets the evidence. The new Discussion (p. 9) and Appendix B present plan verification as a key advantage over GCA, yet the text does not describe it and no ablation isolates it: the cited 60.3 → 54.6 result removes attribute filtering, not the repair step. ViperGPT is still not cited.

2. The comparisons are unchanged. Tables 2–3 and the setup paragraph of Section 5 are identical to the submission, so the imported cells listed in your response are still not marked. On SPHERE, Qwen2.5-VL-7B scores 54.1 in the SPHERE paper and 63.5 in your run (31.6 vs. 61.4 on D+C), so the imported closed-source rows are not on the same footing as your runs. For Auto-Research, Table 3 pairs 71.5 with a 63.3 backbone, while Fig. 9 reports 66.67 → 71.50 on the agent's own split, so I still cannot tell whether these rows share questions and scoring.

3. The diagnosis and claims are unchanged. We agreed on the takeaway, and the new Table 5 supports it: better depth and segmentation tools add less than one point. But the abstract, Section 3, the "consistent improvements across categories" sentence (now lines 431–432) and the conclusion are word-for-word the same. The paper now says both that sensing is the bottleneck (abstract, Section 3) and that verification is (new Discussion).

The new results strengthen the MultihopSpatial evidence, which I already counted as a strength. What I asked about was the description, the comparisons and the claims. I will look again if these are brought in line with your response.

Reviewer_xmYDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTSR, which uses predefined, fixed programs to obtain additional spatial information through operations such as depth estimation and coordinate transformation, thereby supporting spatial reasoning in vision-language models (VLMs).

Critical strengths

None

Critical weaknesses
  1. Limited novelty. The paradigm of using external tools to acquire geometric information and support spatial reasoning in VLMs has been extensively explored. Although the paper discusses its differences from prior work such as GCA, MSSR, and APC, these distinctions appear relatively minor and primarily concern implementation refinements, leaving the methodological novelty unclear.
  2. The proposed method is limited in generalization. The proposed executor relies on a predefined set of spatial operations and a compact entity-level geometric representation. Although these operations can be composed to address different questions, it remains unclear how the framework handles tasks requiring unsupported geometric computations. For example, determining whether an object can fit inside a container requires reasoning about their shapes and spatial extents, highlighting the narrow applicability of the fixed executor.
Questions for authors

Can TTSR also improve performance on other types of spatial reasoning tasks, such as estimating room dimensions, comparing object sizes, and navigation?

3 comments

We sincerely thank you for your constructive review and for asking us to clarify how TTSR differs from prior work. We address your concerns below.

W1 (novelty). Please see the General Response. We would also like to clarify one point in the summary. TTSR does not run predefined, fixed programs. The plan is generated for each question, such as entities, references, constraints, dependencies, and composed from a set of spatial operators.

W2 (generalization). The current operators are directional relations (E.g. left/right, up/down, front/behind, and compounds such as front-left) in camera- or entity-relative frames, distance ranking (nearest/farthest), attribute verification, 2D overlap, and open-set counting (Sections 4.2–4.3, Appendix E.2).

  • Scope evaluated. For relational and compositional spatial QA, our operators cover three benchmarks that differ in scenes, reference frames, and constraint combinations.

  • Physical fit. Whether an object fits inside a container is a feasibility problem: it depends on object shape, container boundaries, and permissible orientations. TTSR represents each entity by its box center and median depth, which captures neither shape nor interior volume, so we do not claim this capability. Adding it would require a new relation type for the planner and an evidence computation for it. Grounding and filter-before-rank composition would be unchanged. We will add this to the Limitations.

Q1 (room dimensions, object sizes, navigation).

Object sizes: SPHERE includes size questions. TTSR improves Qwen3-VL-8B from 83.0% to 88.7% on Size and from 51.1% to 55.6% on Distance + Size (Table 3).

Room dimensions and navigation: Room-dimension estimation requires scene-level extent rather than relations between entities, and navigation requires path planning over free space. TTSR does not currently support these.

Reviewer_xmYDOpenReview ↗

Official comment by Reviewer xmYD

Thank you for the further clarification. However, my concern about the method’s limited generalization remains. Your explanation also highlights a trade-off relative to GCA: generating task-specific code gives GCA greater flexibility to accommodate different computations and potentially benefit from improvements in LLM reasoning and coding capabilities.

I acknowledge that TTSR generates question-specific plans rather than running predefined programs. Nevertheless, these plans remain constrained by a predefined set of spatial operators. In my view, this restricts the framework’s potential capability ceiling and its applicability to questions requiring computations beyond the supported operator set.

Dear Reviewer xmYD,

Thank you for your active engagement in the discussion and for your constructive follow-up question! We appreciate the opportunity to clarify our perspective.

Our position is that flexibility is not the bottleneck in current spatial QA benchmarks. The reliability of the evidence is. A method can be free to compute anything and return an answer that cannot be trusted. We give three pieces of evidence.

1. The GCA failures we traced did not arise from insufficient flexibility. In these cases, the required computation was expressible and the program executed without errors. In our general response, we identified two failure types:

  • Eligible object. GCA returned an option that fails the question's attribute or position condition in TTSR's checks. For example, it selected a round clock when the question required a “square or rectangular” one.

  • Wrong reference. GCA's program measured distance from the camera rather than from the object or person the question names.

Neither type needs more expressive code. The fix is a different origin, or one more filter before ranking, which TTSR enforces.

Free-form generation does not enforce that every condition in the question is checked. TTSR verifies the plan against the question text and enforces entity grounding and filtering before answering. Removing that filtering alone costs 5.7 points overall and 10.8 points on 2-hop questions (Table 4).

2. More directly, with the same backbone, TTSR outperforms GCA on all three benchmarks.

BenchmarkGCATTSRDifference
MultiHop Spatial37.760.3+22.6
ViewSpatial37.548.9+11.4
SPHERE64.268.6+4.4

3. TTSR provides additional gains with a stronger backbone. Please see our W2+Q3+Q4 response for reviewer MBnZ. We evaluated TTSR with Qwen3.8-27B. With this stronger backbone, TTSR improves accuracy by 25.6 percentage points on MultiHop Spatial and 9.7 percentage points on ViewSpatial relative to the backbone alone. On SPHERE, the gain narrows to 0.6 percentage points, where the backbone already achieves 75.28% accuracy.

On the capability ceiling. The operator set bounds the coverage of the current implementation, rather than imposing a fixed ceiling on the TTSR's framework. A new computation can be incorporated by adding a relation type to the planner and a corresponding evidence function. The rest of TTSR’s pipeline (grounding, plan verification, and filtering before ranking) remains unchanged.

We would be grateful if you would consider these points in your final assessment. Thank you so much.

Best regards,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response

We sincerely thank all four expert reviewers for carefully reviewing our manuscript and raising constructive feedback and insightful questions. Below we address the common questions raised by all four reviewers.

What is the difference between MSSR, APC, and GCA.

The core departure. APC, GCA, and MSSR improve the input the VLM reasons over. APC re-renders the scene from the requested viewpoint, GCA formalizes a reference frame and objective, and MSSR curates the relevant perceptual facts. In all three, the VLM is responsible for working out how the question's conditions combine: APC using VLM reading the transformed scene, GCA applying the code the VLM writes, and MSSR to its reasoning agent. TTSR makes that combination explicit and executes it. The plan specifies which entities to ground, which attributes determine candidate eligibility, which entity is the reference for each relation, and in what order filtering and ranking occur. The plan is checked on the question, and spatial operators then execute it over the computed evidence. TTSR is inspired by human problem-solving: it first interprets the question, constructs a question-conditioned reasoning plan, grounds the relevant entities in visual evidence, and excludes candidates that fail the specified constraints before evaluating spatial relations.

Comparing with APC. APC addresses the viewpoint change problem. It builds a sparse 3D abstraction, transforms it into the requested perspective, and passes it back to the VLM. TTSR targets beyond this: resolving a multi-condition question when the reference frame is known.

  1. multi-conditions remain checkable. APC's visual prompt renders objects as equal-sized colored cubes, so shape, color, material, and size are not from the rendered scene. TTSR keeps object appearance and verifies such conditions per candidate during reasoning.

  2. Relations and comparisons are computed not delegated. Take "Which white rectangular object on the left is closest to the desk?" A viewpoint change makes the layout easier to read, but the VLM must still filter by appearance, apply the position constraint, and compare distances. TTSR's plan encodes these dependencies. Verification establishes eligibility, and the executor ranks only the eligible candidates. Distance shows the point clearly: a rigid viewpoint transform preserves Euclidean distances, so once positions and the reference are known, the nearest object can be computed directly instead of being read off a rendering or a coordinate list.

Our ablation tests the point. The closest analogue to APC's numerical prompt is the "w/ Direct evidence" variant in Table 4. The VLM receives the same grounded entities, transformed into the requested reference frame (Eq. 5), but needs to evaluate the relations and comparisons itself. It reaches 42.0%, compared with 60.3% when TTSR computes relation-specific evidence. In the main comparison with the same Qwen3-VL-8B backbone, APC obtains 28.0% on MultiHop Spatial and 40.1% on ViewSpatial, compared with 60.3% and 48.9% for TTSR (Please refer to Table 2).

Comparing with GCA. GCA formalizes a task as a reference frame plus an objective and then has the VLM generate code to realize them. As we explained in Appendix A.2, its VLM is the tool orchestrator. The failure point that remains is the procedure itself: generated code can run successfully and still measure the wrong quantity. In TTSR, the plan is a structured object checked against the question by a lexical pass, which covers attribute, position, and ranking expressions from the question text. Two reasoning traces from our GCA runs show the difference:

Wrong reference. "Which object is farthest from the person in the image?" GCA's program places the person at the camera origin, ranks objects by np.linalg.norm(object_centroid), and returns lit desk lamp (D). TTSR binds the visible person as the reference, grounds them, and measures every candidate from that position. A better reconstruction would not fix GCA's choice of origin.

Eligible object. "Which square or rectangular object is closest to the cat on the top shelf?" (options: wall clock, wall phone, fireplace, DVD player). GCA computes distances from the cat to all four options and returns wall clock (A). The program runs without error but never checks shape. TTSR verifies shape, rejects the circular clock, and ranks the remaining candidates.

Geometry was not the problem in either case. The first question needed a different reference and the second needed a different comparison set. The second failure type is systematic. Removing question-conditioned attribute filtering from TTSR (Table 4, third row) lowers accuracy from 60.3% to 54.6%, and 2-hop accuracy from 61.3% to 50.5%. In the three benchmark comparison, TTSR outperforms GCA on all three benchmarks by a significant margin.

Comparing with MSSR. MSSR curates a compact set of relevant perceptual information for a reasoning agent. TTSR addresses a complementary bottleneck: the relevant information needs be checked and combined according to the question. We compile the question's requirements into per-candidate conditions, verify appearance against visual evidence, and run spatial comparisons only over eligible candidates. The final decision therefore does not depend on the agent noticing every requirement. With Qwen3-VL-8B, MSSR obtains 36.1% on MultiHop Spatial and 46.5% on ViewSpatial, compared with 60.3% and 48.9% for TTSR. TTSR also uses about a quarter of MSSR's GPU-hours on MultiHop Spatial, in Table 7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.