acceptodds
Under review as a conference paper at ICLR 2027

GeoState: Queryable 3D Scene State for Compositional Spatial Reasoning

Abstract

Reliable 3D spatial reasoning from images requires not only accurate perception, but also representations that support explicit geometric computation. Existing vision-language models often entangle perception, geometry, and reasoning within a single language-based process. In contrast, visual-programming approaches first extract heterogeneous perception outputs such as masks, depth, and camera parameters and subsequently generate reasoning programs. We argue that this intermediate representation is a critical but underexplored design choice in visual programming. We introduce GeoState, a geometry-grounded visual-programming framework that consolidates heterogeneous monocular perception outputs into a reasoning-friendly, object-centric 3D representation. Each object is represented by its semantic identity, 3D position, orientation, and extent in a shared coordinate frame, enabling spatial relations to be computed directly through simple geometric operations. GeoState further uses a planner-guided program synthesis to decompose spatial queries into high-level geometric operations and executable code over these representations. Without task-specific spatial reasoning supervision or fine-tuning, GeoState substantially outperforms prior visual-programming approaches and achieves performance competitive with or exceeding spatially fine-tuned VLMs on stratified subsets of 3DSRBench and Omni3D-Bench. Controlled ablations further show that, with the underlying perception modules fixed, consolidating their outputs into object-centric geometry consistently improves reasoning over directly composing heterogeneous model-specific representations. These results highlight the importance of reasoning-friendly intermediate representations for reliable executable 3D spatial reasoning.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
visual programming, spatial reasoning, 3D

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wLQEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work designs a training-free visual-programming that takes monocular image and textual 3D spatial questions as input, and consolidate heterogeneous outputs from pretrained monocular perception models, such as segmentation, depth, camera calibration, and object orientation, into a unified object-centric 3D representation containing semantic identity, orientation and so on. A planner agent decomposes the input spatial query into high-level geometric operations, and a program agent compiles the plan into executable code that queries this representation and performs explicit geometric computation. The method is evaluated on 3DSRBench and Omni3D-Bench, with ablations suggesting that consolidating separate perception outputs into the proposed object-centric representation substantially improves downstream reasoning accuracy.

Critical strengths
  1. Solid pipeline design and implementation. The major strength of the paper is in effective system integration. Rather than relying on a single new perception or reasoning module, authors organize several pretrained perception models, geometric operations, and LLM agents seamlessly into a single pipeline. Specifically, the unified object-centric 3D interface appears to enable complementary components to work together more effectively than simply exposing the same tools independently, leading to substantial system-level gains.

  2. Consistency spatial representation in agentic spatial spatial reasoning. This work points out that simply providing an agent with multiple heterogeneous perception tools leaves substantial burden on the reasoning program to associate objects, reconcile coordinate systems, and compose geometric quantities. GeoState addresses this by consolidating these outputs into a unified object-centric 3D scene state in a shared coordinate system. This provides a coherent geometric substrate on which the agent can perform explicit spatial computations and reduces the need for integration during reasoning.

Critical weaknesses
  1. A lack of strong novelty. This work appears to be an integration of existing monocular perception modules, and package them into an object-centric oriented-3D-box abstraction and expose it as a composite API to an LLM-generated program. Hence this work may not be sufficient as a novel contribution. Semantic label, 3D center, orientation are widely used in 3D perception, and most components of the pipeline are existing pretrained models or standard geometric procedures.

  2. Insuffucient motivation. The motivation regarding the limitations of spatially fine-tuned VLMs is not sufficiently substantiated. The paper mentioned on the example in Fig. 1, even a spatially fine-tuned VLM may estimate an inaccurate 3D direction and subsequently make an error during an intermediate geometric computation. However, the experiments only compare final question-answering accuracy and do not directly evaluate the quality of the intermediate 3D quantities inferred by fine-tuned VLMs. Therefore, it is unclear whether their errors originate from 3D perception, geometric computation, or subsequent semantic reasoning.

Questions for authors
  1. The authors should answer how well models like SpatialReasoner and VST can explicitly output intermediate variables, including object orientation, 3D direction, relative position, and local axes, and how are those results compared with annotations or geometric ground truth. The prediction results given by GeoState should also be compared with above mentioned values to show where GeoState’s primary advantage actually comes from.

  2. The authors should answer which specific factors (standardizing the coordinate system, object identity, orientation convention, or interfaces) actually yields the performance gains (shown in Table 2). For instance, you could isolate the effects of standardizing the coordinate system or object association, remove orientation or metric scale information, or use unaligned outputs from individual modules. Such an analysis is necessary to substantiate the claim that the paper has truly identified the role of "representation consistency."

  3. The authors should explicitly test the consistency and unified GeoState representation. For instance, one could examine whether the 3D position/orientation of an object remains stable when queried repeatedly, whether the relationships between different pairs of objects satisfy fundamental geometric constraints, whether the spatial relations are reciprocal (e.g., if A is to the left of B, then B must be to the right of A), whether the state of a given object remains consistent throughout a multi-step reasoning process, or how the results change when small perturbations are applied to camera intrinsics, depth, or orientation.

6 comments

We thank reviewer wLQE for acknowledging our work and providing valuable suggestions. Our responses are as follows:

W1: Lack of strong novelty

We thank the reviewer for recognizing that exposing separate perception tools leaves the reasoning program responsible for reconciling their outputs. Our contribution is the idea of adopting an intermediate object-level geometric representation whose construction makes heterogeneous perception outputs jointly usable by generated reasoning programs. The popular visual programming approaches like VisProg and VADAR just expose different models as tools and do not utilize such an object-centric interface as GeoState. We do not claim novelty in the individual perception models, oriented boxes, or planning and repair strategies.

GeoState constructs object-level position, semantic orientation axes, and extent in a shared camera-coordinate frame. This moves geometric integration from generated reasoning code into a fixed construction procedure, allowing programs to compose spatial operations directly over compatible object attributes. The obj3d() API is how this representation is constructed and exposed. The contribution concerns the representation and the computation it enables.

The controlled experiments establish the value of this design choice. In Table 2, with the underlying perception models fixed and before adding the planner, replacing separate 3D APIs with the constructed representation improves overall accuracy from 35.8% to 51.7% (+15.9 points) and multi-object accuracy from 37.8% to 61.2% (+23.4 points). These results support the combined benefit of the representation and its construction procedure.

The representation also benefits other reasoning frameworks. VADAR equipped with obj3d() achieves 53.4%, compared with 33.3% for the evaluated original VADAR configuration. Integrating obj3d() into SpatialClaw increases its Omni3D-Bench accuracy from 61.5 to 64.5 (+3.0 points). These integration experiments support applicability beyond GeoState’s own agent pipeline, complementing the controlled comparison in Table 2.

We will sharpen the contribution statement around constructing compatible object geometry for direct use by reasoning programs, explain how its construction and use differ from prior representations, and present planning and repair as supporting components.

W2/Q1: Direct evaluation of intermediate geometry from VLM and GeoState.

We agree that Figure 1 and final QA accuracy alone do not establish the quality of intermediate geometric predictions. Following your suggestion, we directly compare SpatialReasoner’s explicitly reported geometry with GeoState’s constructed geometry against dataset annotations.

We evaluate on 268 images from the SUN RGB-D validation portion of Omni3D, using objects whose category names are unique within each image. Both methods receive the RGB image and object names. We extract predicted object 3D position, camera parameters, semantic orientations from GeoState’s output and the chain of thought of SpatialReasoner. For evaluation we use annotated object identities, 3D boxes and object poses with LLM and human verified front direction. The results below use the common valid subset for each metric.

Intermediate quantityMetricGeoStateSpatialReasoner
3D Object positionHorizontal center error ↓0.33 m0.51 m
3D Object positionGravity-aligned forward-distance AbsRel ↓0.120.20
Relative 3D positionPairwise center-distance error ↓0.24 m0.32 m
Inter-object directionAngular error ↓ / Acc@30° ↑16.7° / 66.7%28.6° / 52.0%
Depth orderAccuracy ↑85.2%76.9%
Semantic frontHeading error ↓ / Acc@30° ↑13.9° / 75.5%35.2° / 44.0%

GeoState achieves better position and pairwise-geometry results, including a 14.7% improvement in inter-object direction Acc@30°. Because these relational quantities are computed using shared deterministic operations, their differences reflect the quality of the extracted geometric estimates rather than different downstream arithmetic procedures.

We will add this diagnostic and revise the motivation accordingly. The results support more accurate explicit geometry relative to SpatialReasoner on this evaluation; they do not establish that geometric estimation is the dominant source of its final QA errors.

Q2: Which specific factors (standardizing the coordinate system, object identity, orientation convention, or interfaces) actually yields the performance gains

Thank you for bringing this question. Because obj3d() couples geometric construction and coordinate conversion, simply removing a component can change the available geometric information or invalidate downstream attributes. We therefore complement the joint ablation in Table 2 with paired trajectory analysis, holding the underlying perception models fixed, to identify the errors associated with its gains and regressions.

Across 101 question instances where exactly one method answers correctly, obj3d() succeeds on 71 and separate tools succeed on 30. Through LLM-assisted categorization followed by human verification, we categorize the incorrect trajectories as follows:

Error categorySeparate tools fail; obj3d() succeedsobj3d() fails; separate tools succeedNet advantage
Geometric computation4429+15
Object association140+14
Coordinate conventions131+12
Total7130+41

Geometric computation accounts for the largest number of separate-tool failures, but no single category dominates the net advantage. The associated net gains are similar in magnitude: +15 cases for geometric computation, +14 for object association, and +12 for coordinate conventions.

The inspected trajectories suggest three mechanisms:

  • Geometric computation: obj3d() supplies constructed object geometry, reducing the processing that generated programs must implement. Its construction procedure includes, for example, filtering object point clouds before computing geometric attributes. This category covers geometric construction and calculation errors, excluding frame/convention errors.

  • Object association: Some separate-tool programs introduce additional VQA or string checks that incorrectly select or reject an object. The corresponding successful obj3d() programs avoid these checks, suggesting an indirect benefit from simpler object-reference handling.

  • Coordinate conventions: obj3d() supplies positions and orientation vectors in a shared frame, reducing the conversions required in generated code. Observed failures include comparing ray-relative orientation angles without transforming them into comparable vectors.

These observations support reduced geometric integration and auxiliary programming demands as an explanation for the improvement. The analysis provides a descriptive breakdown of the observed net advantage; it does not independently isolate the causal effect of each representation component. We will include the category definitions, error-assignment procedure, and representative trajectories in the revision.

Q3: the consistency and unified GeoState representation

Thank you for asking about representation consistency. We now directly evaluate intermediate-state stability across queries and answer consistency across input variants.

  1. Intermediate-state stability. We compare the estimated geometry of the same object in the same image across queries using 100 images from 3DSRBench with 238 queries. We evaluate two conditions: query-dependent boxes, where object localization is performed separately for each query, and fixed boxes, where the same object box is supplied across queries. The latter controls for variation in the input bounding box.
State difference across queriesQuery-dependent boxes: meanFixed boxes: mean
Center displacement0.026 m0.015 m
Center displacement / depth0.27%0.18%
Semantic-front angular difference0.94°0.15°
Up-axis angular difference0.08°0.04°

The statistics are computed over valid query pairs for each object. Complete-attribute availability is 94.6% and 94.7%, respectively. The representation exhibits low mean variation across queries: mean center displacement is 1.5–2.6 cm, and mean semantic-front difference is 0.15–0.94°. Holding the boxes fixed further reduces the observed variation, consistent with object localization contributing to state variability.

  1. Answer consistency. In 3DSRBench evaluation protocol, we perform circular evaluation, which requires the model predicting consistently for different text option permutations and for original/flipped images. We analyze 120 question groups comprising 546 variants from 3DSRBench. We compare selected option content rather than option letters, independently of correctness. Permutation consistency requires agreement across option orders; flip consistency requires agreement under the verified mirrored-answer mapping; full consistency requires satisfying all applicable checks. Missing answers count as inconsistent.
MethodRaw accuracyPermutation consistencyFlip consistencyFull consistencyFully consistent and correct
GeoState(GPT-4o, obj3d())60.878.370.070.050.8
GeoState(GPT-4o, separate tools)52.969.255.857.541.7
SpatialReasoner54.978.355.059.241.7
VST-7B-RL54.655.844.147.537.5

All values are percentages. Accuracy is measured over variants; consistency metrics are measured over question groups.

The controlled comparison shows improved answer consistency. Holding the GPT-4o backbone and underlying perception models fixed, obj3d() increases full consistency from 57.5% to 70.0% (+12.5 percentage points) and fully-consistent-and-correct performance from 41.7% to 50.8% (+9.1 points).

Together, these experiments provide direct evidence of low mean variation in GeoState’s intermediate geometry and improved downstream answer consistency relative to separate perception APIs. We will add both evaluations and their protocols to the revision.

Reviewer_hbtNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents GeoState, a training-free visual-programming framework for answering 3D spatial questions from a single image. It combines pretrained perception models to construct a queryable, object-centric scene representation. A planner breaks each question into reasoning steps, which a program agent translates into executable code, using execution feedback to revise faulty programs. Evaluations on subsets of 3DSRBench and Omni3D-Bench report gains over prior visual-programming methods and results competitive with spatially fine-tuned VLMs.

Critical strengths
  1. GeoState exposes perception outputs through a unified object-centric interface, simplifying executable spatial reasoning and making intermediate geometric quantities inspectable.

  2. The 2-agent framework is intuitive and experiments supported the design choice.

Critical weaknesses
  1. The experimental design doesn't properly demonstrate the effectiveness of the proposed representation. Despite the fact that multi-object category sees substantial improvement, there is no analysis on the reason behind it.

  2. Missing baselines to compare against. For example, ViperGPT [1] and PySpatial [2].

  3. Limited comparison with stronger contemporary VLMs. The baselines mainly include GPT-4o, Gemini 2.0 Flash, and Qwen2.5-VL-based models, which are outdated in current date, stronger baselines should be used and investiagted.

[1] Surís, Dídac, Sachit Menon, and Carl Vondrick. "Vipergpt: Visual inference via python execution for reasoning." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[2] Luo, Zhanpeng, et al. "pyspatial: Generating 3d visual programs for zero-shot spatial reasoning." arXiv preprint arXiv:2603.00905 (2026).

Questions for authors
  1. Can you provide targeted analyses on what benefit does the representation provided?

  2. Does GeoState retain its advantage over stronger contemporary VLMs?

3 comments

We thank reviewer hbtN for acknowledging our work and providing valuable suggestions. Our responses are as follows:

W1&Q1: Analysis on what benefit does the representation provided

Thank you for asking what drives the representation’s benefit. We analyzed the Table 2 comparison using the same underlying perception models. obj3d() moves geometric construction and coordinate conversion from generated programs into a fixed object-level representation. Our trajectory analysis examines which errors this reduces.

Across 101 question instances where exactly one method answers correctly, obj3d() succeeds on 71 and separate tools succeed on 30. Using LLM and human verification, we categorize the error in the incorrect program:

Error categorySeparate tools fail; obj3d() succeedsobj3d() fails; separate tools succeedNet advantage
Geometric computation4429+15
Object association140+14
Coordinate conventions131+12
Total7130+41

The inspected programs illustrate three mechanisms:

  1. More reliable geometric computation (+15 net cases). obj3d() provides precomputed object geometry, reducing the geometric operations that generated programs must implement. For example, obj3d() filters object point clouds to reduce outliers before computing geometric attributes.

  2. Simpler object-reference handling (+14 net cases). In the inspected trajectories, separate-tool programs sometimes introduce additional VQA or string checks that incorrectly select or reject an object. The corresponding obj3d() programs avoid these checks, suggesting an indirect benefit from simpler object-handling logic.

  3. Fewer coordinate-conversion errors (+12 net cases). obj3d() expresses positions and orientation vectors in a shared frame. Generated programs can therefore compare these attributes directly, avoiding errors such as comparing ray-relative orientation angles without the required transformations.

These results support a specific explanation: the representation reduces the geometric integration and auxiliary logic that the reasoning program must implement.

For multi-object questions, 28 of the 34 improved instances involve geometric computation (18) or coordinate conventions (10); the remaining six involve object association. These cases illustrate the benefit of compatible object geometry when reasoning requires comparing multiple objects’ positions and orientations.

Overall, the analysis supports reduced geometric integration and auxiliary programming demands as an explanation for the gains.

W2: missing ViperGPT and PySpatial baseline

Thank you for suggesting these comparisons. We have evaluated ViperGPT and PySpatial using their official implementations on the 3DSRBench subset and Omni3D benchmark used in our evaluation.

Questions3DSRBenchOmni3D
ViperGPT20.126.7
pySpatial29.244.2
GeoState59.050.2

GeoState outperforms both requested baselines, exceeding PySpatial by 29.8 percentage points on 3DSRBench and 6.0 points on Omni3D. We will include these results and the evaluation configurations in the revised manuscript.

W3&Q2: Comparison with stronger contemporary VLM

Thank you for highlighting this evaluation gap. Our original GPT4-o backbone choice facilitated fair comparison with prior visual-programming work, but it does not establish competitiveness against contemporary VLMs. We therefore evaluated the most recent open-source VLM Qwen3.8-27B both through direct inference and as GeoState’s reasoning backbone, keeping GeoState’s perception tools unchanged.

Model3DSRBenchOmni3D
Qwen3.8-27b55.158.9
GeoState(gpt-4o)59.050.2
GeoState(Qwen3.8-27b)60.759.7

On 3DSRBench, GeoState improves over direct inference with the same Qwen backbone by 5.6 percentage points. Replacing GPT-4o with Qwen also improves GeoState’s overall accuracy from 59.0% to 60.7% on 3DSRBench and from 50.2% to 59.7% on Omni3D, despite retaining the same perception tools.

The benefits vary by question type. Relative to direct Qwen inference, GeoState(Qwen3.8-27b) improves multi-object accuracy from 53.1% to 58.2% and orientation accuracy from 48.3% to 63.3%, while height accuracy decreases from 47.4% to 42.1% and location accuracy remains unchanged.

These results show that GeoState remains beneficial with the newer backbone under our evaluation protocol, while identifying categories where explicit geometric reasoning does not improve performance. We will include these comparisons and their inference settings in the revision.

ModelHeightLocationMulti-objectOrientationOverall
Qwen3.8-27B47.468.448.353.155.1
GeoState(gpt-4o)47.471.936.767.359.0
GeoState(Qwen3.8-27B)42.168.463.358.260.7
MethodNumeric (CT)Numeric (Other)Y/NMulti-choiceTotal
qwen3.8-27b32.941.969.370.958.9
GeoState(GPT-4o)24.235.864.059.350.2
GeoState(qwen3.8-27b)40.034.272.065.159.7
AuthorsOpenReview ↗

Follow-up on Author Response and Updated Manuscript (Highlighted in Blue)

Dear Reviewer hbtN,

As the discussion deadline is approaching, we wanted to kindly follow up to let you know that we have uploaded our revised manuscript with all newly added experiments and analyses highlighted in blue, directly addressing each of your questions:

  1. Targeted Analysis of obj3d() Gains (Section 4.3, Table 2(b)): Added the paired trajectory analysis across the 101 discordant instances on 3DSRBench, detailing how obj3d() reduces errors in geometric computation (+15 net cases), object association (+14), and coordinate conventions (+12).
  2. Comparison with ViperGPT, PySpatial, and Other Agentic Baselines (Section 4.1, Section 4.2, & Table 1): Added official-implementation evaluations of ViperGPT and PySpatial (alongside MSSR, GCA, SpatialAgent-ReAct, and SpatialClaw) on both 3DSRBench and Omni3D-Bench, where GeoState outperforms ViperGPT and PySpatial on both benchmarks.
  3. Evaluation with Stronger Contemporary VLM Backbone (Section 4.2 & Table 1): Added direct inference and GeoState backbone evaluations with Qwen3.8-27B, showing that GeoState (Qwen3.8-27B) improves over direct Qwen3.8-27B inference on both benchmarks, with substantial gains on orientation (+15.0 points) and multi-object reasoning (+5.1 points).

Please let us know if our responses and the updated manuscript have addressed your concerns, or if there are any remaining questions we can clarify before the discussion period concludes. We would be very glad to discuss further. Thank you again for your time and helpful feedback!

Best regards,

Authors

Reviewer_FtMQOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

GeoState is a training-free visual programming framework for single-image 3D spatial reasoning. It integrates pretrained perception models into an object-centric geometric interface, then uses a planner and a program agent to generate and execute spatial reasoning programs.

Critical strengths
  1. The controlled ablations support the practical benefit of consolidating heterogeneous perception outputs into a unified geometric interface.

  2. Supplementary experiments include comparisons using the same geometric API, different agent backbones, and runtime and failure analyses, providing useful evidence about system behavior.

Critical weaknesses

The methodological novelty appears limited. The framework combines established components: pretrained perception tools, object-level 3D geometry, planner–program decomposition, and execution-based repair. The experiments demonstrate the usefulness of this integration, but do not sufficiently establish a substantive methodological advance over existing agentic spatial reasoning approaches.

The closest related work is inadequately discussed and compared. Despite visual programming being the central approach, the related-work section emphasizes VLM-based methods and provides limited coverage of recent programmatic approaches. Relevant work such as GCA [2], MSSR [2], SpatialClaw [3], and SpatialAgent [4] is missing. These methods already combines agents with distinct responsibilities and external spatial perception tools. Without clearer distinctions and stronger comparisons, the incremental contribution remains difficult to assess.

[1] Geometrically-Constrained Agent for Spatial Reasoning.

[2] Pursuing Minimal Sufficiency in Spatial Reasoning.

[3] SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning.

[4] SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence. (Introduces SpatialAgent.)

Questions for authors

I will consider raising score if questions are properly addressed:

What is the substantive methodological contribution beyond packaging existing perception outputs into a unified API and applying a conventional planning, programming, and repair pipeline?

How does GeoState differ from related works, particularly in its organization of geometric information and division of agent responsibilities? Can comparisons with these methods substantiate the claimed advantage?

7 comments

We thank reviewer FtMQ for acknowledging our work and providing valuable suggestions. Our responses are as follows:

W1&Q1: limited methodological novelty

We thank the reviewer for recognizing the controlled ablations and the practical benefit of our geometric interface. We agree that the individual perception models, geometric primitives, and planning/repair strategies are established components. Our central contribution concerns exploring an intermediate object-level geometric representation that makes heterogeneous perception outputs jointly usable by generated reasoning programs. To our knowledge, the related approaches do not explore the same jointly constructed object-state interface as GeoState.

GeoState constructs object-level position, semantic orientation axes, and extent in a shared camera-coordinate frame. This moves geometric integration out of the generated reasoning program, allowing the program to compose spatial operations directly over compatible object attributes. The obj3d() API constructs and exposes this representation.

The controlled evidence distinguishes this contribution from simply presenting different perception tools or agents. In Table 2, before adding the planner, replacing separate 3D APIs with the consolidated representation improves overall accuracy from 35.8% to 51.7% (+15.9 points) and multi-object accuracy from 37.8% to 61.2% (+23.4 points).

The representation also benefits other reasoning frameworks. In Supplementary Table 8, VADAR+obj3d() experiment achieves 53.4%, compared with 33.3% for the evaluated original VADAR configuration. Furthermore, we tried integrating obj3d() into SpatialClaw and increased its Omni3D-Bench score from 61.5 to 64.5 (+3.0 points). These results support the applicability of our composite geometric interface beyond GeoState’s own agent pipeline.

We will clarify that our contribution lies in the geometric representation and its construction procedure, with planning and repair serving as supporting components. We will also expand the comparison with the methods identified by the reviewer, acknowledging their existing geometric representations and specifying how their construction and use differ from GeoState.

W2&Q2: Discussion and comparison with related works

We thank the reviewer for highlighting related agentic VLM works. We clarify their relationship to GeoState through the reasoning setting, the geometric representation, and an additional integration experiment.

  1. Difference in reasoning setting. GeoState follows an answer-producing visual-programming setting, as in VADAR, VisProg, and ViperGPT: an LLM generates a program whose execution may call external tools and produces the answer. Its explicit geometric computations and answer derivation are therefore inspectable. The agentic frameworks you mentioned organize reasoning differently:
  • GCA formalizes the task’s reference frame and objective, then guides tool use and code generation through these constraints iteratively.

  • MSSR iterates between a perception agent that performs visual programming to extract geometric evidence and a reasoning agent that requests missing information and removes redundant evidence.

  • SpatialClaw interleaves code execution, inspection of intermediate results, and revision within a persistent Python environment.

  • SpatialAgent orchestrates specialized perception tools through Plan-Execute or ReAct workflows. All these agentic frameworks utilize external tools to produce new evidences to help LLM or VLM make better predictions. Visual programming can be a component of these broader frameworks. Our distinction therefore concerns how program execution and model reasoning are organized, rather than treating our work and agentic reasoning as mutually exclusive categories.

  1. Difference in geometric representation. GeoState’s central contribution concerns the representation on which generated programs operate. Its fixed construction procedure integrates segmentation, depth, calibration, and orientation predictions into object position, semantic orientation axes, and extent along those axes in a shared camera-coordinate frame. The obj3d() API constructs and exposes these compatible attributes.

In contrast, MSSR, SpatialAgent, GCA and SpatialClaw expose separate perception tools to infer these information separately. GeoState jointly constructs the object geometry, including reconciling model-specific conventions and fitting extent along semantic axes, so generated programs can directly compose spatial computations over it.

Accordingly, GeoState addresses a different design question from GCA’s task constraints, MSSR’s evidence selection, SpatialClaw’s action interface, and SpatialAgent’s tool orchestration. The object-level abstraction can also accommodate additional properties by defining their association with the object and their conventions and semantics.

  1. Compatibility with SpatialClaw. Our representation can also complement an iterative agentic architecture. On our evaluated Omni3D-Bench setting, incorporating obj3d() improves SpatialClaw’s total score from 61.5 to 64.5 (+3.0 points). This demonstrates that the utility of our composite geometric interface extends beyond GeoState’s own planner–program pipeline and can benefit an agent that already supports adaptive code execution and intermediate feedback.
MethodNumeric (CT)Numeric (Other)Y/NMulti-choiceTotal
SpatialClaw41.453.070.669.761.5
SpatialClaw+obj3d()44.252.574.672.064.5

We will revise the related-work discussion to make these distinctions and the demonstrated compatibility explicit.

Thanks for the detailed responses. I appreciate the additional experiments and clarifications regarding the methodological contribution and comparisons with related agentic spatial reasoning approaches. These results address my main concerns.

I hope the authors will incorporate these clarifications and experimental results into the revised manuscript and clearly highlight the changes. I would be happy to raise my rating accordingly.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Additional Evaluations

Dear Area Chair and Reviewers,

As the discussion period draws to a close, we thank you for your constructive feedback throughout the review process. We have uploaded our revised manuscript with additions highlighted in blue. For convenience, we summarize the principal updates made in response to your feedback below:

  • Clarified contribution and motivation (Abstract, Secs. 1 and 3 — Reviewers FtMQ, wLQE): We clarify that our core contribution is an object-centric geometric representation and its construction from heterogeneous perception outputs, exposed through obj3d(). Planning and repair serve as supporting components.
  • Expanded baselines and agentic comparisons (Secs. 2 and 4.2, Table 1 — Reviewers FtMQ, hbtN): We expand the related-work discussion and add experimental comparisons with ViperGPT, Qwen3.8-27B, pySpatial, MSSR, GCA, SpatialAgent-ReAct, and SpatialClaw. These comparisons demonstrate GeoState’s strong overall performance, while further gains from integrating obj3d() into the higher-performing SpatialClaw framework support our representation’s usefulness beyond our own pipeline.
  • Targeted trajectory, 3D geometry, and consistency diagnostics (Sec. 4.3, Tables 2–3, App. E — Reviewers wLQE, hbtN): We add paired trajectory error analysis, direct evaluation of GeoState’s and SpatialReasoner’s intermediate geometry against SUN RGB-D annotations, and evaluations of cross-query state stability and answer consistency under option permutations and image flips.

Thank you again for your time and insightful suggestions, which have helped us strengthen the manuscript and its evaluation.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.