- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work designs a training-free visual-programming that takes monocular image and textual 3D spatial questions as input, and consolidate heterogeneous outputs from pretrained monocular perception models, such as segmentation, depth, camera calibration, and object orientation, into a unified object-centric 3D representation containing semantic identity, orientation and so on. A planner agent decomposes the input spatial query into high-level geometric operations, and a program agent compiles the plan into executable code that queries this representation and performs explicit geometric computation. The method is evaluated on 3DSRBench and Omni3D-Bench, with ablations suggesting that consolidating separate perception outputs into the proposed object-centric representation substantially improves downstream reasoning accuracy.
Critical strengths
-
Solid pipeline design and implementation. The major strength of the paper is in effective system integration. Rather than relying on a single new perception or reasoning module, authors organize several pretrained perception models, geometric operations, and LLM agents seamlessly into a single pipeline. Specifically, the unified object-centric 3D interface appears to enable complementary components to work together more effectively than simply exposing the same tools independently, leading to substantial system-level gains.
-
Consistency spatial representation in agentic spatial spatial reasoning. This work points out that simply providing an agent with multiple heterogeneous perception tools leaves substantial burden on the reasoning program to associate objects, reconcile coordinate systems, and compose geometric quantities. GeoState addresses this by consolidating these outputs into a unified object-centric 3D scene state in a shared coordinate system. This provides a coherent geometric substrate on which the agent can perform explicit spatial computations and reduces the need for integration during reasoning.
Critical weaknesses
-
A lack of strong novelty. This work appears to be an integration of existing monocular perception modules, and package them into an object-centric oriented-3D-box abstraction and expose it as a composite API to an LLM-generated program. Hence this work may not be sufficient as a novel contribution. Semantic label, 3D center, orientation are widely used in 3D perception, and most components of the pipeline are existing pretrained models or standard geometric procedures.
-
Insuffucient motivation. The motivation regarding the limitations of spatially fine-tuned VLMs is not sufficiently substantiated. The paper mentioned on the example in Fig. 1, even a spatially fine-tuned VLM may estimate an inaccurate 3D direction and subsequently make an error during an intermediate geometric computation. However, the experiments only compare final question-answering accuracy and do not directly evaluate the quality of the intermediate 3D quantities inferred by fine-tuned VLMs. Therefore, it is unclear whether their errors originate from 3D perception, geometric computation, or subsequent semantic reasoning.
Questions for authors
-
The authors should answer how well models like SpatialReasoner and VST can explicitly output intermediate variables, including object orientation, 3D direction, relative position, and local axes, and how are those results compared with annotations or geometric ground truth. The prediction results given by GeoState should also be compared with above mentioned values to show where GeoState’s primary advantage actually comes from.
-
The authors should answer which specific factors (standardizing the coordinate system, object identity, orientation convention, or interfaces) actually yields the performance gains (shown in Table 2). For instance, you could isolate the effects of standardizing the coordinate system or object association, remove orientation or metric scale information, or use unaligned outputs from individual modules. Such an analysis is necessary to substantiate the claim that the paper has truly identified the role of "representation consistency."
-
The authors should explicitly test the consistency and unified GeoState representation. For instance, one could examine whether the 3D position/orientation of an object remains stable when queried repeatedly, whether the relationships between different pairs of objects satisfy fundamental geometric constraints, whether the spatial relations are reciprocal (e.g., if A is to the left of B, then B must be to the right of A), whether the state of a given object remains consistent throughout a multi-step reasoning process, or how the results change when small perturbations are applied to camera intrinsics, depth, or orientation.
6 comments
Q2: Which specific factors (standardizing the coordinate system, object identity, orientation convention, or interfaces) actually yields the performance gains
Thank you for bringing this question. Because obj3d() couples geometric construction and coordinate conversion, simply removing a component can change the available geometric information or invalidate downstream attributes. We therefore complement the joint ablation in Table 2 with paired trajectory analysis, holding the underlying perception models fixed, to identify the errors associated with its gains and regressions.
Across 101 question instances where exactly one method answers correctly, obj3d() succeeds on 71 and separate tools succeed on 30. Through LLM-assisted categorization followed by human verification, we categorize the incorrect trajectories as follows:
| Error category | Separate tools fail; obj3d() succeeds | obj3d() fails; separate tools succeed | Net advantage |
|---|---|---|---|
| Geometric computation | 44 | 29 | +15 |
| Object association | 14 | 0 | +14 |
| Coordinate conventions | 13 | 1 | +12 |
| Total | 71 | 30 | +41 |
Geometric computation accounts for the largest number of separate-tool failures, but no single category dominates the net advantage. The associated net gains are similar in magnitude: +15 cases for geometric computation, +14 for object association, and +12 for coordinate conventions.
The inspected trajectories suggest three mechanisms:
-
Geometric computation:
obj3d()supplies constructed object geometry, reducing the processing that generated programs must implement. Its construction procedure includes, for example, filtering object point clouds before computing geometric attributes. This category covers geometric construction and calculation errors, excluding frame/convention errors. -
Object association: Some separate-tool programs introduce additional VQA or string checks that incorrectly select or reject an object. The corresponding successful
obj3d()programs avoid these checks, suggesting an indirect benefit from simpler object-reference handling. -
Coordinate conventions:
obj3d()supplies positions and orientation vectors in a shared frame, reducing the conversions required in generated code. Observed failures include comparing ray-relative orientation angles without transforming them into comparable vectors.
These observations support reduced geometric integration and auxiliary programming demands as an explanation for the improvement. The analysis provides a descriptive breakdown of the observed net advantage; it does not independently isolate the causal effect of each representation component. We will include the category definitions, error-assignment procedure, and representative trajectories in the revision.
Q3: the consistency and unified GeoState representation
Thank you for asking about representation consistency. We now directly evaluate intermediate-state stability across queries and answer consistency across input variants.
- Intermediate-state stability. We compare the estimated geometry of the same object in the same image across queries using 100 images from 3DSRBench with 238 queries. We evaluate two conditions: query-dependent boxes, where object localization is performed separately for each query, and fixed boxes, where the same object box is supplied across queries. The latter controls for variation in the input bounding box.
| State difference across queries | Query-dependent boxes: mean | Fixed boxes: mean |
|---|---|---|
| Center displacement | 0.026 m | 0.015 m |
| Center displacement / depth | 0.27% | 0.18% |
| Semantic-front angular difference | 0.94° | 0.15° |
| Up-axis angular difference | 0.08° | 0.04° |
The statistics are computed over valid query pairs for each object. Complete-attribute availability is 94.6% and 94.7%, respectively. The representation exhibits low mean variation across queries: mean center displacement is 1.5–2.6 cm, and mean semantic-front difference is 0.15–0.94°. Holding the boxes fixed further reduces the observed variation, consistent with object localization contributing to state variability.
- Answer consistency. In 3DSRBench evaluation protocol, we perform circular evaluation, which requires the model predicting consistently for different text option permutations and for original/flipped images. We analyze 120 question groups comprising 546 variants from 3DSRBench. We compare selected option content rather than option letters, independently of correctness. Permutation consistency requires agreement across option orders; flip consistency requires agreement under the verified mirrored-answer mapping; full consistency requires satisfying all applicable checks. Missing answers count as inconsistent.
| Method | Raw accuracy | Permutation consistency | Flip consistency | Full consistency | Fully consistent and correct |
|---|---|---|---|---|---|
| GeoState(GPT-4o, obj3d()) | 60.8 | 78.3 | 70.0 | 70.0 | 50.8 |
| GeoState(GPT-4o, separate tools) | 52.9 | 69.2 | 55.8 | 57.5 | 41.7 |
| SpatialReasoner | 54.9 | 78.3 | 55.0 | 59.2 | 41.7 |
| VST-7B-RL | 54.6 | 55.8 | 44.1 | 47.5 | 37.5 |
All values are percentages. Accuracy is measured over variants; consistency metrics are measured over question groups.
The controlled comparison shows improved answer consistency. Holding the GPT-4o backbone and underlying perception models fixed, obj3d() increases full consistency from 57.5% to 70.0% (+12.5 percentage points) and fully-consistent-and-correct performance from 41.7% to 50.8% (+9.1 points).
Together, these experiments provide direct evidence of low mean variation in GeoState’s intermediate geometry and improved downstream answer consistency relative to separate perception APIs. We will add both evaluations and their protocols to the revision.
We thank reviewer wLQE for acknowledging our work and providing valuable suggestions. Our responses are as follows:
We thank the reviewer for recognizing that exposing separate perception tools leaves the reasoning program responsible for reconciling their outputs. Our contribution is the idea of adopting an intermediate object-level geometric representation whose construction makes heterogeneous perception outputs jointly usable by generated reasoning programs. The popular visual programming approaches like VisProg and VADAR just expose different models as tools and do not utilize such an object-centric interface as GeoState. We do not claim novelty in the individual perception models, oriented boxes, or planning and repair strategies.
GeoState constructs object-level position, semantic orientation axes, and extent in a shared camera-coordinate frame. This moves geometric integration from generated reasoning code into a fixed construction procedure, allowing programs to compose spatial operations directly over compatible object attributes. The
obj3d()API is how this representation is constructed and exposed. The contribution concerns the representation and the computation it enables.The controlled experiments establish the value of this design choice. In Table 2, with the underlying perception models fixed and before adding the planner, replacing separate 3D APIs with the constructed representation improves overall accuracy from 35.8% to 51.7% (+15.9 points) and multi-object accuracy from 37.8% to 61.2% (+23.4 points). These results support the combined benefit of the representation and its construction procedure.
The representation also benefits other reasoning frameworks. VADAR equipped with
obj3d()achieves 53.4%, compared with 33.3% for the evaluated original VADAR configuration. Integratingobj3d()into SpatialClaw increases its Omni3D-Bench accuracy from 61.5 to 64.5 (+3.0 points). These integration experiments support applicability beyond GeoState’s own agent pipeline, complementing the controlled comparison in Table 2.We will sharpen the contribution statement around constructing compatible object geometry for direct use by reasoning programs, explain how its construction and use differ from prior representations, and present planning and repair as supporting components.
We agree that Figure 1 and final QA accuracy alone do not establish the quality of intermediate geometric predictions. Following your suggestion, we directly compare SpatialReasoner’s explicitly reported geometry with GeoState’s constructed geometry against dataset annotations.
We evaluate on 268 images from the SUN RGB-D validation portion of Omni3D, using objects whose category names are unique within each image. Both methods receive the RGB image and object names. We extract predicted object 3D position, camera parameters, semantic orientations from GeoState’s output and the chain of thought of SpatialReasoner. For evaluation we use annotated object identities, 3D boxes and object poses with LLM and human verified front direction. The results below use the common valid subset for each metric.
GeoState achieves better position and pairwise-geometry results, including a 14.7% improvement in inter-object direction Acc@30°. Because these relational quantities are computed using shared deterministic operations, their differences reflect the quality of the extracted geometric estimates rather than different downstream arithmetic procedures.
We will add this diagnostic and revise the motivation accordingly. The results support more accurate explicit geometry relative to SpatialReasoner on this evaluation; they do not establish that geometric estimation is the dominant source of its final QA errors.