acceptodds
Under review as a conference paper at ICLR 2027

PGM: Agentic Spatial Reasoning through Programmable Geometric Model

Abstract

Spatial reasoning from images requires measuring object relations and evaluating how they change under hypothetical scene modifications. Connecting these computations requires a scene representation that preserves object identity and keeps geometry consistent across successive operations. We introduce the Programmable Geometric Model (PGM), which couples reconstructed scene geometry with executable operations in a structured Python object. PGM organizes object point clouds, oriented bounding boxes, semantic orientations, and camera poses in a shared coordinate system. Its editable object states allow measurements and rendered views to reflect the same scene configuration after each modification. A Program Agent generates Python programs to query, inspect, and modify PGM, then uses execution results and diagnostics to guide subsequent programs. The scene state persists across executions, allowing dependent computations to build on earlier operations. This formulation supports quantitative queries and hypothetical reasoning through explicit geometric computation, with no task-specific training. Experiments on MMSI-Bench, MindCube-1K, and Omni3D-Bench show improvements across multiple agent backbones. Ablation studies further support the contributions of scene editing and iterative program interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Modality, Spatial Reasoning, Computer Vision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7qiFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the Programmable Geometric Model (PGM), which organizes object point clouds, oriented bounding boxes, semantic orientations, and camera poses recovered from images into a queryable and editable scene representation. A Program Agent performs geometric queries, object transformations, and renderings within a persistent Python environment, using execution feedback to guide subsequent reasoning. Its key design choice is to have queries and renderings share the updated object states, thereby supporting reasoning over hypothetical scene modifications. The paper reports improvements across multiple agent backbones on MMSI-Bench, MindCube-1K, and Omni3D-Bench, and uses ablation studies to analyze the contributions of scene editing operations and iterative program interaction.

Critical strengths
  1. PGM associates object identity, local point clouds, bounding boxes, and editable poses, so that geometric queries and renderings share the same updated state. This design helps reduce inconsistencies across different geometric representations during multi-step program execution.

  2. Modularity and transparency: the generated programs, numerical outputs, rendered views, and execution diagnostics provide a traceable computational process. The system uses frozen perception and language models, relying on neither additional spatial reasoning training data nor model fine-tuning. Its modular design also makes it easy to swap perception components or agent backbones.

  3. The paper evaluates multiple agent backbones on three spatial reasoning benchmarks, with separate ablations on scene operations and the interaction mechanism. The experiments provide reasonably broad evidence for the practicality of the system.

Critical weaknesses
  1. The novelty of the paper needs further clarification. Persistent Python execution, explicit geometric entities, and feedback-driven reasoning all have precedents in prior work. Compared with these works, the proposed agent loop appears to be a straightforward combination of existing components. Encapsulating different modalities to achieve “persistence and consistency at the scene-representation level” seems to little more than component integration and tricks. 
  2. The mechanisms related to object granularity and instance discovery are under-specified. The initial entities are constructed from SAM3 masks and their corresponding point clouds. The paper does not clearly explain how segmentation prompts are chosen, how object granularity is determined, or how missing objects are discovered and parts refined when reasoning requires them. If only a single granularity is used, how is it guaranteed to match the task's needs without omissions? If multiple granularities are used, one must consider that moving a whole object requires synchronously updating its parts, and modifying a part may require updating the whole-object representation. Can this method guarantee cross-object consistency in such cases? 
  3. Missing analysis of inference cost and comparisons under matched budgets. The paper does not report per-question end-to-end latency, cumulative input/output tokens, or the number of language model and perception tool calls. Since the method relies on multi-round program generation and feedback, I believe it is reasonable to expect the additional inference compute cost to be reported and weighed against the gains.
Questions for authors
  1. The paper claims to support adding objects, but it does not specify the geometric source of newly added entities, the input parameters, the conditions under which this is invoked, or an execution example, nor whether perception models are re-invoked in this case. 

  2. I hope the authors can support, in some form, an analysis and comparison of inference cost. 

  3. Can the method handle compositional relations or other more complex relations? Are there any special mechanisms for handling them?

Reviewer_K23LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a Programmable Geometric Model (PGM) for visual spatial question answering. Frozen perception models recover object geometry, orientations, and camera poses; these are organized into editable entities with a shared interface for measurement, transformation, and rendering. A language-model agent generates Python programs in a persistent execution loop to query and modify the scene. The paper reports improvements across MMSI-Bench, MindCube-1K, and Omni3D-Bench, together with operation and interaction ablations on Omni3D.

Critical strengths
  1. The reported improvements are substantial across multiple backbones and datasets. For example, GPT-4o improves from 30.3 to 40.0 on MMSI-Bench, from 42.3 to 68.1 on MindCube-1K, and from 42.9 to 52.3 on Omni3D. Comparisons with GPT-4o-based pySpatial also show improvements of 2.7, 5.4, and 8.1 points, respectively. These results provide evidence that the overall system is useful, beyond comparisons against unaugmented models alone.

  2. The fixed-perception ablations provide some support for editable scene state: the read-only configuration scores 46.7 versus 52.3 for the full model on Omni3D. Keeping object identity, geometry, and transformations consistent is a sensible system design, and the interface makes the intended reasoning process interpretable.

Critical weaknesses
  1. The distinct research contribution of the scene representation is not sufficiently isolated. The paper itself describes pySpatial as operating over reconstructed geometry and SpatialClaw as providing iterative execution in a persistent Python workspace. PGM's differentiator appears to be explicit object ownership and consistent propagation of edits across geometric representations. This is useful engineering, but the experiments do not establish how much it adds over an existing code agent equipped with the same perception outputs, geometric operations, and a conventional editable scene representation. Table 4's geometry-records configuration G has no reported result. The read-only versus editable comparison supports access to editing operations, but does not by itself isolate the advantage of PGM's particular organization of those operations. This limits my confidence in the significance of the claimed contribution.

  2. The experimental evidence does not adequately separate geometric modeling from additional inference effort or characterize when the extra machinery is warranted. The main paper gives a 300-second Python execution limit, but does not report actual end-to-end latency, reconstruction cost, token usage, model-call counts, or a matched-budget direct-reasoning baseline. Table 5 shows a 5.2-point improvement from single-pass to iterative execution, making this distinction consequential. Although newer base models are included, the cross-benchmark pySpatial comparisons rely on GPT-4o, leaving the relative advantage under stronger backbones less clear. Figure 3 illustrates successful PGM computations but does not show the same inputs answered by a direct VLM, so these examples do not establish a capability gap. The paper also acknowledges a Rotation regression for Qwen3-VL (87.0 to 83.8), without a corresponding failure analysis connecting errors to perception, geometry, or program reasoning. Together, these gaps leave the scope and practical value of the gains insufficiently characterized.

Questions for authors
  1. Can you clarify which baselines were rerun and whether perception outputs, prompts, model versions, and inference budgets were matched? What are the actual interaction limits and average end-to-end costs, including reconstruction and model calls?

  2. Why is the geometry-records result G missing from Table 4? Is there evidence comparing PGM with an agent given the same geometric information and editing primitives but without PGM's explicit entity/state organization? Such a comparison would materially affect my assessment of the central contribution.

  3. For the Figure 3 examples, what do the corresponding direct VLM and prior agentic baselines actually predict under the same inputs? Are the displayed programs illustrative sketches or actual execution traces? Please also clarify whether stronger-backbone, comparable-budget comparisons are available, and whether the Rotation regression can be attributed to reconstruction/orientation errors or subsequent reasoning.

These are requests to clarify or substantiate the current claims, rather than suggestions to expand the task to a new benchmark.

Reviewer_HEL4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed programmable geometric model, which is an agentic framework for solving spatial reasoning problems. It first use visual foundation models to reconstruct 3d scenes from visual inputs, then organized the objects in reconstruction scene into an editable representation. During reasoning, the LLM agent iteratively measure the spatial relations or manipulate object represetations to based on the requirement of the spatial reasoning tasks. Experiments on multiple benchmark datasets show that PGM is better than both specialized VLMs and other agentic frameworks

Critical strengths

It introduced a geometric scene representation with explictly eidtable states which enables reliable spatial measurements and scene modification during agentic reasoning across changing object configurations.

Critical weaknesses

The general idea of the method is to 1) first reconstruct 3d point clouds of the scene from rgb inputs and 2) organize the reconstruction into editable geometric scene representation 3) and instruct LLM agent to manipulate the representation for reasoning. 1) and 3) have been extensively explored in many existing works (e.g. python execution and iterative feedback are already used in spatial claw). So the only novelty lie in 2). However, from my perspective, the proposed representation mainly combines basic geometric operations, such as moving and rotating objects, which can already be easily implemented in existing Python-based frameworks. Therefore, its methodological novelty appears limited.

If we provide SpatialClaw with tools of object-level geometric operations or simply prompt the agent to do object-level geometric operations if necessary, would the performance be similar to the method in the paper

Questions for authors

see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.