- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces the Programmable Geometric Model (PGM), which organizes object point clouds, oriented bounding boxes, semantic orientations, and camera poses recovered from images into a queryable and editable scene representation. A Program Agent performs geometric queries, object transformations, and renderings within a persistent Python environment, using execution feedback to guide subsequent reasoning. Its key design choice is to have queries and renderings share the updated object states, thereby supporting reasoning over hypothetical scene modifications. The paper reports improvements across multiple agent backbones on MMSI-Bench, MindCube-1K, and Omni3D-Bench, and uses ablation studies to analyze the contributions of scene editing operations and iterative program interaction.
Critical strengths
-
PGM associates object identity, local point clouds, bounding boxes, and editable poses, so that geometric queries and renderings share the same updated state. This design helps reduce inconsistencies across different geometric representations during multi-step program execution.
-
Modularity and transparency: the generated programs, numerical outputs, rendered views, and execution diagnostics provide a traceable computational process. The system uses frozen perception and language models, relying on neither additional spatial reasoning training data nor model fine-tuning. Its modular design also makes it easy to swap perception components or agent backbones.
-
The paper evaluates multiple agent backbones on three spatial reasoning benchmarks, with separate ablations on scene operations and the interaction mechanism. The experiments provide reasonably broad evidence for the practicality of the system.
Critical weaknesses
- The novelty of the paper needs further clarification. Persistent Python execution, explicit geometric entities, and feedback-driven reasoning all have precedents in prior work. Compared with these works, the proposed agent loop appears to be a straightforward combination of existing components. Encapsulating different modalities to achieve “persistence and consistency at the scene-representation level” seems to little more than component integration and tricks.
- The mechanisms related to object granularity and instance discovery are under-specified. The initial entities are constructed from SAM3 masks and their corresponding point clouds. The paper does not clearly explain how segmentation prompts are chosen, how object granularity is determined, or how missing objects are discovered and parts refined when reasoning requires them. If only a single granularity is used, how is it guaranteed to match the task's needs without omissions? If multiple granularities are used, one must consider that moving a whole object requires synchronously updating its parts, and modifying a part may require updating the whole-object representation. Can this method guarantee cross-object consistency in such cases?
- Missing analysis of inference cost and comparisons under matched budgets. The paper does not report per-question end-to-end latency, cumulative input/output tokens, or the number of language model and perception tool calls. Since the method relies on multi-round program generation and feedback, I believe it is reasonable to expect the additional inference compute cost to be reported and weighed against the gains.
Questions for authors
-
The paper claims to support adding objects, but it does not specify the geometric source of newly added entities, the input parameters, the conditions under which this is invoked, or an execution example, nor whether perception models are re-invoked in this case.
-
I hope the authors can support, in some form, an analysis and comparison of inference cost.
-
Can the method handle compositional relations or other more complex relations? Are there any special mechanisms for handling them?