acceptodds
Withdrawn from ICLR 2027

World As a Program: Programmable Spatial Reasoning for Multimodal LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Although MLLMs have advanced rapidly, spatial reasoning remains limited by implicit, geometrically unconstrained representations, while explicit tool-based approaches often rely on external perception modules for object grounding and geometric acquisition, underusing native visual-semantic capabilities. Recent advances in 3D reconstruction can recover explicit 3D representations of real-world environments, where scene entities are directly accessible and manipulable, suggesting 3D assets as a natural programmable substrate for spatial reasoning. This motivates a natural question: Can explicit 3D world states be made directly programmable for MLLMs? To this end, we propose WoRld As a Program (WRAP), a framework that converts standard video observations into an addressable and executable 3D scene state for MLLMs, where visual objects are exposed as Grounded Variables for 3D Program Reasoning. WRAP explicitly binds visual object instances to their 3D entities as grounded variables, then enables the MLLM to generate executable reasoning programs over these variables while delegating only deterministic computation to an external executor. Extensive experiments on both video spatial reasoning and 3D scene editing benchmarks demonstrate the effectiveness and generality of WRAP. Our method achieves state-of-the-art (SOTA) performance on ReVSI, improving video spatial reasoning accuracy by up to 5.2% over prior SOTA methods, while also outperforming existing approaches on EditRoom-DB test set, achieving an absolute IoU improvement of 0.19 over prior methods. These results demonstrate the programmable 3D representations benefit both video spatial reasoning and scene manipulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Model, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iVR5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WoRld As a Program, a framework for spatial reasoning and 3D scene editing with multimodal language models. It associates visually tagged objects with reconstructed 3D point clouds, exposes these objects as variables, and generates executable programs for geometric computation or scene modification. An adaptive mechanism selects between direct visual reasoning and program-based reasoning. The model is trained through spatial grounding followed by spatial program learning. Experiments cover three video spatial reasoning benchmarks and EditRoom-DB, supplemented by component ablations and analyses of grounding and program correctness.

Critical strengths
  1. The framework provides a clear connection between visual object references and executable geometric representations. Persistent object identifiers make the generated programs interpretable and allow geometric computations to operate on a shared scene representation.
  2. The component ablations and oracle-substitution experiments provide useful diagnostic evidence. In particular, replacing predicted object grounding with oracle grounding increases accuracy from 68% to 92%, whereas replacing generated programs with oracle programs increases it to 77%. This helps identify grounding as an important remaining bottleneck.
Critical weaknesses
  1. The incremental contribution over existing program-based spatial reasoning methods needs stronger experimental isolation.Direct comparisons with closely related methods such as pySpatial and VADAR, or controlled implementations of their interfaces using the same backbone, geometry, and supervision, would help establish the specific benefit of WRAP’s persistent visual–geometric binding.
  2. The reliability analysis does not sufficiently characterize errors in the reconstructed scene representation. WRAP depends on depth estimation, segmentation, and instance merging, but the experiments mainly vary reconstruction frame count or substitute oracle grounding. Controlled tests involving depth noise, incomplete masks, occlusions, and incorrect identity merging would clarify when persistent grounded variables remain reliable and when errors propagate into program execution.
Questions for authors
  1. How do the experiments isolate the contribution of the proposed method from the effects of training data and model capacity?
  2. Do the editing metrics penalize unintended modifications to objects that were not supposed to be edited?
  3. How is the correctness of generated programs assessed beyond successful execution?
Reviewer_RkYvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WRAP connects video observations to programmable 3D scene representations for spatial reasoning and scene editing. It reconstructs scene geometry, segments and associates object instances across frames, and binds visual object IDs to their corresponding 3D entities. An MLLM then either answers directly or generates Python programs over these entities for external execution. The framework uses two-stage supervised training and achieves strong results on spatial reasoning and scene editing benchmarks, with particularly notable gains in distance and directional reasoning.

Critical strengths
  1. A clear and practical connection between visual grounding and geometric computation. WRAP binds visually marked objects to programmable 3D entities, allowing the MLLM to identify objects and generate executable geometric operations. This interface makes the reasoning process inspectable and supports precise computation beyond implicit visual estimation.

  2. Strong results on geometry-intensive tasks. WRAP performs well on ReVSI, particularly on distance and directional reasoning. The component ablations and oracle-substitution experiments provide useful evidence about the roles of object grounding and program generation.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of the proposed method. WRAP uses Qwen3.5, but the main results lack a clear comparison with the original Qwen3.5 models at the same sizes. The training conditions of the direct-answer baseline in Table 5 are also unclear. Consequently, it is difficult to distinguish improvements from the proposed interface, the backbone or the additional training. Although the paper mentioned close works like VADAR, pySpatial, and SpatialClaw, more detailed comparison with at least one closely related programmatic approach is missing, so it dosn't control these factors well. This makes the central advantage over existing methods insufficiently established.

  2. The generalization claim exceeds the evidence provided. Stage 2 includes training examples for both spatial reasoning and scene editing. Success on these two tasks therefore demonstrates applicability under task-specific supervision, rather than transfer to new tasks. Moreover, the editing experiments start from existing 3D scene assets instead of testing the complete video-to-3D editing pipeline. These results do not yet establish that the framework itself provides the broader generality claimed.

Questions for authors

Below are my questions for authors, I will raise score if they are properly addressed:

For Table 5, please clarify the training data and training stages used by each variant, especially the direct-answer baseline.

What evidence isolates the benefit of grounded-variable binding over existing programmatic methods such as pySpatial or SpatialClaw, using the same backbone and perception modules? Please also discuss some important related work that were not mentioned in paper, including Geometrically-Constrained Agent[1], SpaceTools[2], MSSR[3], and SpatialAgent[4].

Is 3D reconstruction and object extraction performed before the model decides whether to use 3D reasoning? What is the end-to-end latency relative to direct answering, including preprocessing, and how does this change when multiple questions share the same scene?

Why do Tables 1 and 2 use different generations of proprietary models? Please clarify, and whether comparisons with stronger models are available, such as GPT-5.4 or Gemini 3.1 Pro. Because models like Gemini-2.5 Pro and GPT-4o are quite old models compared to Qwen3.5, the backbone in the paper.

[1] Geometrically-Constrained Agent for Spatial Reasoning

[2] SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RL

[3] Pursuing Minimal Sufficiency in Spatial Reasoning

[4] SpatialScore: Towards Comprehensive Evaluation for Spatial Intelligence

Reviewer_eETKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes WRAP, a framework that takes standard multi-view video observations as input and combines several off-the-shelf components, including 3D reconstruction and instance segmentation, to construct object-level 3D representations. Visual objects are then bound to their corresponding 3D entities through symbolic Grounded Variables. Based on these variables, a fine-tuned MLLM learns when to invoke explicit 3D reasoning and generates executable spatial programs over the reconstructed scene. The framework is evaluated on four benchmarks covering video spatial reasoning and 3D scene editing, where it achieves strong or state-of-the-art performance.

Critical strengths
  1. This work studies an important and timely problem, i.e., enabling MLLMs to reason over explicit and programmable 3D world states rather than relying solely on implicit visual representations. I find the overall formulation meaningful for spatial reasoning and agentic multimodal systems. The paper is also well written and well organized.

  2. The proposed framework is efficient and easy to follow. This work decomposes the pipeline into visual-geometric entity binding, grounded 3D variables, adaptive activation, and executable program reasoning forms a coherent pipeline with a clear division of responsibilities between the MLLM and the external executor.

Critical weaknesses
  1. Lack of novelty. The framework achieves strong empirical performance, but much of its capability is inherited from off-the-shelf pretrained components, including 3D reconstruction, segmentation, visual grounding, and MLLM-based program generation. The proposed components mainly define interfaces and orchestration mechanisms between these existing capabilities. As a result, the work appears stronger as a system integration and data-engineering contribution than as a fundamentally new spatial reasoning methodology.

  2. Thoroughness of ablation analysis. The 3D Object Instantiation mainly combines off-the-shelf reconstruction and segmentation with standard geometric back-projection and instance association, and the 2D–3D Entity Binding is essentially a Set-of-Mark-style symbolic correspondence between visual instances and 3D point clouds. The adaptive 3D activation part is implemented as a learned binary routing decision triggered by a special token. While these components are practically useful as part of the complete system, their individual methodological novelty appears limited.

Questions for authors
  1. Stability of the framework. Since WRAP composes several pretrained components and relies on MLLM-generated grounding decisions, activation decisions, and executable programs, it would be useful to characterize the end-to-end stability of the system. If any components involve stochastic inference, the authors are encouraged to report results over multiple runs/seeds with mean and standard deviation. More generally, robustness to variations in segmentation, reconstruction, grounding, and program generation would help establish whether the proposed pipeline provides a reliably consistent 3D reasoning substrate rather than only strong single-run performance.

  2. The effect of the framework from additional supervision. This work is not only a new inference pipeline but also introduces substantial task-specific supervision, including Stage-1 grounding training and a newly constructed 27K Stage-2 spatial-program dataset. The appendix further shows that much more Stage-2 supervision improves performance. Hence, it is currently ambiguous to determine how much of the reported gain comes from the framework design itself versus the additional fine-tuning data. A controlled comparison using the same backbone, training data, and optimization budget—with and without Grounded Variables/executable 3D reasoning—would substantially strengthen the central claim of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.