acceptodds
Withdrawn from ICLR 2027

GRASE: Geometry-native Real-to-sim Agentic Scene Modeling from Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-to-sim conversion aims to turn a captured video of a real environment into an editable 3D scene while preserving the geometry and layout of what was observed. Existing methods often follow a one-pass, loosely coupled paradigm: each stage only take the limited feedbacks from previous stages, allowing errors to propagate without reliable correction and leaving the final scene inconsistent. We present GRASE, an agentic framework that closes this loop by treating the captured multi-view video observations as a standing reference throughout scene modeling. Instead of committing to intermediate estimation, GRASE revises its current hypothesis whenever rendered evidence disagrees with the capture, overcoming the fixed intermediate assumptions of conventional pipelines. Specifically, GRASE first fits the architectural shell and its openings from depth and multi-view cues. An agent then places generated assets in a shared coordinate frame while checking their spatial relationships accordingly. Finally each asset’s geometry and placement are refined iteratively to improve agreement across the observed views. By coupling these steps through feedback, GRASE localizes structural discrepancies and revises only the responsible components, while preserving cross-view consistency. Under this pipeline, the resulting 3D scenes are faithful to the input,consistent across views and suitable for simulations. In our experiments, GRASE substantially outperforms other state-of-the-art methods, including a direct realto-sim conversion using GPT6-Astra.

Primary area:
generative models
Keywords:
scene genaration, agentic harness, real-to-sim

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_f4FzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRASE, an agentic real-to-sim framework that converts monocular indoor videos into editable 3D scenes. The method combines explicit geometric reconstruction with a VLM-based verification and repair loop. It first reconstructs room surfaces and openings from estimated depth and multi-view geometry, then generates and registers object assets, and finally performs object-level and part-wise refinement by repeatedly comparing rendered scenes against the original video observations.

The key idea is to maintain the input video as a persistent reference instead of committing to intermediate reconstruction results. The system uses a collection of existing components, including Depth Anything 3, SAM 3, GPT-6-based VLM reasoning, TRELLIS.2, Blender operations, ICP, and silhouette-based optimization. Experiments on Replica and Imaginarium show substantial improvements over SAM3D and GPT-6-based baselines in geometry and rendering metrics.

Critical strengths
  1. The paper addresses a practically important real-to-sim problem and correctly emphasizes that visually plausible reconstruction is insufficient for simulation. Explicit treatment of room topology, object support relationships, and geometric consistency is meaningful for building editable 3D environments.

  2. The persistent render-and-verify formulation is intuitive and the reported quantitative improvements are substantial. In particular, explicitly grounding reconstruction in geometric evidence appears to significantly improve both architectural shell reconstruction and object/scene alignment.

Critical weaknesses
  1. The main contribution is difficult to separate from a large amount of system engineering. GRASE combines many existing components and classical geometric operations, including DA3, SAM 3, GPT-6, TRELLIS.2, Blender MCP, RANSAC, ICP, silhouette optimization, and hand-designed geometric refinement. While their integration is effective, it is unclear what the central methodological contribution is beyond the orchestration and verification pipeline. The current ablation mainly isolates the geometric critic and does not sufficiently disentangle the contributions of the many other components.

  2. The degree of automation, efficiency, and practical cost are insufficiently documented. The paper describes both reconstruction stages as being executed by a tool-using VLM agent, but it is unclear whether any manual correction or scene-specific intervention is required. The paper also does not report end-to-end reconstruction time, number of VLM/tool calls, API cost, or detailed runtime breakdown. These omissions make it difficult to assess scalability, reproducibility, and practical usefulness of such a complex pipeline.

Questions for authors
  1. Is the complete pipeline fully automatic after providing the input video? If any human intervention, manual correction, or scene-specific tuning is required, please clarify where it occurs and approximately how much human time is needed per scene.

  2. What is the end-to-end reconstruction time per scene? Please also clarify the approximate number of VLM calls, TRELLIS.2 generations, agent revision rounds, and, if applicable, API or computational cost.

  3. Given the complexity of the pipeline, can the authors clarify which components are essential to the main contribution? In particular, how much performance is attributable to persistent agentic verification itself versus explicit shell fitting, support-graph reasoning, part-wise refinement, and silhouette optimization?

  4. The paper repeatedly motivates the reconstructed scenes as suitable for simulation, but articulation, materials, and physical properties are not recovered. Could the authors clarify the intended meaning of “simulation-ready” and whether the reconstructed scenes have been evaluated in any downstream simulation or embodied task?

Reviewer_VQkuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes an agentic vision system for 3D scene reconstruction. This involves a set of computer vision tools being used at inference, such as multi-view 3D point map and camera poses estimation (depth anything v3) along with an object segmentation model. Then the scene is reconstructed in three ways: either via a 3D generative model invocation (Trellis2), authoring blender primitives for simple objects and parametric primitive fit for the walls/celling and other scene elements. The method is then evaluated against SAM3D and a vanilla GPT-6 VLM agent on two synthetic datasets with simulated indoor environments

Critical strengths

I get the overall motivation for primitive-based surface fitting for walls and other non-object surfaces. And I think this direction is probably the most interesting thing in the paper --- how can they be coupled with geometric observations and made "fittable" to the observed points. In its current state, however, it's muddled with unclear implementation details and motivation. If it's an agentic approach, why do we use only a certain class of surface primitives: quadratic surfaces and highmaps (if my understanding is correct). Why can't we have a wider, more expressive set? It feels like we are getting the worst of both worlds --- it probably takes a long time to run and is not that much better than a more traditional plane-fitting based baseline.

Critical weaknesses
  • Why do we need a scene graph (edges specifically), say “on” relationship in Figure 2? Is it ever used?

  • With this parametric, manifold style approach that authors used for the walls — can we reconstruct objects with the same approach and mitigate using other generative priors?

  • I read multiple times the parts about non-planar surface representation and I still struggle to understand what’s exactly going on. A lot of details is given but they are not helping to understand the whole picture. I’m not sure what loft and local quadratic displacement fields are exactly, what are the parameters being optimised there? What is b in equation 3? This is some sort of 2D local quadric approximating the surface or/and a 2D highmap.. which is also parametric? Why do we need both? Why do we only employ this representation for free-form surface editing? In the supplementary the details are not given — the description is also vague and just repeats the main paper.

  • What’s the representation used for finding the intersection between the surface patches? Is this intersection boundary optimisable (aka the region of influence of these 2D primitives). Or is it fixed by the segmentation? Judging from the figure of Figure 3 they seem to be able to change their regions of influence. In figure 3, however, the final result looks completely unrelated tothe

  • Why does SAM3D have such low results on benchmarking for object quality, even though the visuals looks good (table 1)? It does predict pose, which is often noisy, but it definitely shouldn't have garbage scores. What’s the protocol of the anchor view selection for SAM3D? I’m a distrustful of these numerical results / eval correctness.

  • What’s the iteration protocol? How is this pipleline structured for the agent? i understand the steps, but what’s the actual implementation? This paper would greatly benefit of What do authors specify and what’s left for an agent? is this a description of what happens in practice or what authors designed as a set of tools and protocols? This paper specifically would benefit a lot from attaching the whole source code directly, otherwise I don't understand what is it that authors actually did themselves.

  • What is the “hard-crease critic”? It is only named in the ablation but never before. Is it a VLM what decides if an object can be generated natively? This a recurring problem in this paper that there's a lot of new concepts are introduced somewhere randomly and are never properly explained.

Questions for authors
  • Why authoring blender primitives for celling / walls is not possible? Why do we have to go and use a third representation for them?

  • Why are results only presented for dataset scenes? I assume if this method works as well as promised it should work in the wild. The only real scene is attached in the supplementary and it also seems to be from a dataset with high-quality GT mesh --- in fact the video is not real but a trajectory inside of a 3D reconstructed mesh.

  • Is the agent sandboxed during execution from the GT geometry for these dataset scenes? So far we haven't seen the performance of this system on anything which is not supplied with almost perfect GT geometry.

Reviewer_6t5XOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents GRASE, an agentic real-to-sim 3D scene modeling method. GRASE uses Depth Anything 3 and SAM3 for initial depth and segmentation processing. It then combines agentic primitive proposals and TRELLIS.2 generation with geometric optimization and a feedback loop, with special rules implemented for walls/floor/ceiling. Experiments show GRASE outperforms naive application of GPT-6 on select indoor scenes.

Critical strengths
  • The specific part-based refinement scheme seems to have not been done before in such agentic pipelines.
  • The pipeline takes advantage of many existing tools and demonstrates significant engineering effort.
Critical weaknesses
  • The evaluation is extremely lacking; it only spans 23 scenes and the baselines only consist of presumably naive GPT-6 calls. No existing real-to-sim methods (e.g, [1, 2]) are included in the comparison. There is only a single ablation, covering one of the geometry correction steps.
  • The method's novelty seemingly only lies in the integration of specific geometric optimization and prompt engineering tricks. It is likely that they are over-engineered for the evaluation scenes and do not provide general insights.

[1] Yin et al., "VIGA: Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning", ECCV 2026

[2] Ranawaka et al., "SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation", arXiv 2026

Questions for authors
  • How does GRASE compare with works such as VIGA or SimFoundry?
  • Considering the fact that GRASE has so many parts, which ones are most important and what are their effects?
Reviewer_X6MmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper presents a pipeline for reconstructing an editable indoor scene from monocular video.
  • The output consists of a room shell with explicit openings, individual object meshes and a support graph.
  • The pipeline uses Depth Anything 3 to estimate depth and camera poses, a GPT-6 VLM to identify scene elements and their relationships, and SAM 3 masks to extract point clouds for individual surfaces and objects.
  • The room shell is built from fitted planes with shared corners defined by triple-plane intersections.
  • Large objects are reconstructed using either Blender-MCP primitives or TRELLIS.2; small objects use TRELLIS.2 throughout. Placement combines ICP, VLM-guided adjustments, and silhouette-based coordinate search. Part-wise stretch edits and support constraints further refine shape and contact. A VLM agent checks intermediate results and can roll back unsuccessful steps.
  • The method is evaluated on 8 Replica and 15 Imaginarium scenes against SAM3D and two GPT-6 agent baselines, with substantial improvements across the reported shell, object, scene, and input-view metrics.
Critical strengths

The strongest aspect of the paper, in my view, is the division of responsibilities between the VLM and the geometric solvers. The VLM handles discrete decisions—topology, adjacency, surface type, which component needs correction, and which frames to inspect—while bounded, robust solvers estimate the numerical parameters. This is a sensible design, and I can see it being useful beyond this particular pipeline. The shared-corner shell formulation in Equation 1 is especially appealing because it gives a watertight representation while preserving editability.

Critical weaknesses
  • I am not convinced that the current experiments establish that the reconstructed scenes are “simulation-ready.” The paper reports no interpenetration or floating-object statistics, physics-settling tests, or downstream robot task. The support constraints are useful, but their presence alone does not demonstrate physical validity.
  • The claim of faithfulness to the input also needs qualification. Occluded objects are completed through image editing before being passed to TRELLIS.2, so some geometry is generated without direct observational support. The limitations further acknowledge that materials, articulation, and physical properties are not recovered. These limitations matter when interpreting the real-to-sim claim.
  • The stochastic agent appears to be run only once per scene. Without repeated runs or reporting of runtime, token usage, API cost, and failure rate, it is difficult to assess reliability and practical overhead.
  • The baseline selection is a concern. Two of the three baselines are the authors’ own GPT-6 prompting setups, while SAM3D is a single-image object reconstruction method. These comparisons are informative, but they leave open how GRASE compares with structured reconstruction methods such as SceneScript (ECCV’24), SpatialLM (NeurIPS’25), or Scan2CAD (CVPR’19), and with real-to-sim pipelines such as ACDC (CoRL’24) or DRAWER (CVPR’25).
Questions for authors

Can you compare with SceneScript or SpatialLM for the shell, and ACDC or DRAWER for the full scene? Can you report mean and standard deviation over multiple agent runs, together with wall-clock time, VLM calls and tokens, API cost, and failure rate per scene? Which image-editing model is used for object completion, and how much of the final geometry depends on these completions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.