acceptodds
Under review as a conference paper at ICLR 2027

SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

Abstract

Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc and simplistic physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to guarantee physical grounding. Across diverse scenes, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: it predicts real-robot policy outcomes more accurately than baseline methods and more reliably replays recorded manipulations without trajectory adaptation. Link to the anonymous [project page](https://scenerig.github.io/).

Primary area:
applications to robotics, autonomy, planning
Keywords:
real-to-sim, agentic system, scene reconstruction, simulation-ready

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xv61OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. The main contribution is the agentic loop of investigation-move-edit-end to refine scene setup.

Critical strengths
  1. It is a very important research question that the reconstructed scene needs to look faithful and physically stable.
  2. A very comprehensive experiment on robot applications.
  3. It is a smart choice to let the agent make decisions rather than output numerical values to reduce the chance of hallucinations. However, there is some concern about novelty; see weakness #3.
Critical weaknesses
  1. Did not demonstrate the importance of the agent. Agent only involved in "select object", "which type of adjustment is needed". An ablation that completely removes the agent is needed, since the alignment score can determine whether any adjustment is needed.
  2. Did not clarify why choosing those scene reconstruction methods. Missing citation of TabletopGen and PAT3D. It would be nice to see a comparison against those two methods:
  • TabletopGen, which has similar input, output, and setup;
  • PAT3D uses text input, but it uses an image as guidance in a pure optimization way.
  1. The design of move is very similar to SceneConductor; should discuss more and state the difference.
  2. The design of support-ordered physical settling is very similar to phi-Scene; should discuss more and state the difference.
  3. The video on the project page "Task: Put mustard bottle in the left bin" of SimFoundry looks like a coincidence: the left robot arm accidentally pushed the left bin, which fell over and caused the task to fail. I would like to see another video of this to see if it is a coincidence.
  4. I would like to see one sample of where SimFoundry/SceneRig succeeded but did not meet the agreement, and some analysis of why they failed.
Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Xv61 (1/3)

We thank the reviewer for recognizing the importance of the problem, our comprehensive robot experiments, and our choice to let the agent make decisions rather than output numerical values. We address your concerns as follows.


Q1. An ablation that completely removes the agent is needed, since the alignment score can determine whether any adjustment is needed.

Thank you for the suggestion. We ran this ablation on all 80 scenes with Claude Opus 5, replacing only the pose-refinement agent with a fixed rule. The earlier agentic stages are shared. The heuristic refines objects in support order, lowest alignment score first, applies Move to position, rotation, and scale, keeps a move only if the score rises, and uses up to three Move calls per object. In the table below, source-view, VLM, and cost columns cover all 80 scenes; all-view (35 orbit views for synthetic scenes), ARI, and layout columns cover the 44 simulator scenes. VLM scores are mean-centered Bradley–Terry scores from a separate comparison of these three variants, and time is wall-clock pose refinement per scene. We added this ablation to Appendix D.1 of the revised paper.

Pose refinementSource-view PSNR ↑Source-view SSIM ↑All-view PSNR ↑All-view SSIM ↑ARI ↑OSPA@10 (cm) ↓F1@10 ↑VLM Completeness ↑VLM Layout ↑VLM Appearance ↑Move calls / scene ↓Time / scene (min) ↓
Before refinement13.1110.53712.5070.5380.8303.320.925-0.013-0.413-0.332––
Agent-free heuristic13.3070.54312.5200.5380.8963.190.9260.0040.0030.07026.638.6
SceneRig (agent)13.2890.54312.5230.5380.8783.100.9330.0090.4100.26211.823.3

The two variants are close on image metrics, and the heuristic has higher ARI. The agent places objects more accurately in 3D (F1@5 0.804 vs. 0.778) and is preferred by the VLM judges for layout and appearance, while using fewer than half as many Move calls and spending less time on pose refinement per scene.

Our visual comparison illustrates why score-based acceptance is insufficient. The mustard bottle is upright in the input, but lies on its side in the shared starting reconstruction. The heuristic improves its alignment score from 0.109 to 0.223 yet leaves this pose error unresolved. The agent inspects the render against the input, recognizes that the bottle should stand upright, and corrects its pose through simulated edits. The essential contribution is this inspect–edit–simulate–inspect loop: visually assess the outcome and retain, revert, or revise the edit, rather than treating a score increase as sufficient evidence that the edited pose matches the object in the original input image.

AuthorsOpenReview ↗

Response to Reviewer Xv61 (2/3)

Q2. Did not clarify why choosing those scene reconstruction methods. Missing citation of TabletopGen and PAT3D. It would be nice to see a comparison against those two methods.

Thanks for pointing out these papers. Our baselines cover agentic reconstruction with a generator–verifier loop (VIGA), physically plausible real-to-sim reconstruction (REST3D), state-of-the-art image-to-scene reconstruction without physics (SceneConductor), and real-to-sim reconstruction evaluated on real-robot manipulation tasks (SimFoundry).

We evaluated TabletopGen and PAT3D on our 80-scene benchmark. In this reproduction, they completed 59 and 67 scenes, respectively. Specifically, for TabletopGen, 17 scenes failed the top-view quality check after ten attempts, three inputs were rejected by Seedream for image completion, and one scene lacked a detected main support object. For PAT3D, the 13 failures comprise six image-API rejections, one cyclic relation graph, two layout failures, one physics failure, two failures in our source-camera export, and one mesh-conversion timeout.

We show quantitative results in the following table (each baseline is compared with SceneRig on its completed subset) and qualitative results on our anonymous page. SceneRig outperforms both baselines on every metric except source-view SSIM, where PAT3D is marginally higher. We added the scores to Appendix D.2 and the five-sample, three-method visual comparison to Appendix D.3 of the revised paper.

MethodSource PSNR ↑Source SSIM ↑Source LPIPS ↓ARI ↑FG-IoU ↑Object PSNR* ↑Object SSIM* ↑Object LPIPS* ↓VLM Completeness ↑VLM Layout ↑VLM Appearance ↑OSPA@10 ↓F1@10 ↑
TabletopGen7.9270.5160.7520.5060.4189.4990.3580.301-1.273-1.744-2.1727.280.567
SceneRig (same 59 scenes)13.3880.5360.5170.8770.81813.6810.4410.1661.2731.7442.1723.190.926
PAT3D9.7950.5380.7430.1240.1259.5070.3780.315-1.572-2.794-2.7949.270.212
SceneRig (same 67 scenes)13.2720.5340.5170.8750.81513.5480.4280.1701.5722.7942.7943.010.948

Q3. The design of Move is very similar to SceneConductor; should discuss more and state the difference.

Thank you for the suggestion. Both methods use agents that compare renders with the input image and correct objects with local pose adjustments, but they differ in how a correction is validated. SceneConductor's planner and specialist agents apply their corrections directly in Blender, without physics simulation; in Table 1, only 8.8% of its scenes pass our stability test, versus 100% for SceneRig. In SceneRig, every edit is checked in simulation. The agent decides which object and pose component to correct, and Move searches bounded adjustments, simulates each candidate pose, and automatically rejects physically unstable ones. When Move cannot express a fix, such as standing a toppled object upright, the agent writes a direct code edit (Edit); the edited objects are settled in simulation, and the agent keeps or undoes the edit after inspecting the settled result. For example, as shown on this page, a laptop that rests on a stand in the photo was initialized under the stand and too small. Enlarging it with Move was rejected because the laptop tipped over in simulation, so the agent wrote a code edit that enlarged the laptop and placed it on the stand; the settled result matched the photo better, and the agent kept it.


Q4. The design of support-ordered physical settling is very similar to φ-Scene; should discuss more and state the difference.

We agree that both methods infer support relations with a VLM and settle objects in support order. The designs differ in three ways. (1) Image re-fitting during settling. φ-Scene removes penetration by optimizing only translation and then fixes each settled pose. After each object settles, SceneRig re-fits it to its masked 3D points (translation, yaw, and scale) and keeps the fit only if it reduces the error and the object stays stable. (2) Joint release. φ-Scene keeps settled objects as fixed passive bodies; SceneRig ends with a joint release in which all objects interact. (3) Settling as initialization. φ-Scene's settled scene is its output; in SceneRig, the agent then corrects poses with edits that are simulated and checked against the image. Finally, we validate our scenes on real robots through policy evaluation and trajectory replay, whereas φ-Scene evaluates on synthetic 3D-Front scenes without robot experiments.

AuthorsOpenReview ↗

Response to Reviewer Xv61 (3/3)

Q5. The video "Task: Put mustard bottle in the left bin" of SimFoundry looks like a coincidence: the left robot arm accidentally pushed the left bin, which fell over and caused the task to fail. I would like to see another video of this.

We thank the reviewer for the careful observation. We show two more Mustard-to-bin episodes, with the real, SceneRig, and SimFoundry rollouts side by side, under Question 1 on this page. In Episode 1 (π0.5), the real robot and SceneRig both place the mustard in the left bin, while in SimFoundry the left arm knocks the left bin over again and the mustard ends up in the right bin. In Episode 2 (LingBot-VLA 2.0), the real arms also push the left bin aside and the task fails; SceneRig reproduces this failure, while SimFoundry places the mustard in the left bin. In both episodes, SceneRig's outcome matches the real outcome and SimFoundry's does not.


Q6. I would like to see one sample of where SimFoundry/SceneRig succeeded but did not meet the agreement, and some analysis of why they failed.

We show two such episodes, with the real and both simulated rollouts side by side, under Question 2 on this page.

Episode 1 (LingBot-VLA 2.0, Everything-to-bin) (real: failure; SceneRig: failure; SimFoundry: success). The pear starts near the far edge of the robot's reach. In the real rollout, the gripper pushes it farther away, out of reach; SceneRig reproduces this, and both fail. SimFoundry reconstructs the pear smaller, which makes it easier to grasp, so its rollout succeeds. Here the disagreement stems from a reconstruction error that SceneRig avoids.

Episode 2 (π0.5, Fruit-to-plate) (real: failure; SceneRig and SimFoundry: success). The real robot places the banana and the pear but leaves the apple beside the plate. A possible reason is appearance: in the real image, the red apple and the red plate look similar, which may make it hard for the policy to notice that the apple is still outside the plate, whereas both simulated renderings show slightly higher contrast between them.

Reviewer_GrF1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an agentic system that transforms a single image into an editable simulation scene. It improves existing pipelines in terms of geometric, visual, and physical fidelity through an iterative edit-and-validate loop. The generated scenes are evaluated on downstream robotics tasks, with replay experiments demonstrating consistent results between simulation and the real world.

Critical strengths
  1. The paper addresses the practical challenge of creating physically realistic simulation scenes for training and evaluating downstream robotics applications. Generating scenes from a single image has substantial potential, particularly as AI-generated images could provide diverse environments for robot training.

  2. The paper improves scene quality across three key dimensions: visual, geometric, and physical fidelity. Its use of a symbolic scene graph provides an explicit representation of spatial relationships that could help enforce geometric consistency.

  3. The experiments are comprehensive, covering evaluations across all three dimensions, ablations of key modules, and checks of consistency between simulated and real-world robotic performance in both closed-loop and open-loop settings.

Critical weaknesses
  1. The paper lacks an evaluation of scene consistency across viewpoints and time. Would images of the same scene captured from different viewpoints produce consistent reconstructions? Similarly, would an image sequence collected during policy execution yield a temporally consistent scene?

  2. The evaluation is limited to small tabletop scenes. It remains unclear how well the proposed method would scale to larger environments, such as rooms, entire floors, houses, or cities. Evaluating the method at larger scales would help establish its broader applicability and strengthen the contribution.

  3. The methodological novelty relative to existing real-to-simulation approaches is not sufficiently clear. In particular, the paper could better explain how its design improves physical, visual, and geometric fidelity compared with prior work. A more explicit comparison of the methodological differences would help clarify its contributions.

Questions for authors
  1. What factors prevent the open-loop replay experiments from achieving a 100% success rate? Analyzing these limitations could provide useful insights into opportunities for further improvement.

  2. What failure modes arise during the iterative refinement process, and how does the edit-and-validate loop address them over successive iterations?

5 comments
AuthorsOpenReview ↗

Response to Reviewer GrF1 (1/2)

We thank the reviewer for recognizing the potential of single-image simulation scenes for robotics and the comprehensiveness of our experiments across visual, geometric, and physical fidelity. We address your concerns as follows.


Q1. Would images of the same scene captured from different viewpoints produce consistent reconstructions? Similarly, would an image sequence collected during policy execution yield a temporally consistent scene?

Different viewpoints. SceneRig targets single-image real-to-sim, as do CAST, MIDI, TableVerse, REST3D, and RoboSnap. Independently reconstructed views can resolve occlusion and depth ambiguity differently, so cross-view consistency is not guaranteed. Our modular geometric input already accepts the head camera's metric depth in the robotics experiments. A multi-view model such as VGGT (Wang et al., 2025) could supply camera poses and point maps in a shared frame, against which our verifiers and pose-refinement tools could compare renders. We now note this limitation in the revised paper, with multi-view inputs as future work.

Policy execution. We reconstruct each episode's scene once from the calibrated head camera before execution; simulated contacts and dynamics then move its objects. Our open-loop replay evaluates whether the same recorded commands reproduce the real execution's final object arrangement. Reconstructing a consistent scene from every rollout frame, despite object motion and arm occlusion, remains outside our current scope.


Q2. The evaluation is limited to small tabletop scenes. It remains unclear how well the proposed method would scale to larger environments, such as rooms, entire floors, houses, or cities. Evaluating the method at larger scales would help establish its broader applicability.

We agree that our evaluation focuses on tabletop scenes, which our pose-refinement tool (Move) mainly targets. This follows our goal: building simulation-ready tabletop scenes from a single image for real-to-sim policy evaluation and open-loop trajectory replay.

Following your suggestion, we tested SceneRig on five indoor and outdoor images beyond our benchmark, including two rooms (four from recent papers, one our own); see the interactive results. SceneRig recovers most furniture and objects, but complex rooms remain challenging. Our segmentation granularity is tuned for tabletop objects: in the bedroom, folded blankets are segmented separately from the bed and reconstructed as rigid meshes, which looks unnatural. Rooms also bring heavier occlusion, depth ambiguity, and dense contact, where correcting one object can disturb its neighbors. Entire floors, houses, or cities extend beyond a single image and would require multi-view input, which we leave to future work.


Q3. The methodological novelty relative to existing real-to-simulation approaches is not sufficiently clear. The paper could better explain how its design improves physical, visual, and geometric fidelity compared with prior work.

The key difference is physics in the loop. Prior methods align objects with the image only before physics, or skip physics; SceneRig simulates every edit and checks the settled result against the input image before keeping it. In our experiments, SceneRig achieves the best 3D layout and object appearance with every scene stable (Tables 1–2), and higher sim-to-real agreement and replay success than SimFoundry on real robots.

The table compares SceneRig with recent image-to-scene methods, all of which build the scene from one RGB image (PAT3D starts from text). Columns: physically plausible, penetration-free and settled under gravity; agent-driven correction, an LLM/VLM agent chooses corrections; image-based pose refinement, the image guides pose estimation, typically once before physics; image re-check after edits, the image is checked after every simulated edit; real-robot validation, simulated outcomes are compared with real robots.

MethodPhysically plausible reconstructionAgent-driven correctionImage-based pose refinementImage re-check after editsReal-robot validation
TabletopGen (Wang et al., 2025)✓✗✓✗✗
PAT3D (Lin et al., 2026)✓✗✗✗✗
REST3D (Ma et al., 2026)✓✗✗✗✗
SimuScene (Lee et al., 2026)✓✗✓✗✗
SceneConductor (Kim et al., 2026)✗✓✓✗✗
φ-Scene (Li et al., 2026)✓✗✗✗✗
SimFoundry (Ranawaka et al., 2026)✓✗✓✗✓
RoboSnap (Zhang et al., 2026)✓✗✓✗✓
SceneRig (ours)✓✓✓✓✓
AuthorsOpenReview ↗

Response to Reviewer GrF1 (2/2)

Q4. What factors prevent the open-loop replay experiments from achieving a 100% success rate?

Open-loop replay executes the recorded commands without feedback from object states, so a small geometric error in the reconstruction can change the outcome. One source of such errors is the image-to-3D model. We show an example on our anonymous page. In this Marker-to-cup episode, the generated marker's height-to-width ratio differs slightly from that of the real marker. SceneRig rescales each object uniformly, which matches its overall size but cannot correct its proportions, so the gripper contacts the marker at a slightly different position than in the real episode. Because the marker is small and placing it into the cup requires high accuracy, this small difference leaves the marker beside the cup rather than inside it. Although such failures occur, SceneRig still replays 20 of 25 demonstrations successfully, compared with 9 for SimFoundry, and it succeeds on every demonstration that SimFoundry replays successfully.


Q5. What failure modes arise during the iterative refinement process, and how does the edit-and-validate loop address them over successive iterations?

We observe two failure modes: a correction that improves image alignment but is physically infeasible, and a fix that Move cannot express. Move handles most corrections (94% of tool calls for Opus 5), and every candidate is simulated; unstable ones are rejected and reverted, so failed attempts do not accumulate across iterations. When Move cannot express a fix, the agent writes a code edit (Edit), which is also simulated and kept only after the agent compares the settled result with the photo. We show a success and a limitation on our anonymous page. In the success, physics rejects enlarging a laptop with Move because it tips over, and a code edit places it on its stand. In the limitation, the agent stands one of two lying bottles upright but not the other: writing a correct 3D rotation by hand is hard for current VLMs. The scene stays physically valid, but that bottle is misplaced.

Dear Reviewer GrF1,

Thank you again for your thoughtful review! We’re glad you found value in our improved scene reconstruction results and our evaluation of both closed-loop and open-loop robot performance. As the discussion period draws to a close, could you let us know whether our response has addressed your concerns? We would be happy to clarify any remaining points.

Best regards, Authors

Reviewer_yuWZOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SceneRig, an agentic framework for reconstructing simulation-ready 3D scenes from a single image. It combines scene graph parsing, object reconstruction, metric initialization, and physics-grounded pose refinement. A key component is the refinement agent, which identifies object placement errors and uses geometric search and physics simulation to validate corrections.

The method is evaluated on reconstruction quality, physical stability, real-to-sim policy evaluation, and open-loop trajectory replay. It outperforms the evaluated baselines in object placement and stability, and achieves 80% trajectory replay success compared with 36% for SimFoundry.

Critical strengths

1. The agentic pipeline is well structured.

I like how the authors separate high-level decisions from low-level optimization. The agent identifies which object needs correction and what type of edit to make, while the Move tool searches for the actual adjustment. The use of physics simulation to validate edits also avoids some of the problems with directly asking VLMs to modify 3D coordinates.

2. The paper includes real robot experiments.

Besides reconstruction metrics, the authors evaluate whether the reconstructed scenes can reproduce robot manipulation outcomes. In particular, open-loop trajectory replay improves from 36% to 80% over SimFoundry. This is a useful test because a scene can look correct in the rendered image but still fail when the robot interacts with it.

Critical weaknesses

1. The contribution of agentic refinement over existing physics-based optimization is not sufficiently isolated.

The paper demonstrates the benefit of the bounded Move tool over direct code editing (Tables 3 and 11). However, it is less clear how much the agent itself contributes compared with a deterministic optimization pipeline.

Recent methods such as φ-Scene (Li et al., 2026), REST3D (Ma et al., 2026), and SimuScene (Lee et al., 2026) also use geometric constraints and physics-based refinement to correct object placement or geometry. Although REST3D is included in the comparisons, the current experiments do not clearly distinguish the benefit of agent-guided correction from the benefit of the underlying optimization and simulation tools.

I would like to see an ablation where the same Move tool is used with a simple rule-based object selection strategy, such as iteratively selecting the object with the lowest alignment score. This would help clarify whether agentic reasoning is necessary for selecting effective corrections.

2. Physical stability does not necessarily imply accurate physical behavior.

The method achieves 100% stability on the reconstruction benchmark, but some of the recovery procedures in Appendix B.2 modify physical properties, including the center of mass, friction, and damping, to obtain stable configurations.

While these modifications may be useful for preventing objects from toppling, they could also change how objects respond during manipulation. This distinction is important for the paper's claim of simulation readiness.

I suggest evaluating whether the reconstructed physical properties reproduce actual object behavior during manipulation, rather than relying primarily on static stability and final task success. Recent real-to-sim work such as RoboSnap (Zhang et al., 2026) provides a relevant reference for trajectory replay and real-to-sim policy evaluation. It would also help to report how frequently the physical-property recovery procedures are triggered and whether disabling them affects downstream performance.

Questions for authors
  1. How much does agent reasoning contribute to pose refinement? Have the authors compared the proposed agent with a heuristic that selects the object with the largest alignment error and applies the same bounded Move tool? This seems important for understanding which part of the pipeline is responsible for the improvement.

  2. How reliable are the intermediate verifiers? The geometry stage uses deterministic rules, while material and lighting refinement depend mainly on VLM verification. Could the authors report the verifier approval rate, average number of revisions per stage, and how often the stage reaches its iteration budget without approval? Such statistics would help characterize the effectiveness of the agentic loops.

  3. How often are physical properties modified to achieve stability? Appendix B.2 describes adjustments to collision geometry, center of mass, friction, and damping. Could the authors report their frequency and evaluate their impact on downstream manipulation or trajectory replay?

  4. How often does the agent use direct code edits rather than bounded Move operations? Since direct edits rely on agent judgment rather than automatic acceptance criteria, it would be useful to understand their frequency and impact on final reconstruction quality.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yuWZ (1/3)

We thank the reviewer for finding our agentic pipeline well structured and our real-robot experiments a useful test beyond visual appearance. We address your concerns as follows.


Q1. How much does agent reasoning contribute to pose refinement? Compare the proposed agent with a heuristic that selects the object with the largest alignment error and applies the same bounded Move tool.

Thank you for the suggestion. We ran this ablation on all 80 scenes with Claude Opus 5, replacing only the pose-refinement agent with a fixed rule. The earlier agentic stages are shared. The heuristic refines objects in support order, lowest alignment score first, applies Move to position, rotation, and scale, keeps a move only if the score rises, and uses up to three Move calls per object. In the table below, source-view, VLM, and cost columns cover all 80 scenes; all-view (35 orbit views for synthetic scenes), ARI, and layout columns cover the 44 simulator scenes. VLM scores are mean-centered Bradley–Terry scores from a separate comparison of these three variants, and time is wall-clock pose refinement per scene. We added this ablation to Appendix D.1 of the revised paper.

Pose refinementSource-view PSNR ↑Source-view SSIM ↑All-view PSNR ↑All-view SSIM ↑ARI ↑OSPA@10 (cm) ↓F1@10 ↑VLM Completeness ↑VLM Layout ↑VLM Appearance ↑Move calls / scene ↓Time / scene (min) ↓
Before refinement13.1110.53712.5070.5380.8303.320.925-0.013-0.413-0.332––
Agent-free heuristic13.3070.54312.5200.5380.8963.190.9260.0040.0030.07026.638.6
SceneRig (agent)13.2890.54312.5230.5380.8783.100.9330.0090.4100.26211.823.3

The two variants are close on image metrics, and the heuristic has higher ARI. The agent places objects more accurately in 3D (F1@5 0.804 vs. 0.778) and is preferred by the VLM judges for layout and appearance, while using fewer than half as many Move calls and spending less time on pose refinement per scene.

Our visual comparison illustrates why score-based acceptance is insufficient. The mustard bottle is upright in the input, but lies on its side in the shared starting reconstruction. The heuristic improves its alignment score from 0.109 to 0.223 yet leaves this pose error unresolved. The agent inspects the render against the input, recognizes that the bottle should stand upright, and corrects its pose through simulated edits. The essential contribution is this inspect–edit–simulate–inspect loop: visually assess the outcome and retain, revert, or revise the edit, rather than treating a score increase as sufficient evidence that the edited pose matches the object in the original input image.

AuthorsOpenReview ↗

Response to Reviewer yuWZ (2/3)

Q2. How often are physical properties modified to achieve stability? Appendix B.2 describes adjustments to collision geometry, center of mass, friction, and damping. Could the authors report their frequency and evaluate their impact on downstream manipulation or trajectory replay?

Frequency. Physical-property recovery is a rare fallback. In our main benchmark, it was retained for 7/709 objects (0.99%) in 7/80 scenes (8.75%): two plush toys, a vase with flowers, sunglasses, a slice of toast, a figurine, and a brush. For these objects, recovery jointly adjusts the center of mass and inertia, flattens the collider base, and sets friction and angular damping.

This module is designed for handling the imperfect geometry generated by the off-the-shelf image-to-mesh model. Soft or composite objects, such as plush toys or a vase with flowers, become single rigid meshes whose base and mass distribution may not match the real object, so an object that stands upright in the image can topple in simulation. Better geometry could reduce the need for it.

Impact on robotics application. The fallback is not used in our robotics evaluation. No physical-property override was applied in any of the 125 evaluated scenes: 0/25 for open-loop trajectory replay and 0/100 for real-to-sim policy evaluation. The reported robotics results therefore do not rely on this fallback. We have added these statistics to Appendix B.2.


Q3. I suggest evaluating whether the reconstructed physical properties reproduce actual object behavior during manipulation, rather than relying primarily on static stability and final task success.

As noted in the previous answer, no physical-property override is applied in any robotics scene. In addition, we agree that static stability is necessary but not sufficient, so we also test behavior under contact through open-loop trajectory replay. In replay, the recorded joint and gripper commands run without feedback from object states, so each grasp, lift, and release succeeds only if the simulated objects respond to contact as the real objects did. At the object level, 39 of 45 evaluated objects across the 25 replays reach their targets. Synchronized real and simulated replays are available on our anonymous page.

AuthorsOpenReview ↗

Response to Reviewer yuWZ (3/3)

Q4. Could the authors report the verifier approval rate, average number of revisions per stage, and how often the stage reaches its iteration budget without approval?

We checked the geometry, material, and lighting stages on our 80-scene benchmark for each of the five VLM backbones (table below). Each stage allows at most two attempts, so it is revised at most once. Under our default setting with Claude Opus 5 as the underlying VLM:

  • Approval rate: 98.75% (geometry), 93.75% (root material), and 100% (lighting).
  • Average revisions per stage: 0.15, 0.36, and 0.08.
  • Budget reached without approval: 1, 5, and 0 of 80 scenes.

The table below gives the share (count) of the 80 scenes approved on the first attempt, approved on the second attempt, or not approved after two attempts, and the mean revision count. These statistics are added to Appendix B.3.

StageVLM backboneApproved on first attemptApproved on second attemptNot approved after two attemptsMean revision count
GeometryClaude Opus 5 (default model)85.00% (68)13.75% (11)1.25% (1)0.1500
Claude Fable 573.75% (59)21.25% (17)5.00% (4)0.2625
GPT-5.6-terra5.00% (4)7.50% (6)87.50% (70)0.9500
Kimi K348.75% (39)22.50% (18)28.75% (23)0.5125
GPT-6 Astra97.50% (78)2.50% (2)0.00% (0)0.0250
MaterialClaude Opus 5 (default model)63.75% (51)30.00% (24)6.25% (5)0.3625
Claude Fable 551.25% (41)38.75% (31)10.00% (8)0.4875
GPT-5.6-terra2.50% (2)12.50% (10)85.00% (68)0.9750
Kimi K337.50% (30)32.50% (26)30.00% (24)0.6250
GPT-6 Astra71.25% (57)20.00% (16)8.75% (7)0.2875
LightingClaude Opus 5 (default model)92.50% (74)7.50% (6)0.00% (0)0.0750
Claude Fable 595.00% (76)5.00% (4)0.00% (0)0.0500
GPT-5.6-terra47.50% (38)32.50% (26)20.00% (16)0.5250
Kimi K376.25% (61)22.50% (18)1.25% (1)0.2375
GPT-6 Astra97.50% (78)2.50% (2)0.00% (0)0.0250

Q5. Verifier feedback improves the scene. For each rejected first attempt, we compare the render the verifier reviewed before and after the revision on the Claude Opus 5 experiment. As shown in the table, the revisions raise PSNR and SSIM in all three stages and lower LPIPS for geometry and lighting, indicating that the revisions bring the scene closer to the input. In the table, each entry reads first attempt → second attempt for the scenes whose stage was rejected on the first attempt.

Stage (revised scenes)PSNR ↑SSIM ↑LPIPS ↓nCLIP ↓
Geometry (12)13.63 → 14.140.587 → 0.5910.587 → 0.5510.136 → 0.135
Root material (29)13.70 → 14.100.471 → 0.4840.520 → 0.5270.114 → 0.115
Lighting (6)13.10 → 14.710.556 → 0.5670.515 → 0.4780.076 → 0.066

Q6. How often does the agent use direct code edits rather than bounded Move operations? Since direct edits rely on agent judgment rather than automatic acceptance criteria, it would be useful to understand their frequency and impact on final reconstruction quality.

We counted pose-refinement tool calls on all benchmark scenes for each of the five VLM backbones (table below). For our default Claude Opus 5, the agent made 945 Move and 64 Edit calls (11.81 and 0.80 per scene on average): Edit accounts for 6.3% of calls and is used in 24/80 scenes. Across backbones, Edit accounts for 6.3–15.7% of calls, so Move remains the primary tool. Counts include failed and later undone calls.

Regarding impact, each Edit is simulated and rendered before the agent keeps or undoes it. The code-edits-only ablation (Tables 3 and 12), which removes Move, shows that relying on Edit alone lowers reconstruction quality and replay success. Therefore, in our system, Edit is an occasional fallback to Move, not a substitute.

VLM backboneMove callsEdit callsEdit share of callsScenes using EditCalls per scene (Move / Edit)
Claude Opus 5 (default model)945646.34%24/80 (30.00%)11.81 / 0.80
Claude Fable 5911828.26%26/80 (32.50%)11.39 / 1.02
GPT-5.6-terra720739.21%24/80 (30.00%)9.00 / 0.91
Kimi K378611512.76%31/80 (38.75%)9.82 / 1.44
GPT-6 Astra88316515.74%55/80 (68.75%)11.04 / 2.06

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.