- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. The main contribution is the agentic loop of investigation-move-edit-end to refine scene setup.
Critical strengths
- It is a very important research question that the reconstructed scene needs to look faithful and physically stable.
- A very comprehensive experiment on robot applications.
- It is a smart choice to let the agent make decisions rather than output numerical values to reduce the chance of hallucinations. However, there is some concern about novelty; see weakness #3.
Critical weaknesses
- Did not demonstrate the importance of the agent. Agent only involved in "select object", "which type of adjustment is needed". An ablation that completely removes the agent is needed, since the alignment score can determine whether any adjustment is needed.
- Did not clarify why choosing those scene reconstruction methods. Missing citation of TabletopGen and PAT3D. It would be nice to see a comparison against those two methods:
- TabletopGen, which has similar input, output, and setup;
- PAT3D uses text input, but it uses an image as guidance in a pure optimization way.
- The design of move is very similar to SceneConductor; should discuss more and state the difference.
- The design of support-ordered physical settling is very similar to phi-Scene; should discuss more and state the difference.
- The video on the project page "Task: Put mustard bottle in the left bin" of SimFoundry looks like a coincidence: the left robot arm accidentally pushed the left bin, which fell over and caused the task to fail. I would like to see another video of this to see if it is a coincidence.
- I would like to see one sample of where SimFoundry/SceneRig succeeded but did not meet the agreement, and some analysis of why they failed.
Questions for authors
See weakness.
6 comments
Response to Reviewer Xv61 (2/3)
Q2. Did not clarify why choosing those scene reconstruction methods. Missing citation of TabletopGen and PAT3D. It would be nice to see a comparison against those two methods.
Thanks for pointing out these papers. Our baselines cover agentic reconstruction with a generator–verifier loop (VIGA), physically plausible real-to-sim reconstruction (REST3D), state-of-the-art image-to-scene reconstruction without physics (SceneConductor), and real-to-sim reconstruction evaluated on real-robot manipulation tasks (SimFoundry).
We evaluated TabletopGen and PAT3D on our 80-scene benchmark. In this reproduction, they completed 59 and 67 scenes, respectively. Specifically, for TabletopGen, 17 scenes failed the top-view quality check after ten attempts, three inputs were rejected by Seedream for image completion, and one scene lacked a detected main support object. For PAT3D, the 13 failures comprise six image-API rejections, one cyclic relation graph, two layout failures, one physics failure, two failures in our source-camera export, and one mesh-conversion timeout.
We show quantitative results in the following table (each baseline is compared with SceneRig on its completed subset) and qualitative results on our anonymous page. SceneRig outperforms both baselines on every metric except source-view SSIM, where PAT3D is marginally higher. We added the scores to Appendix D.2 and the five-sample, three-method visual comparison to Appendix D.3 of the revised paper.
| Method | Source PSNR ↑ | Source SSIM ↑ | Source LPIPS ↓ | ARI ↑ | FG-IoU ↑ | Object PSNR* ↑ | Object SSIM* ↑ | Object LPIPS* ↓ | VLM Completeness ↑ | VLM Layout ↑ | VLM Appearance ↑ | OSPA@10 ↓ | F1@10 ↑ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| TabletopGen | 7.927 | 0.516 | 0.752 | 0.506 | 0.418 | 9.499 | 0.358 | 0.301 | -1.273 | -1.744 | -2.172 | 7.28 | 0.567 |
| SceneRig (same 59 scenes) | 13.388 | 0.536 | 0.517 | 0.877 | 0.818 | 13.681 | 0.441 | 0.166 | 1.273 | 1.744 | 2.172 | 3.19 | 0.926 |
| PAT3D | 9.795 | 0.538 | 0.743 | 0.124 | 0.125 | 9.507 | 0.378 | 0.315 | -1.572 | -2.794 | -2.794 | 9.27 | 0.212 |
| SceneRig (same 67 scenes) | 13.272 | 0.534 | 0.517 | 0.875 | 0.815 | 13.548 | 0.428 | 0.170 | 1.572 | 2.794 | 2.794 | 3.01 | 0.948 |
Q3. The design of Move is very similar to SceneConductor; should discuss more and state the difference.
Thank you for the suggestion. Both methods use agents that compare renders with the input image and correct objects with local pose adjustments, but they differ in how a correction is validated. SceneConductor's planner and specialist agents apply their corrections directly in Blender, without physics simulation; in Table 1, only 8.8% of its scenes pass our stability test, versus 100% for SceneRig. In SceneRig, every edit is checked in simulation. The agent decides which object and pose component to correct, and Move searches bounded adjustments, simulates each candidate pose, and automatically rejects physically unstable ones. When Move cannot express a fix, such as standing a toppled object upright, the agent writes a direct code edit (Edit); the edited objects are settled in simulation, and the agent keeps or undoes the edit after inspecting the settled result. For example, as shown on this page, a laptop that rests on a stand in the photo was initialized under the stand and too small. Enlarging it with Move was rejected because the laptop tipped over in simulation, so the agent wrote a code edit that enlarged the laptop and placed it on the stand; the settled result matched the photo better, and the agent kept it.
Q4. The design of support-ordered physical settling is very similar to φ-Scene; should discuss more and state the difference.
We agree that both methods infer support relations with a VLM and settle objects in support order. The designs differ in three ways. (1) Image re-fitting during settling. φ-Scene removes penetration by optimizing only translation and then fixes each settled pose. After each object settles, SceneRig re-fits it to its masked 3D points (translation, yaw, and scale) and keeps the fit only if it reduces the error and the object stays stable. (2) Joint release. φ-Scene keeps settled objects as fixed passive bodies; SceneRig ends with a joint release in which all objects interact. (3) Settling as initialization. φ-Scene's settled scene is its output; in SceneRig, the agent then corrects poses with edits that are simulated and checked against the image. Finally, we validate our scenes on real robots through policy evaluation and trajectory replay, whereas φ-Scene evaluates on synthetic 3D-Front scenes without robot experiments.
Response to Reviewer Xv61 (3/3)
Q5. The video "Task: Put mustard bottle in the left bin" of SimFoundry looks like a coincidence: the left robot arm accidentally pushed the left bin, which fell over and caused the task to fail. I would like to see another video of this.
We thank the reviewer for the careful observation. We show two more Mustard-to-bin episodes, with the real, SceneRig, and SimFoundry rollouts side by side, under Question 1 on this page. In Episode 1 (π0.5), the real robot and SceneRig both place the mustard in the left bin, while in SimFoundry the left arm knocks the left bin over again and the mustard ends up in the right bin. In Episode 2 (LingBot-VLA 2.0), the real arms also push the left bin aside and the task fails; SceneRig reproduces this failure, while SimFoundry places the mustard in the left bin. In both episodes, SceneRig's outcome matches the real outcome and SimFoundry's does not.
Q6. I would like to see one sample of where SimFoundry/SceneRig succeeded but did not meet the agreement, and some analysis of why they failed.
We show two such episodes, with the real and both simulated rollouts side by side, under Question 2 on this page.
Episode 1 (LingBot-VLA 2.0, Everything-to-bin) (real: failure; SceneRig: failure; SimFoundry: success). The pear starts near the far edge of the robot's reach. In the real rollout, the gripper pushes it farther away, out of reach; SceneRig reproduces this, and both fail. SimFoundry reconstructs the pear smaller, which makes it easier to grasp, so its rollout succeeds. Here the disagreement stems from a reconstruction error that SceneRig avoids.
Episode 2 (π0.5, Fruit-to-plate) (real: failure; SceneRig and SimFoundry: success). The real robot places the banana and the pear but leaves the apple beside the plate. A possible reason is appearance: in the real image, the red apple and the red plate look similar, which may make it hard for the policy to notice that the apple is still outside the plate, whereas both simulated renderings show slightly higher contrast between them.
Response to Reviewer Xv61 (1/3)
We thank the reviewer for recognizing the importance of the problem, our comprehensive robot experiments, and our choice to let the agent make decisions rather than output numerical values. We address your concerns as follows.
Q1. An ablation that completely removes the agent is needed, since the alignment score can determine whether any adjustment is needed.
Thank you for the suggestion. We ran this ablation on all 80 scenes with Claude Opus 5, replacing only the pose-refinement agent with a fixed rule. The earlier agentic stages are shared. The heuristic refines objects in support order, lowest alignment score first, applies
Moveto position, rotation, and scale, keeps a move only if the score rises, and uses up to threeMovecalls per object. In the table below, source-view, VLM, and cost columns cover all 80 scenes; all-view (35 orbit views for synthetic scenes), ARI, and layout columns cover the 44 simulator scenes. VLM scores are mean-centered Bradley–Terry scores from a separate comparison of these three variants, and time is wall-clock pose refinement per scene. We added this ablation to Appendix D.1 of the revised paper.Movecalls / scene ↓The two variants are close on image metrics, and the heuristic has higher ARI. The agent places objects more accurately in 3D (F1@5 0.804 vs. 0.778) and is preferred by the VLM judges for layout and appearance, while using fewer than half as many
Movecalls and spending less time on pose refinement per scene.Our visual comparison illustrates why score-based acceptance is insufficient. The mustard bottle is upright in the input, but lies on its side in the shared starting reconstruction. The heuristic improves its alignment score from 0.109 to 0.223 yet leaves this pose error unresolved. The agent inspects the render against the input, recognizes that the bottle should stand upright, and corrects its pose through simulated edits. The essential contribution is this inspect–edit–simulate–inspect loop: visually assess the outcome and retain, revert, or revise the edit, rather than treating a score increase as sufficient evidence that the edited pose matches the object in the original input image.