SceneRig: An Agentic System for Simulation-Ready 3D Scene Reconstruction from Single Images

Simulation is a powerful tool for robotics, but building diverse, simulation-ready scenes at scale remains difficult. Reconstructing scenes from single images is a promising alternative, but visual similarity alone is insufficient: objects must also be recovered, accurately placed, and physically grounded. Existing methods that rely on post-hoc and simplistic physics settling often misplace objects, producing scenes that look correct but behave poorly under manipulation. We present SceneRig, an agentic system that turns a single image into an editable, simulation-ready scene. SceneRig combines geometric, visual, and physical tools to reconstruct a faithful 3D scene. An agent then performs edit-and-validate loops inside a physics simulator to guarantee physical grounding. Across diverse scenes, SceneRig improves instance segmentation, object-centric appearance, and 3D placement while keeping scenes physically stable. Furthermore, SceneRig is effective for downstream robotics: it predicts real-robot policy outcomes more accurately than baseline methods and more reliably replays recorded manipulations without trajectory adaptation. Link to the anonymous [project page](https://scenerig.github.io/).