acceptodds
Under review as a conference paper at ICLR 2027

IMSCENE: AN AGENTIC FRAMEWORK FOR RECONSTRUCTING SIMULATION-READY 3D SCENES FROM A SINGLE IMAGE

Abstract

Physically reliable 3D scenes are essential for embodied navigation, robotic manipulation, and interactive environment simulation. Existing image-to-3D methods generate plausible objects and recover scene layouts, but inconsistencies between generated geometry, estimated poses, and inter-object relationships can still lead to interpenetration, floating objects, and unstable support. Resolving these inconsistencies from a single image requires reconciling incomplete visual evidence with geometric and physical constraints on the reconstructed scene. We present ImScene, an agentic framework for reconstructing editable 3D scenes from a single image. A detection agent corrects missing, duplicate, and mislabeled instances, while a vision-language model constructs a physical scene graph encoding support, containment, and attachment relations. Next, we reconstruct a scene point cloud from estimated depth and establish a gravity-aligned metric coordinate system. Guided by the scene graph, a differentiable optimizer then jointly refines the poses and scales of objects generated by an image-to-3D model, using image and depth alignment, surface contact constraints, and penalties for interpenetration and unsupported placement. Finally, a simulator-guided agent identifies unstable object placements from simulated motion and support contacts, then applies bounded pose adjustments or rigid-body settling to improve stability while maintaining consistency with the input image. Comprehensive comparisons demonstrate improved consistency with the input image and greater physical plausibility, providing a stronger foundation for subsequent scene editing and simulation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
single-view 3D reconstruction, agentic scene generation, vision-language models, differentiable optimization, physical plausibility

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_amK1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework that reconstructs simulation-ready 3D indoor scenes from a single image. Specifically, a detection agent corrects the object inventory, a VLM infers a physical scene graph, and SAM 3D assets are placed in a gravity-aligned metric frame. The poses and scales are then jointly optimized with mask/depth alignment, collision, finite support region, and room losses. Finally, a Mujoco-guided VLM agent applies bounded pose edits to fix remaining unstable objects. Results show better stability and mostly better geometric metrics than 6 baselines on 56 scenes.

Critical strengths

The problem is important, and the pipeline is well engineered with clear details. Comparisons cover many recent baselines, and both cumulative and leave-one-out loss ablations are provided.

Critical weaknesses
  • The technical contribution over prior work is not clear to me. Most components come from existing works: support-relation constraints (CAST), agentic scene graph with simulator-guided refinement (REST3D), and finite support regions (HSM, SceneSmith). The paper should state more explicitly what is new compared to REST3D and CAST.
  • According to Table3, S0 already reaches 74.85% stability, compared to 10.62% for SAM 3D. This suggests most of the gain comes from projecting objects onto their support surfaces. An ablation on the detection agent and the VLM scene graph (eg. replacing them with simple heuristics) would help show whether the agentic parts are essential.
  • It would be beneficial to cite and discuss PAT3D: Physics-Augmented Text-to-3D Scene Generation (ICLR 2026), which also optimizes scene layouts with differentiable simulation.
Questions for authors
  • What are the runtime and VLM token cost for reconstructing one scene?
  • How often are the fallback rules for invalid scene graphs triggered, and what are typical failure cases (eg. the 9% objects remaining unstable)? Showing a few failure cases would help.
1 comment

W1. ImScene couples finite mesh-patch contact parameterization with joint source-view and physical optimization: contact height follows the patch while planar translation, yaw and scale optimize mask/depth alignment, collisions and containment. CAST uses relation-specific SDF costs; REST3D uses hierarchical six-DoF cross-entropy search scored by simulation and canonical-layout consistency. HSM populates hierarchical support regions with motifs, and SceneSmith uses designer, critic and orchestrator agents for generation. Our focus is preserving alignment to a given observation during physical correction.

W2. Support projection contributes substantially by resolving many penetrations and unstable placements, but further pose optimization remains necessary. Replacing the VLM graph with a geometric heuristic, fixing assets, observations, raw poses and corrected inventory, reduces stability from 89.49% to 74.54%, raises collisions from 9.28% to 17.19% and lowers IoU from 0.6359 to 0.4742. This isolates the graph's downstream effect through contact projection and optimization, conditional on the corrected inventory.

Using 963 final instances with author-reviewed categories and inherited detection boxes as references, we count a detection as correct when one-to-one box matching gives IoU ≥0.5 and its category matches an accepted reference label. Misses and mislabels remain in the denominator. Scene-averaged coverage improves from 91.21% to 93.94% with the Agent (867/963 to 903/963 overall), through 28 category corrections and 8 additional correct matches.

W3. Thank you; we will cite and discuss PAT3D. It generates scenes from text, using a synthesized image for assets and VLM-derived structure, then differentiable rigid-body contact simulation to optimize initial configurations toward scene-tree-consistent equilibria. ImScene reconstructs a given observation using mask/depth alignment and mesh-grounded contact constraints, followed by separate MuJoCo repair with preservation budgets.

Q1. Across all 56 scenes, mean times are 7.40 s for single-pass detection, segmentation and depth, 19.07 s for support-graph inference, 124.0 s for joint optimization (median 75.9 s), and 60.95 s for matched VLM-guided repair. SAM3D takes roughly 30 s/object on one NVIDIA RTX 4090. Graph inference averages 1.02 requests (USD 0.0125/scene); repair averages 5.98 requests and 1.91 budgeted simulations (USD 0.75/scene).

Q2. Graph fallback occurred in 0/56 scenes. After three unsuccessful requests, fallback will assign ground support to missing-parent objects and redirect one edge per remaining cycle to ground. These structural checks do not guarantee correct support assignments.

After original repair, 92/963 objects fail stability thresholds: 87 for rotation, 70 for displacement and six for speed, with overlaps. These counts pool objects; stability averages scenes equally. Replaying all 92 outcomes with geometry, initial pose, mass, friction and gravity unchanged identifies three failure mechanisms. For 26/92, instability depends on motion of supporting or neighboring objects: fixing all other objects makes the target stable. Another 8/92 remain unstable with fixed surroundings but stabilize when contacts outside the support-ancestor chain are disabled, identifying interference from non-support contacts. The remaining 58/92 still rotate/tip (55) or translate/slide (3) after both interventions: their local contact configuration cannot maintain the initial pose under gravity even without moving neighbors or non-support object contacts.

Reviewer_TrNLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ImScene reconstructs editable 3D scenes from a single RGB image. It combines agentic instance correction, physical scene-graph inference, depth-based metric initialization, contact-constrained joint pose and scale optimization, and simulator-guided stabilization. The optimizer grounds inferred relations in finite regions of generated meshes, while the final agent accepts physical improvements subject to visual and geometric preservation budgets. Evaluation covers 56 indoor scenes, with reference-geometry comparisons on 3D-FRONT and ScanNet++ and five-second rigid-body stability tests in MuJoCo.

Critical strengths
  1. The framework addresses a practical scene-reconstruction problem by integrating visual fidelity and physical plausibility in a unified pipeline.
  2. The experiments demonstrate improvements in scene stability and geometric reconstruction, with further stability gains from agentic refinement.
Critical weaknesses
  1. Restricted refinement limits recovery from initialization errors. Support regions are selected using the child's initial bottom height and lateral position; parent and region bindings are frozen after a one-time feasibility check. Optimization adjusts only two horizontal translations, yaw, and uniform scale, with height determined by contact and no independent roll or pitch correction (Section 3.3). Support heights follow parent transforms, but incorrect parent or region assignments cannot be revised during refinement. If a sideways vase is incorrectly initialized upright, this parameterization cannot recover its tilt and corresponding side contact. A wrong shelf binding can similarly conflict with depth or mask evidence, forcing optimization to compromise within an incorrect configuration. The paper neither reports support-assignment accuracy nor tests recovery from these initialization errors, leaving the robustness of the restricted formulation unestablished.
  2. Insufficient analysis of parameter robustness. The loss weights, support margins, and repair thresholds jointly govern visual and physical trade-offs, but their selection procedure and sensitivity are not analyzed. In particular, loss-removal ablations do not establish whether the reported gains remain consistent under different nonzero weightings of collision and visual objectives.
Questions for authors
  1. How accurate are the support-parent and shelf-level assignments? How does the method handle incorrect assignments or initial roll/pitch errors under fixed bindings and restricted pose optimization?
  2. How were the loss weights, support margins, and repair thresholds selected? Were they fixed across all scenes, and how sensitive are the results to the relative weighting of collision and visual losses?
2 comments

Q1. Parent errors decrease from 22/963 to 20/963 after initialization. Shelf accuracy is 148/152 (97.37%) after initialization and optimization, conditional on correct parents and adjudicable correspondence.

The joint optimizer uses four variables per ordinary object: two planar translations, yaw and uniform scale, with contact-derived height and fixed roll/pitch. A wrong parent or shelf binding therefore imposes an incorrect contact height or region that optimization cannot revise, potentially compromising mask/depth alignment. For example, clothing remains on a lower shelf despite upper-shelf image evidence. The agentic stage additionally provides full 6-DoF rigid-body settling for designated loose objects such as cushions, allowing roll/pitch correction and non-horizontal contacts within pose bounds and visual/geometric preservation budgets. This branch retains the declared support and does not reassign parents/shelves.

Perturbations select the first eligible movable leaf in fixed name order after initialization. We rebind the selected object to the nearest alternative patch on the same parent with a height difference of ≥8 cm; +15° local pitch separately changes one non-mounted leaf per scene. Across all 56 scenes, default/alternative-patch/pitch stability is 89.49%/88.26%/80.47%, mask IoU is 0.6359/0.5804/0.6214, and alternative patches raise collisions from 9.28% to 13.21%. These tests precede agentic repair and measure optimizer sensitivity with bindings and tilt fixed.

Q2. Parameters were selected heuristically during development, without systematic tuning. The weights rescale losses with different numerical magnitudes and gradients: mask IoU loss, normalized depth distance and sparse penetration penalties contribute differently to the joint update. They balance the influence of these terms and the trade-off between physical correction and image fidelity. All 56 archived Full runs share collision/mask/depth weights 60/1/1, support weight 4, prior weight 0.1, margin 0.01 m, and 320 steps plus up to three 80-step refinement rounds triggered by collision checks.

Retrospective paired tests preserve each scene's configuration with frozen observations and a fixed seed. With mask/depth weights fixed at 1/1, collision weights 30/60/120 yield stability 89.56/89.49/89.37%, collisions 10.36/9.28/7.78% and IoU 0.6383/0.6359/0.6379. Margins 0.005/0.01/0.02 m yield stability 89.95/89.49/89.33%, collisions 8.62/9.28/9.26%, IoU 0.6384/0.6359/0.6333 and COM-outside-support 1.50/0.92/0.33%. COM evaluation uses transformed parent polygons and a fixed zero signed-distance threshold; margins alter only the penalty. Stability changes little over these ranges, while collision weights reduce collisions and larger margins improve containment with lower IoU.

Thank you to the author for the reply. I think all my concerns have been resolved. I will raise my score.

Reviewer_xs6QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ImScene reconstructs editable, physically plausible 3D scenes from a single RGB image. It combines pretrained detection, segmentation, depth, and object-generation models with VLM-based instance correction and physical-relation inference. Generated objects are placed in a gravity-aligned frame and bound to finite support regions. A differentiable optimizer refines poses and scales using image/depth alignment and geometric constraints, followed by a simulator-guided agent that proposes bounded repairs. Evaluation covers 56 scenes, including geometric comparisons, release-based stability tests, ablations, and three scripted robotic-interaction demonstrations.

Critical strengths
  1. The system addresses a practical limitation of compositional reconstruction: individually plausible objects may form an inconsistent assembly with penetration, floating, or unstable support. Combining source-view alignment, finite support regions, collision constraints, and simulation feedback is a sensible approach with potential practical utility.

  2. The experiments provide evidence that geometric refinement improves physical consistency. The cumulative and individual-loss ablations are informative, particularly about the contribution of collision handling. The paper also reports the trade-off between geometric alignment and physical stability rather than presenting improvement as uniform across all metrics.

Critical weaknesses
  1. The methodological advance appears incremental, and the distinctive benefits are not sufficiently isolated. Relation-guided physical correction already appears in CAST, and REST3D, which is included in the experiments, is an especially close predecessor combining physical scene understanding with constrained refinement. The concrete additions here concern support-region constraints, the joint objective, and simulator-guided repair. These may be useful, but the evidence does not yet establish a substantial advance over existing refinement strategies. The final agent improves stability from 89.49% to 90.80%; there is no matched comparison against support-aware rigid-body settling, simple local search using the same acceptance rules, or a general-purpose visual agent with the same assets and feedback. The two qualitative Astra examples are insufficient to establish this distinction. A controlled comparison would be more informative than requiring unavailable implementations of every related method.

  2. The evaluation leaves uncertainty about both comparative physical reliability and downstream utility. First, ImScene and REST3D use inferred installation relations to exempt wall/ceiling installations from gravity, whereas graph-free baselines do not receive corresponding labels. This measures a legitimate system capability, but confounds installation labeling with placement quality; a common-label control is needed to isolate the latter. Second, support bindings are frozen after initialization, and ordinary optimization is restricted to horizontal support regions. Targeted evidence on incorrect bindings, clutter, inclined contacts, and complex support configurations is limited. Finally, five-second release tests establish a restricted form of stability, while the three supplementary interactions use predefined IK targets and action sequences. Without task success rates, failure analysis, or matched interaction baselines, these demonstrations provide limited evidence for reliable robotic usability.

Overall, I recognize the engineering value and demonstrated improvements in static physical consistency, but remain unconvinced that the methodological increment and downstream benefits are sufficiently established. My provisional recommendation is weak rejection.

Questions for authors
  1. Which specific design changes account for gains over prior relation-guided physical refinement? Can the authors provide a comparison using the same generated assets and initialization, or existing evidence that isolates this contribution?

  2. For the simulator-guided stage, how does the agent compare with rigid-body settling or simple local search under the same feedback, acceptance criteria, and simulation-call budget? A tool-equipped VLM baseline, for example a VIGA-style loop or Astra, would also help assess whether the specialized framework is necessary; model, tool access, and cost should be reported and controlled.

  3. How do the physical rankings change if all methods receive the same installation labels? Please distinguish the benefit of inferring attachments from the benefit of improving geometry and placement.

  4. How frequently are initial parent/surface assignments incorrect, and what happens when they are? Can the authors provide difficult cases and failures involving occlusion, clutter, or non-horizontal support?

  5. Is there task-level evidence beyond the three scripted demonstrations, such as matched grasping/pushing success rates across scenes? If not, the downstream claims should be limited to the demonstrated static stability and qualitative interaction examples.

1 comment

Q1. CAST uses relation-specific SDF costs; REST3D uses simulator-scored pose search. ImScene derives contact height from finite mesh patches, jointly optimizing planar translation/yaw/scale against source-mask/depth alignment, collision and containment losses. Matched assets/initialization (56 scenes): refinement raises stability 74.85% (S0)→89.49% (pre-repair) and lowers collisions 30.32%→9.28%. Without collision suppression: 77.54% stability, 29.09% collisions. Support containment reduces COM-outside-support 6.22%→0.64%, with smaller stability gain: 88.87%→89.49%.

Q2. The four controllers share assets, initialization, feedback, edit bounds and acceptance criteria (56 scenes, ≤8 budgeted simulation calls/repair scene). Local search retains support projection/collision checks with six support-plane translation directions, ±3° support-normal rotations and targeted settling. Both VLM workflows share Qwen3.8 Max, observations/tools/execution; generic instructions are observe–act–verify, ours inspect failing objects/support ancestors for support-aware edits.

S: stability (%), R: recovered non-installed objects (both 5 s/10 s); means: T runtime (s), C simulation/VLM calls. SRTCBefore repair89.49/89.37−−−Settling89.61/89.502/215.443.61/0Local search90.19/90.086/630.703.75/0Generic VLM89.82/89.713/349.790.64/6.02Our VLM90.14/90.026/660.951.91/5.98\begin{array}{lrrrr}&S&R&T&C\\\\\text{Before repair}&89.49/89.37&-&-&-\\\\\text{Settling}&89.61/89.50&2/2&15.44&3.61/0\\\\\text{Local search}&90.19/90.08&6/6&30.70&3.75/0\\\\\text{Generic VLM}&89.82/89.71&3/3&49.79&0.64/6.02\\\\\text{Our VLM}&90.14/90.02&6/6&60.95&1.91/5.98\end{array}

Local search/ours have similar stability averaged equally across scenes; generic recovers fewer objects at similar VLM calls. Eight generic runs (ours: none) stop after the format retry, scored at their last accepted scenes; tool-use reliability is included.

Q3. Rankings remain unchanged. Native installation labels: ImScene (inferred), REST3D (scene tree) only. ImScene's frozen source-image labels transfer via shared instance IDs (SAM3D) or source-mask matching (MIDI/SceneGen/Gen3DSR; IoU ≥0.5 and ≥0.1 above next-best). Unmatched objects: AI review of source RGB/native instance masks. A shared wall/ceiling installation/opening definition enables evaluation of graph-free methods without physical graph prediction.

n: valid scenes; stability (%; 5 s/10 s). nNativeCommonImScene5689.49/89.3789.49/89.37REST3D5686.19/86.2186.19/86.21SAM3D5610.62/10.5521.89/21.71MIDI555.41/5.3014.39/14.16SceneGen543.15/3.5411.81/11.96Gen3DSR550.36/0.188.07/7.89\begin{array}{lrrr}&n&\text{Native}&\text{Common}\\\\\text{ImScene}&56&89.49/89.37&89.49/89.37\\\\\text{REST3D}&56&86.19/86.21&86.19/86.21\\\\\text{SAM3D}&56&10.62/10.55&21.89/21.71\\\\\text{MIDI}&55&5.41/5.30&14.39/14.16\\\\\text{SceneGen}&54&3.15/3.54&11.81/11.96\\\\\text{Gen3DSR}&55&0.36/0.18&8.07/7.89\end{array}

Common labels improve graph-free baselines, still below ImScene. Objects, geometry, collision proxies and per-method scoring stay fixed, isolating label effects.

Q4. Occlusion/clutter can obscure parents/shelves. Wrong frozen bindings impose incorrect contact-height/containment constraints, so fewer collisions can coexist with support/height errors and poorer image alignment. In one cluttered shelf case, clothing remains on a lower shelf despite upper-shelf image evidence. Initialization: parent errors 22/963→20/963; shelf accuracy is 148/152 (97.37%), given correct parents and adjudicable correspondence.

After initialization, we rebind one eligible leaf to the nearest same-parent patch ≥8 cm away in height. Across 56 scenes, stability falls 89.49%→88.26%, mask IoU 0.6359→0.5804, and collisions rise 9.28%→13.21%. The 4-DoF optimizer requires horizontal support, with no independent roll/pitch correction. Agentic settling can handle non-horizontal contacts of designated loose objects using full 6-DoF poses, subject to repair bounds and preservation checks.

Q5. Frozen π0.5-DROID basic supported pushing: 13/56 scenes (23.21%) succeed on the first attempt, 16/56 (28.57%) within two independently reset 20 s attempts. RGB, robot state and text drive all actions after initialization. We initialize the gripper using the target position (5 cm above it) and choose from 12 fixed bases. Success: hand–target contact, post-contact horizontal movement and separation from a stationary-robot control both ≥2 cm, upward non-robot support contact, ≤3 cm lowest-point height change, and speed ≤0.15 m/s for five consecutive steps (0.33 s); direction/orientation are unrestricted.

Grasp-and-lift: five selected scenes, narrow targets (1.5–7.5 cm), fewer approach obstacles, valid assisted initialization. Frozen π0.5-DROID joint-position policy; geometry-based IK fallback after failure and scene reset (≤2 attempts, 20 s each). Gripper mount/cameras match the DROID reference; pad sliding friction is 2.0, with scene geometry/object dynamics unchanged. Success: ≥8 cm lowest-point lift versus both initial state and stationary-robot control, no non-robot contact, and bilateral finger contact for 1 s. Pure π0.5: 1/5; simulator-geometry IK approaches, closes, lifts and holds, recovering three more: 4/5 (80.0%). These development results show assisted grasp feasibility with π0.5 plus a controller in selected scenes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.