- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes an agentic framework that reconstructs simulation-ready 3D indoor scenes from a single image. Specifically, a detection agent corrects the object inventory, a VLM infers a physical scene graph, and SAM 3D assets are placed in a gravity-aligned metric frame. The poses and scales are then jointly optimized with mask/depth alignment, collision, finite support region, and room losses. Finally, a Mujoco-guided VLM agent applies bounded pose edits to fix remaining unstable objects. Results show better stability and mostly better geometric metrics than 6 baselines on 56 scenes.
Critical strengths
The problem is important, and the pipeline is well engineered with clear details. Comparisons cover many recent baselines, and both cumulative and leave-one-out loss ablations are provided.
Critical weaknesses
- The technical contribution over prior work is not clear to me. Most components come from existing works: support-relation constraints (CAST), agentic scene graph with simulator-guided refinement (REST3D), and finite support regions (HSM, SceneSmith). The paper should state more explicitly what is new compared to REST3D and CAST.
- According to Table3, S0 already reaches 74.85% stability, compared to 10.62% for SAM 3D. This suggests most of the gain comes from projecting objects onto their support surfaces. An ablation on the detection agent and the VLM scene graph (eg. replacing them with simple heuristics) would help show whether the agentic parts are essential.
- It would be beneficial to cite and discuss PAT3D: Physics-Augmented Text-to-3D Scene Generation (ICLR 2026), which also optimizes scene layouts with differentiable simulation.
Questions for authors
- What are the runtime and VLM token cost for reconstructing one scene?
- How often are the fallback rules for invalid scene graphs triggered, and what are typical failure cases (eg. the 9% objects remaining unstable)? Showing a few failure cases would help.
W1. ImScene couples finite mesh-patch contact parameterization with joint source-view and physical optimization: contact height follows the patch while planar translation, yaw and scale optimize mask/depth alignment, collisions and containment. CAST uses relation-specific SDF costs; REST3D uses hierarchical six-DoF cross-entropy search scored by simulation and canonical-layout consistency. HSM populates hierarchical support regions with motifs, and SceneSmith uses designer, critic and orchestrator agents for generation. Our focus is preserving alignment to a given observation during physical correction.
W2. Support projection contributes substantially by resolving many penetrations and unstable placements, but further pose optimization remains necessary. Replacing the VLM graph with a geometric heuristic, fixing assets, observations, raw poses and corrected inventory, reduces stability from 89.49% to 74.54%, raises collisions from 9.28% to 17.19% and lowers IoU from 0.6359 to 0.4742. This isolates the graph's downstream effect through contact projection and optimization, conditional on the corrected inventory.
Using 963 final instances with author-reviewed categories and inherited detection boxes as references, we count a detection as correct when one-to-one box matching gives IoU ≥0.5 and its category matches an accepted reference label. Misses and mislabels remain in the denominator. Scene-averaged coverage improves from 91.21% to 93.94% with the Agent (867/963 to 903/963 overall), through 28 category corrections and 8 additional correct matches.
W3. Thank you; we will cite and discuss PAT3D. It generates scenes from text, using a synthesized image for assets and VLM-derived structure, then differentiable rigid-body contact simulation to optimize initial configurations toward scene-tree-consistent equilibria. ImScene reconstructs a given observation using mask/depth alignment and mesh-grounded contact constraints, followed by separate MuJoCo repair with preservation budgets.
Q1. Across all 56 scenes, mean times are 7.40 s for single-pass detection, segmentation and depth, 19.07 s for support-graph inference, 124.0 s for joint optimization (median 75.9 s), and 60.95 s for matched VLM-guided repair. SAM3D takes roughly 30 s/object on one NVIDIA RTX 4090. Graph inference averages 1.02 requests (USD 0.0125/scene); repair averages 5.98 requests and 1.91 budgeted simulations (USD 0.75/scene).
Q2. Graph fallback occurred in 0/56 scenes. After three unsuccessful requests, fallback will assign ground support to missing-parent objects and redirect one edge per remaining cycle to ground. These structural checks do not guarantee correct support assignments.
After original repair, 92/963 objects fail stability thresholds: 87 for rotation, 70 for displacement and six for speed, with overlaps. These counts pool objects; stability averages scenes equally. Replaying all 92 outcomes with geometry, initial pose, mass, friction and gravity unchanged identifies three failure mechanisms. For 26/92, instability depends on motion of supporting or neighboring objects: fixing all other objects makes the target stable. Another 8/92 remain unstable with fixed surroundings but stabilize when contacts outside the support-ancestor chain are disabled, identifying interference from non-support contacts. The remaining 58/92 still rotate/tip (55) or translate/slide (3) after both interventions: their local contact configuration cannot maintain the initial pose under gravity even without moving neighbors or non-support object contacts.