- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SceneCast presents an annotation-free framework for indoor scene synthesis and completion. It constructs training layouts from generated videos using a reconstruction and recognition pipeline, with ViTrim selecting high-quality video prefixes. An autoregressive Transformer predicts object categories and bounding-box parameters, followed by reinforcement learning with geometric and LLM-based rewards. The predicted layouts are instantiated using retrieved furniture assets. Experiments evaluate layout quality, overlap, and conditional completion on bedroom and living/dining-room scenes.
Critical strengths
-
The framework explores a useful alternative to manually annotated 3D layout data by combining synthetic videos, automatic reconstruction, and quality-aware data selection.
-
The code-execution mechanism provides explicit geometric measurements for the LLM reward. The reward and data-curation ablations offer evidence for the benefits of this design, and the method improves the reported layout metrics over the selected baselines.
Critical weaknesses
-
Incomplete comparison with relevant scene synthesis methods. The quantitative evaluation focuses on ATISS, DiffuScene, and VIPScene, while important related methods such as SceneWeaver [1], OptiScene [2], PhyScene [3], and InstructScene [4] are missing from the comparisons. These methods address relevant aspects of layout generation, spatial constraints, and LLM-assisted synthesis. Without comparisons under compatible settings, the advantages of SceneCast over the broader scene synthesis literature remain unclear.
-
Unclear cost–benefit of the data pipeline and reliability of retrieved assets. The framework combines video generation and multiple foundation models to construct pseudo-labels, yet ultimately learns only object categories and bounding boxes. Its benefit over simpler data-construction alternatives is not established relative to the computational cost. Moreover, asset retrieval based on category and box proportions does not guarantee consistent geometry, contact, or collision behavior in the final scene. The relationship to reference-conditioned methods such as SAM3D [5] and SimRecon [6] also deserves discussion and comparison where applicable.
[1] SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent. NeurIPS, 2025.
[2] OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization. NeurIPS, 2025.
[3] PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI. CVPR, 2024.
[4] InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior. ICLR, 2024.
[5] SAM 3D: 3Dfy Anything in Images. CVPR, 2026.
[6] SimRecon: SimReady Compositional Scene Reconstruction from Real Videos. CVPR, 2026.
Questions for authors
-
The model predicts each object’s rotation around the vertical axis. How is this rotation applied to a retrieved asset whose canonical orientation may differ? Is asset orientation standardized before placement?
-
Can SceneCast accommodate valid bounding-box overlaps, such as a chair partially tucked under a table? How does the overlap penalty distinguish these configurations from actual collisions, and is interpenetration checked on the final retrieved meshes?
-
Can the framework be extended to support more detailed scene or object specifications, such as furniture style, material, or a particular object arrangement? Which parts of the current representation and asset retrieval would need to change?
-
Where feasible, please add comparisons with relevant recent methods, such as SceneWeaver [1], OptiScene [2], PhyScene [3], and InstructScene [4]. If direct comparison is unsuitable, please clarify the differences in task settings and supported capabilities.
10 comments
Response to Reviewer Ktia Q3 and Q4.
Q3
Can the framework be extended to support more detailed scene or object specifications, such as furniture style, material, or a particular object arrangement? Which parts of the current representation and asset retrieval would need to change?
R3
The current representation deliberately predicts only category, position, yaw, and size, because these are the attributes that can be recovered reliably from the video-derived supervision. The framework can be extended, but the following extensions require additional supervision and retrieval support; they are not capabilities claimed by the current model.
Fine-grained semantic labels. Fine-grained categories, such as a three-seat sofa rather than a sofa, require an expanded vocabulary, matching Grounded-SAM prompts, retraining of the category head, and retrieval assets with the corresponding labels. The autoregressive layout representation itself need not change, but the pseudo-label and asset pipelines do.
Specified arrangements. A user specification can be represented as a partial layout together with spatial constraints. Supporting this setting would require a text/constraint-conditioning interface and training examples for that interface. The autoregressive decoder could then complete the unspecified objects, while the executable geometric checks could be extended to verify explicit constraints such as “sofa faces the TV” or “bed against the north wall.” This is a natural extension, but is not an input modality supported in the current experiments.
Style and material. These require new per-object attributes and a style-aware retrieval stage. The current video-derived layouts provide category and geometry but not reliable style or material labels; therefore, this extension would require VLM-generated attribute pseudo-labels or a supervised attribute set, together with a style-consistent asset library. It would partially weaken the current annotation-free assumption. We will clarify these requirements and the scope of the present model in the revision.
Q4(W1)
Where feasible, please add comparisons with relevant recent methods, such as SceneWeaver [1], OptiScene [2], PhyScene [3], and InstructScene [4]. If direct comparison is unsuitable, please clarify the differences in task settings and supported capabilities.
R4
We thank the reviewer for suggesting SceneWeaver, OptiScene, PhyScene, and InstructScene. We added an adapted PhyScene evaluation on living rooms under the same category mapping and layout evaluator as ours. PhyScene obtains 145.46 FID, 7.53 KID, 94.8 SCA, and 8.70 CKL. This result is an output-level transfer comparison: PhyScene was originally designed for complete-layout supervision and physical-interaction-aware synthesis, whereas our model is learned from partial video-derived layouts. We therefore do not interpret this result as a comparison of supervision efficiency or of physical-interaction capabilities.
We also investigated InstructScene. Its native task requires a text instruction, which is not available in our setting. Supplying a vLLM-derived text condition adds an external semantic extraction stage and produces a substantially shifted input distribution (FID 260.2 in our preliminary transfer run). We therefore do not present this number as a meaningful direct benchmark. We will instead discuss InstructScene as a text-conditioned method and state this task mismatch explicitly.
Table 3. Living/dining-room results under our common category mapping and evaluation protocol. PhyScene is adapted from its original physical-interaction-oriented setting.
| Method | FID (↓) | KID (↓) | SCA (≈50%) | CKL (↓) |
|---|---|---|---|---|
| PhyScene (adapted) | 145.46 | 7.53 | 94.80 | 8.70 |
| Ours | 128.57 | 5.64 | 62.58 | 1.38 |
Cost-benefit, asset reliability, and reconstruction-based methods
W2
Unclear cost–benefit of the data pipeline and reliability of retrieved assets. The framework combines video generation and multiple foundation models to construct pseudo-labels, yet ultimately learns only object categories and bounding boxes. Its benefit over simpler data-construction alternatives is not established relative to the computational cost. Moreover, asset retrieval based on category and box proportions does not guarantee consistent geometry, contact, or collision behavior in the final scene. The relationship to reference-conditioned methods such as SAM3D [5] and SimRecon [6] also deserves discussion and comparison where applicable.
R
We thank the reviewer for raising the cost-benefit and final-asset questions. The goal of SceneCast is not to claim that its video-to-layout construction is cheaper than a pure forward-pass generator. Rather, it trades a one-time video-generation/reconstruction cost for avoiding task-specific human 3D-layout annotation and for obtaining a reusable layout policy. The submitted paper reports the model-training cost on one RTX 8000: approximately 36 hours for backbone pretraining and 15--20 minutes for SAC fine-tuning, including LLM inference and code execution. The hybrid schedule keeps the latter to only a few dozen LLM calls per run.
What the present ablation establishes is a data-quality benefit under a fixed downstream model: on bedrooms, no filtering and VisionReward+VideoPhy-only curation obtain FID 133.9 and 132.2, respectively, whereas ViTrim with the spatial-consistency signal and entropy weighting obtains 111.1. This supports the need for geometry-aware curation of video-derived supervision.
SAM3D and SimRecon are complementary reference-conditioned reconstruction systems: they aim to recover the geometry of a particular input image/video. SceneCast instead learns a distributional layout prior that can synthesize or complete layouts from an empty or partial state. Their native reconstruction metrics and reference-fidelity objectives are therefore not directly comparable to our distributional layout metrics. A controlled comparison would require substituting their reconstructed layouts into the same pseudo-label pipeline and retraining the same policy. We will discuss this possible use as future work.
Response to Reviewer Ktia Q1 and Q2.
Q1
The model predicts each object’s rotation around the vertical axis. How is this rotation applied to a retrieved asset whose canonical orientation may differ? Is asset orientation standardized before placement?
R1
Thank you for pointing this out. We do not estimate an asset orientation separately at test time. Instead, each asset is canonicalized once at import: we convert it to metres, use a Y-up coordinate system, translate its bounding-box centre to the origin, and rotate it by a multiple of 90∘ so that its canonical front axis is +Z. We store the associated correction ϕa∈{0∘,90∘,180∘,270∘} as import metadata and store the canonical extents ea.
For canonical vertices vacan, placement uses the predicted box centre c, predicted yaw θ, and predicted extents epred:
\qquad s_a=e^{\mathrm{pred}}\oslash e_a.$$ Because $Y$ is the up axis, yaw is a rotation about $Y$. The import correction $\phi_a$ is already incorporated in $v_a^{\mathrm{can}}$; it is not applied a second time in the equation above. Equivalently, an implementation retaining raw vertices would use $R_y(\theta+\phi_a)$ on the raw mesh, but the two parameterizations must not be combined. The generator therefore predicts a continuous layout yaw $\theta$ in a category-level frame, while $\phi_a$ is a fixed per-asset import correction. Replacing one retrieved candidate with another candidate of the same category leaves the predicted layout unchanged; only the candidate-specific mesh and scale ratio $s_a$ change. No orientation parameter is fitted per scene at test time. Only assets for which a front direction matters require a unique orientation label; rotationally symmetric assets are invariant to this choice. The asymmetric categories contain approximately 3,092 distinct assets, and approximately 2,119 assets account for 90% of their placements. This is a one-time import operation, not a per-scene annotation. **Table 1.** Consistency audit of the per-asset canonical frame. The residual scale variation is handled by the per-placement factor $s_a$ in the placement equation above. | Statistic | Value | Interpretation | |---|---:|---| | Assets audited ($\geq 3$ instances) | 4,708 | --- | | CV of canonical extents, median (p90) | 6.4% (27.1%) | scale variation | | Same CV, axis-aligned instances only | 6.4% | not caused by yaw | | Assets with width/depth-swap signature | 4.6% | corrected at import | **Table 2.** Scope of the one-time orientation metadata for distinct 3D-FUTURE assets in the 9,483-room corpus. | Category | Distinct assets | Assets covering 90% of placements | |---|---:|---:| | Double bed | 518 | 177 | | Sofa | 1,219 | 666 | | Cabinet | 1,191 | 1,112 | | TV stand | 164 | 164 | | **Total asymmetric categories** | **≈3,092** | **≈2,119** | ## Q2 Can SceneCast accommodate valid bounding-box overlaps, such as a chair partially tucked under a table? How does the overlap penalty distinguish these configurations from actual collisions, and is interpenetration checked on the final retrieved meshes? ## R2 We agree that pairwise 3D bounding-box IoU is not a complete measure of physical validity. Our hard-validity term penalizes only severe box overlap, with $\mathrm{IoU}>0.5$. A plausible configuration such as a chair partly tucked under a table has much smaller box overlap ($\mathrm{IoU}\approx 0.1$ in our example) and is therefore not penalized by this term. Thus, our geometric reward discourages severe box interpenetration; it does not declare every overlap invalid, nor do we claim that it performs mesh-level collision checking. This limitation follows from the box-level representation used by SceneCast and by the layout-based baselines in our comparison. The methods differ in how they regularize overlap, but none turns a single object box into a part-aware collision model. A principled extension would represent each asset with a union of cuboids or a convex decomposition—for example, separate tabletop and legs, or chair seat, back, and legs. Such a representation could distinguish a chair under a table from true interpenetration, but would require part-level pseudo-labels, longer generation sequences, and a larger RL action space. These are not available in the current video-derived pipeline, and we will state this limitation explicitly.