acceptodds
Under review as a conference paper at ICLR 2027

SceneCast: Learning Indoor Scene Synthesis from Curated Synthetic Videos with LLM Reward

Abstract

3D indoor scene synthesis requires generating objects to form a complete and coherent scene. Existing approaches rely heavily on manually annotated datasets, leading to high costs and limited generalization. We present \ours, an annotation-free framework that trains on large-scale, unlabeled videos, addressing two key challenges: (1) Videos provide only partial views of a scene, requiring the model to extrapolate unseen 3D regions, which is inherently error-prone; (2) Raw videos available online are uncurated, suffering from uncontrolled conditions and variable data quality. First, we design a Transformer-based model that predicts objects sequentially and is fine-tuned with reinforcement learning (RL), where an LLM acts as a spatial common-sense evaluator. The LLM provides reward signals through chain-of-thought reasoning and code-executable geometric checks to refine object configurations. Second, we introduce \trim, a data curation approach that generates training videos with controllable video generation models. Crucially, it retains only high-quality prefix clips via multi-criteria quality assessment with an entropy-based weighting strategy. Evaluated on curated synthetic video sets, our annotation-free approach outperforms state-of-the-art supervised methods in scene quality.

Primary area:
generative models
Keywords:
Indoor Scene Generation, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KtiaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SceneCast presents an annotation-free framework for indoor scene synthesis and completion. It constructs training layouts from generated videos using a reconstruction and recognition pipeline, with ViTrim selecting high-quality video prefixes. An autoregressive Transformer predicts object categories and bounding-box parameters, followed by reinforcement learning with geometric and LLM-based rewards. The predicted layouts are instantiated using retrieved furniture assets. Experiments evaluate layout quality, overlap, and conditional completion on bedroom and living/dining-room scenes.

Critical strengths
  1. The framework explores a useful alternative to manually annotated 3D layout data by combining synthetic videos, automatic reconstruction, and quality-aware data selection.

  2. The code-execution mechanism provides explicit geometric measurements for the LLM reward. The reward and data-curation ablations offer evidence for the benefits of this design, and the method improves the reported layout metrics over the selected baselines.

Critical weaknesses
  1. Incomplete comparison with relevant scene synthesis methods. The quantitative evaluation focuses on ATISS, DiffuScene, and VIPScene, while important related methods such as SceneWeaver [1], OptiScene [2], PhyScene [3], and InstructScene [4] are missing from the comparisons. These methods address relevant aspects of layout generation, spatial constraints, and LLM-assisted synthesis. Without comparisons under compatible settings, the advantages of SceneCast over the broader scene synthesis literature remain unclear.

  2. Unclear cost–benefit of the data pipeline and reliability of retrieved assets. The framework combines video generation and multiple foundation models to construct pseudo-labels, yet ultimately learns only object categories and bounding boxes. Its benefit over simpler data-construction alternatives is not established relative to the computational cost. Moreover, asset retrieval based on category and box proportions does not guarantee consistent geometry, contact, or collision behavior in the final scene. The relationship to reference-conditioned methods such as SAM3D [5] and SimRecon [6] also deserves discussion and comparison where applicable.

[1] SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent. NeurIPS, 2025.

[2] OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization. NeurIPS, 2025.

[3] PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AI. CVPR, 2024.

[4] InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior. ICLR, 2024.

[5] SAM 3D: 3Dfy Anything in Images. CVPR, 2026.

[6] SimRecon: SimReady Compositional Scene Reconstruction from Real Videos. CVPR, 2026.

Questions for authors
  1. The model predicts each object’s rotation around the vertical axis. How is this rotation applied to a retrieved asset whose canonical orientation may differ? Is asset orientation standardized before placement?

  2. Can SceneCast accommodate valid bounding-box overlaps, such as a chair partially tucked under a table? How does the overlap penalty distinguish these configurations from actual collisions, and is interpenetration checked on the final retrieved meshes?

  3. Can the framework be extended to support more detailed scene or object specifications, such as furniture style, material, or a particular object arrangement? Which parts of the current representation and asset retrieval would need to change?

  4. Where feasible, please add comparisons with relevant recent methods, such as SceneWeaver [1], OptiScene [2], PhyScene [3], and InstructScene [4]. If direct comparison is unsuitable, please clarify the differences in task settings and supported capabilities.

10 comments
AuthorsOpenReview ↗

Response to Reviewer Ktia Q1 and Q2.

Q1

The model predicts each object’s rotation around the vertical axis. How is this rotation applied to a retrieved asset whose canonical orientation may differ? Is asset orientation standardized before placement?

R1

Thank you for pointing this out. We do not estimate an asset orientation separately at test time. Instead, each asset is canonicalized once at import: we convert it to metres, use a YY-up coordinate system, translate its bounding-box centre to the origin, and rotate it by a multiple of 90∘90^\circ so that its canonical front axis is +Z+Z. We store the associated correction ϕa∈{0∘,90∘,180∘,270∘}\phi_a \in \{0^\circ,90^\circ,180^\circ,270^\circ\} as import metadata and store the canonical extents eae_a.

For canonical vertices vacanv_a^{\mathrm{can}}, placement uses the predicted box centre cc, predicted yaw θ\theta, and predicted extents eprede^{\mathrm{pred}}:

\qquad s_a=e^{\mathrm{pred}}\oslash e_a.$$ Because $Y$ is the up axis, yaw is a rotation about $Y$. The import correction $\phi_a$ is already incorporated in $v_a^{\mathrm{can}}$; it is not applied a second time in the equation above. Equivalently, an implementation retaining raw vertices would use $R_y(\theta+\phi_a)$ on the raw mesh, but the two parameterizations must not be combined. The generator therefore predicts a continuous layout yaw $\theta$ in a category-level frame, while $\phi_a$ is a fixed per-asset import correction. Replacing one retrieved candidate with another candidate of the same category leaves the predicted layout unchanged; only the candidate-specific mesh and scale ratio $s_a$ change. No orientation parameter is fitted per scene at test time. Only assets for which a front direction matters require a unique orientation label; rotationally symmetric assets are invariant to this choice. The asymmetric categories contain approximately 3,092 distinct assets, and approximately 2,119 assets account for 90% of their placements. This is a one-time import operation, not a per-scene annotation. **Table 1.** Consistency audit of the per-asset canonical frame. The residual scale variation is handled by the per-placement factor $s_a$ in the placement equation above. | Statistic | Value | Interpretation | |---|---:|---| | Assets audited ($\geq 3$ instances) | 4,708 | --- | | CV of canonical extents, median (p90) | 6.4% (27.1%) | scale variation | | Same CV, axis-aligned instances only | 6.4% | not caused by yaw | | Assets with width/depth-swap signature | 4.6% | corrected at import | **Table 2.** Scope of the one-time orientation metadata for distinct 3D-FUTURE assets in the 9,483-room corpus. | Category | Distinct assets | Assets covering 90% of placements | |---|---:|---:| | Double bed | 518 | 177 | | Sofa | 1,219 | 666 | | Cabinet | 1,191 | 1,112 | | TV stand | 164 | 164 | | **Total asymmetric categories** | **≈3,092** | **≈2,119** | ## Q2 Can SceneCast accommodate valid bounding-box overlaps, such as a chair partially tucked under a table? How does the overlap penalty distinguish these configurations from actual collisions, and is interpenetration checked on the final retrieved meshes? ## R2 We agree that pairwise 3D bounding-box IoU is not a complete measure of physical validity. Our hard-validity term penalizes only severe box overlap, with $\mathrm{IoU}>0.5$. A plausible configuration such as a chair partly tucked under a table has much smaller box overlap ($\mathrm{IoU}\approx 0.1$ in our example) and is therefore not penalized by this term. Thus, our geometric reward discourages severe box interpenetration; it does not declare every overlap invalid, nor do we claim that it performs mesh-level collision checking. This limitation follows from the box-level representation used by SceneCast and by the layout-based baselines in our comparison. The methods differ in how they regularize overlap, but none turns a single object box into a part-aware collision model. A principled extension would represent each asset with a union of cuboids or a convex decomposition—for example, separate tabletop and legs, or chair seat, back, and legs. Such a representation could distinguish a chair under a table from true interpenetration, but would require part-level pseudo-labels, longer generation sequences, and a larger RL action space. These are not available in the current video-derived pipeline, and we will state this limitation explicitly.
AuthorsOpenReview ↗

Response to Reviewer Ktia Q3 and Q4.

Q3

Can the framework be extended to support more detailed scene or object specifications, such as furniture style, material, or a particular object arrangement? Which parts of the current representation and asset retrieval would need to change?

R3

The current representation deliberately predicts only category, position, yaw, and size, because these are the attributes that can be recovered reliably from the video-derived supervision. The framework can be extended, but the following extensions require additional supervision and retrieval support; they are not capabilities claimed by the current model.

Fine-grained semantic labels. Fine-grained categories, such as a three-seat sofa rather than a sofa, require an expanded vocabulary, matching Grounded-SAM prompts, retraining of the category head, and retrieval assets with the corresponding labels. The autoregressive layout representation itself need not change, but the pseudo-label and asset pipelines do.

Specified arrangements. A user specification can be represented as a partial layout together with spatial constraints. Supporting this setting would require a text/constraint-conditioning interface and training examples for that interface. The autoregressive decoder could then complete the unspecified objects, while the executable geometric checks could be extended to verify explicit constraints such as “sofa faces the TV” or “bed against the north wall.” This is a natural extension, but is not an input modality supported in the current experiments.

Style and material. These require new per-object attributes and a style-aware retrieval stage. The current video-derived layouts provide category and geometry but not reliable style or material labels; therefore, this extension would require VLM-generated attribute pseudo-labels or a supervised attribute set, together with a style-consistent asset library. It would partially weaken the current annotation-free assumption. We will clarify these requirements and the scope of the present model in the revision.

Q4(W1)

Where feasible, please add comparisons with relevant recent methods, such as SceneWeaver [1], OptiScene [2], PhyScene [3], and InstructScene [4]. If direct comparison is unsuitable, please clarify the differences in task settings and supported capabilities.

R4

We thank the reviewer for suggesting SceneWeaver, OptiScene, PhyScene, and InstructScene. We added an adapted PhyScene evaluation on living rooms under the same category mapping and layout evaluator as ours. PhyScene obtains 145.46 FID, 7.53 KID, 94.8 SCA, and 8.70 CKL. This result is an output-level transfer comparison: PhyScene was originally designed for complete-layout supervision and physical-interaction-aware synthesis, whereas our model is learned from partial video-derived layouts. We therefore do not interpret this result as a comparison of supervision efficiency or of physical-interaction capabilities.

We also investigated InstructScene. Its native task requires a text instruction, which is not available in our setting. Supplying a vLLM-derived text condition adds an external semantic extraction stage and produces a substantially shifted input distribution (FID 260.2 in our preliminary transfer run). We therefore do not present this number as a meaningful direct benchmark. We will instead discuss InstructScene as a text-conditioned method and state this task mismatch explicitly.

Table 3. Living/dining-room results under our common category mapping and evaluation protocol. PhyScene is adapted from its original physical-interaction-oriented setting.

MethodFID (↓)KID (↓)SCA (≈50%)CKL (↓)
PhyScene (adapted)145.467.5394.808.70
Ours128.575.6462.581.38
AuthorsOpenReview ↗

Cost-benefit, asset reliability, and reconstruction-based methods

W2

Unclear cost–benefit of the data pipeline and reliability of retrieved assets. The framework combines video generation and multiple foundation models to construct pseudo-labels, yet ultimately learns only object categories and bounding boxes. Its benefit over simpler data-construction alternatives is not established relative to the computational cost. Moreover, asset retrieval based on category and box proportions does not guarantee consistent geometry, contact, or collision behavior in the final scene. The relationship to reference-conditioned methods such as SAM3D [5] and SimRecon [6] also deserves discussion and comparison where applicable.

R

We thank the reviewer for raising the cost-benefit and final-asset questions. The goal of SceneCast is not to claim that its video-to-layout construction is cheaper than a pure forward-pass generator. Rather, it trades a one-time video-generation/reconstruction cost for avoiding task-specific human 3D-layout annotation and for obtaining a reusable layout policy. The submitted paper reports the model-training cost on one RTX 8000: approximately 36 hours for backbone pretraining and 15--20 minutes for SAC fine-tuning, including LLM inference and code execution. The hybrid schedule keeps the latter to only a few dozen LLM calls per run.

What the present ablation establishes is a data-quality benefit under a fixed downstream model: on bedrooms, no filtering and VisionReward+VideoPhy-only curation obtain FID 133.9 and 132.2, respectively, whereas ViTrim with the spatial-consistency signal and entropy weighting obtains 111.1. This supports the need for geometry-aware curation of video-derived supervision.

SAM3D and SimRecon are complementary reference-conditioned reconstruction systems: they aim to recover the geometry of a particular input image/video. SceneCast instead learns a distributional layout prior that can synthesize or complete layouts from an empty or partial state. Their native reconstruction metrics and reference-fidelity objectives are therefore not directly comparable to our distributional layout metrics. A controlled comparison would require substituting their reconstructed layouts into the same pseudo-label pipeline and retraining the same policy. We will discuss this possible use as future work.

Reviewer_6TXnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To solve the issue that existing methods rely heavily on manually annotated datasets, this paper present an annotation-free framework named SceneCast. It trains on large-scale and unlabeled videos . The LLM acts as a spatial common-sense evaluator , which provides reward signals through chain-of-thought reasoning and code-executable geometric checks to refine object configurations.

Critical strengths
  1. It is valuable for the community to solve the issue that many methods rely heavily on manually annotated datasets. And the paper presents approach somewhat make sense.

  2. The proposed methods outperforms state-of-the-art supervised methods in scene quality.

Critical weaknesses
  1. Although solve the mentioned issue is valuable, the paper presents key contribution about LLM rewards is limited insight for the community. This paper also pointed that there existing many LLM/VLM agent-based scene generation methods. Moreover, the overall pipeline seems like many existing methods directly integrated together, including Soft Actor-Critic (SAC) algorithm, VisionReward, VideoPhy and entropy-based weighting, etc. Therefore, the technical contributions are limited and the proposed method appears to be an incremental system without more new insights than existing relevant methods.

  2. Could the authors extend Table 2 (a) with an ablation of the hybrid strategy? For most SAC steps, the reward rtr_t is computed by the cheap geometric rule, while every 50 steps the LLM performs the full CoT-and-code evaluation and calibrates the rule thresholds for the subsequent interval, it remains unclear which component is primarily responsible for the performance improvement.

Questions for authors

Please refer to weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6TXn.

W1

Although solve the mentioned issue is valuable, the paper presents key contribution about LLM rewards is limited insight for the community. This paper also pointed that there existing many LLM/VLM agent-based scene generation methods. Moreover, the overall pipeline seems like many existing methods directly integrated together, including Soft Actor-Critic (SAC) algorithm, VisionReward, VideoPhy and entropy-based weighting, etc. Therefore, the technical contributions are limited and the proposed method appears to be an incremental system without more new insights than existing relevant methods.

R1

We agree that SAC, generic LLM prompting, VisionReward, VideoPhy, and entropy weighting are established techniques, and we do not claim them as contributions. Our claim is instead a new supervision construction for layout generation when complete 3D layouts and layout annotations are unavailable. SceneCast is not a simple stacking of existing components. The components are coupled to convert partial, noisy video-derived observations into supervision for global layout completion.

Existing layout generators learn from complete, annotated scenes. Generated videos, in contrast, provide only partial and progressively corrupted observations: they neither specify the missing objects in a room nor provide a reliable signal for global completion. Directly training on such observations makes the resulting policy inherit both the incompleteness and the geometric errors of the video. SceneCast addresses this specific supervision gap through a coupled design:

  1. ViTrim turns video quality assessment into geometry-aware selection of valid layout prefixes, rather than treating every reconstructed video frame as supervision.
  2. Code-grounded LLM feedback converts the otherwise missing global-completion and spatial-plausibility criteria into a reward for refining a compact layout policy.

Thus, the LLM is not used as a scene generator, a free-form planner, or a post-hoc evaluator. It is used to supply the missing supervisory signal that partial video observations cannot provide, while executable geometric checks prevent language-only judgments from drifting on scale and collision relations. This distinction is empirically material: adding the spatial-consistency signal in ViTrim changes FID from 132.2 to 111.1, and enabling code execution changes FID/PIoU from 141.3/0.52 to 111.1/0.25 under the same backbone and stopping policy.

We will revise the paper to frame SceneCast as an annotation-free supervision framework for global layout completion, and explicitly identify SAC and generic LLM reasoning as established components rather than contributions.

W2

Could the authors extend Table 2 (a) with an ablation of the hybrid strategy? For most SAC steps, the reward is computed by the cheap geometric rule, while every 50 steps the LLM performs the full CoT-and-code evaluation and calibrates the rule thresholds for the subsequent interval, it remains unclear which component is primarily responsible for the performance improvement.

R2

We thank the reviewer for the helpful request to clarify the role of the hybrid schedule and the choice of its evaluation interval. We therefore report an interval ablation under an otherwise fixed training protocol. We added an interval ablation while keeping the backbone, data, SAC updates, stopping policy, and random seed fixed. Here, KK denotes the number of SAC updates between two LLM evaluations; K=1K=1 corresponds to an LLM call at every update (the implementation flag interval=0).

Relative to the default K=50K=50 setting, dense evaluation (K=1K=1) reduces FID by only 0.6, while K=100K=100 increases FID by 2.3. PIoU shows the same ordering, with only a small change between K=1K=1 and K=50K=50. Thus, calling the LLM at every update produces nearly the same quality as the default schedule; the large improvement of the full method cannot be attributed to an unusually high LLM-call frequency. In contrast, K=100K=100 refreshes the feedback too infrequently and leads to stale guidance. For a fixed number of SAC updates, K=1K=1 also requires approximately 50 times more LLM evaluations. We therefore use K=50K=50 because it preserves essentially the same quality as dense evaluation at substantially lower cost.

Table 1. Ablation of the LLM refresh interval on the bedroom set. All non-interval settings are fixed. Lower is better for FID and PIoU. The dense setting corresponds to an LLM evaluation at every SAC step; the implementation flag for this setting is interval=0.

LLM interval KKLLM evaluation scheduleFID (↓)PIoU (↓)
1 (dense)every SAC step110.50.24
50 (default)every 50 SAC steps111.10.25
100 (sparse)every 100 SAC steps113.40.29
Reviewer_oitYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SceneCast, a framework for learning 3D indoor scene synthesis without annotated 3D data. The authors claim three main contributions. First, they introduce an annotation-free scene completion model: an autoregressive Transformer pretrained on partial layouts extracted from videos, then fine-tuned with reinforcement learning (SAC) using an LLM as a spatial common sense evaluator, which provides reward signals through chain-of-thought reasoning and code-executable geometric checks. Second, they propose ViTrim, a data curation method that generates training videos with a video generation model and keeps only the high quality initial clip, using multiple quality criteria fused by entropy based weighting.

Critical strengths

S1. The ablations are well structured, changing one component at a time, and show clear and consistent gains. In the reward ablation, FID improves progressively from No RL to rule-based, LLM without code, and LLM with code execution. The ViTrim ablation likewise separates no filtering, fixed-prefix trimming, the spatial IoU signal, and entropy based weighting.

S2. The paper is fairly transparent about practical details and limitations. It reports hyperparameters, training cost, the hybrid LLM-call scheme, and the added inference latency from the LLM in the loop. It also discusses potential bias in the reconstructed training data.

Critical weaknesses

W1. The contributions of the video derived training data and the LLM-based reward are not separated. All variants in Table 2(a) are worse than the three baselines, and weaker LLM backends (Table 5) are worse than ATISS and DiffuScene. An LLM also decides when to stop generation at inference time, which the baselines do not use.

W2. The source of the LLM reward’s gain is not explained. Appendix A.1 states that the LLM is called only every 50 steps and a rule-based reward is used otherwise, yet the FID gap between Rule-Geo and the full model is large.

W3. The main plausibility metric (PIoU) is closely related to the quantities used in the reward and the ViTrim filter, and the PIoU of real 3D-FRONT scenes is not reported. Some seed images come from the 3D-FRONT image collection, but their proportion and overlap with evaluation scenes are not stated.

Questions for authors

Q1 (W1). Could you apply the same LLM-based reward fine-tuning to an ATISS model trained on 3D-FRONT, and report results of your model when the LLM stopping signal is replaced by an object count sampled from the training distribution? This would help separate the contributions of the video derived data and the LLM.

Q2 (W2). Were all variants in Table 2(a) trained with the hybrid scheme in Appendix A.1 (LLM called every 50 steps)? Please describe how the rule-based reward used in the remaining steps differs from Rule-Geo, and how the LLM calibrates the rule thresholds.

Q3 (W3). Could you report the PIoU of real 3D-FRONT scenes? Also, what proportion of the seed images comes from the 3D-FRONT image collection, and do these overlap with the scenes used for evaluation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer oitY Q1.

W1

The contributions of the video derived training data and the LLM-based reward are not separated. All variants in Table 2(a) are worse than the three baselines, and weaker LLM backends (Table 5) are worse than ATISS and DiffuScene. An LLM also decides when to stop generation at inference time, which the baselines do not use.

Q1 (W1)

Could you apply the same LLM-based reward fine-tuning to an ATISS model trained on 3D-FRONT, and report results of your model when the LLM stopping signal is replaced by an object count sampled from the training distribution? This would help separate the contributions of the video derived data and the LLM.

R1

We thank the reviewer for the insightful suggestion to make the roles of the video-derived prior, the LLM reward, and the stopping policy more explicit in Table 2(a). Below, we clarify the protocol and provide additional controls that isolate these factors. One clarification is that the full code-execution row is not below the bedroom baselines on FID (111.1 versus 123.06 for ATISS and 124.19 for DiffuScene); however, the intermediate rows do not use exactly the same termination protocol as the supervised baselines. We therefore treat Table 2(a) as a component ladder, rather than as a causal comparison with the baselines.

For transparency, we will report the protocol of every row. The No-RL, Rule-Geo, and Rule+Cat variants use a fixed generation horizon Nmax⁡N_{\max} because they do not use the LLM early-stopping signal. The LLM variants use the LLM-based early-stopping signal, while ATISS uses a learned EOS and DiffuScene uses a sampled scene length. Thus, the first three rows should not be interpreted as fully matched comparisons with complete-scene supervised baselines: their results reflect both the reward configuration and a different termination protocol.

We clarify the relation between Rule-Geo and the hybrid schedule. Rule-Geo uses a fixed geometric threshold kk and does not call the LLM. The hybrid schedule retains the same fixed kk and uses the Rule-Geo reward between LLM evaluations. The LLM updates separate internal reward-calibration hyperparameters, not kk. Thus, kk is held fixed across the compared schedules, while the interval KK controls how often the LLM refreshes those internal parameters. In the revision we will name these parameters and state their update rule explicitly; no claim is made that the LLM calibrates kk.

We performed two controls. First, we replaced the LLM stopping signal with a count-based stopping rule derived from the empirical training-layout count distribution. Relative to the full model, this replacement increases FID by approximately 12.3 and PIoU by 0.21; both metrics are lower-is-better. The reason is that a count-based rule only forces the model to generate a prescribed number of objects; it cannot determine which objects are redundant or whether the current scene is complete. Thus, this control shows that the scene-dependent LLM stopping signal contributes to performance, rather than providing an equivalent stopping mechanism.

Second, we fine-tuned an ATISS model with the same continuous-attribute LLM-RL reward. ATISS improves from FID 123.06 / PIoU 0.48 to 120.58 / 0.36, showing that the reward can improve a backbone trained on complete 3D-FRONT scenes. SceneCast nevertheless achieves FID 111.06 / PIoU 0.25 under the same evaluation, which is consistent with an additional benefit from the video-derived layout prior. We will present this result as evidence that both the reward and the video-derived prior contribute.

Table 1. Row-wise training and termination protocols for the reward ablation.

ConfigurationRewardLLM codeStopping policyLength control
No RL----fixed Nmax⁡N_{\max}fixed
Rule-Geogeometric--fixed Nmax⁡N_{\max}fixed
Rule+Catgeometric + category--fixed Nmax⁡N_{\max}fixed
LLM w/o codeLLMnoLLM early stopadaptive
LLM w/ codeLLMyesLLM early stopadaptive
ATISSsupervised--learned EOSadaptive
DiffuScenesupervised--sampled lengthadaptive
VIPScenesupervised--not applicable (reconstructs a given video)input-video coverage

Table 2. Ablation of the inference-time stopping policy on the bedroom set.

Stopping policyLLM at inferenceFID (↓)PIoU (↓)
LLM-based scene-completion stoppingyes111.060.25
Count-based stopping from empirical layout distributionno123.360.46

Table 3. Experiments on ATISS fine-tuning in the bedroom set.

BackboneFID (↓)KID (↓)SCA (%)CKL (↓)PIoU (↓)
ATISS123.060.7057.201.850.48
ATISS (LLM RL)120.580.6556.721.790.36
AuthorsOpenReview ↗

Response to Reviewer oitY Q2.

W2

The source of the LLM reward’s gain is not explained. Appendix A.1 states that the LLM is called only every 50 steps and a rule-based reward is used otherwise, yet the FID gap between Rule-Geo and the full model is large.

Q2 (W2)

Were all variants in Table 2(a) trained with the hybrid scheme in Appendix A.1 (LLM called every 50 steps)? Please describe how the rule-based reward used in the remaining steps differs from Rule-Geo, and how the LLM calibrates the rule thresholds.

R2

We thank the reviewer for the helpful request to clarify the role of the hybrid schedule and the choice of its evaluation interval. We therefore report an interval ablation under an otherwise fixed training protocol. We added an interval ablation while keeping the backbone, data, SAC updates, stopping policy, and random seed fixed. Here, KK denotes the number of SAC updates between two LLM evaluations; K=1K=1 corresponds to an LLM call at every update (the implementation flag interval=0).

Relative to the default K=50K=50 setting, dense evaluation (K=1K=1) reduces FID by only 0.6, while K=100K=100 increases FID by 2.3. PIoU shows the same ordering, with only a small change between K=1K=1 and `K=50K=50. Thus, calling the LLM at every update produces nearly the same quality as the default schedule; the large improvement of the full method cannot be attributed to an unusually high LLM-call frequency. In contrast, K=100K=100 refreshes the feedback too infrequently and leads to stale guidance. For a fixed number of SAC updates, K=1K=1 also requires approximately 50 times more LLM evaluations. We therefore use K=50K=50 because it preserves essentially the same quality as dense evaluation at substantially lower cost.

Table 4. Ablation of the LLM refresh interval on the bedroom set. All non-interval settings are fixed. Lower is better for FID and PIoU. The dense setting corresponds to an LLM evaluation at every SAC step; the implementation flag for this setting is interval=0.

LLM interval KKLLM evaluation scheduleFID (↓)PIoU (↓)
1 (dense)every SAC step110.50.24
50 (default)every 50 SAC steps111.10.25
100 (sparse)every 100 SAC steps113.40.29
AuthorsOpenReview ↗

Response to Reviewer oitY Q3.

W3

The main plausibility metric (PIoU) is closely related to the quantities used in the reward and the ViTrim filter, and the PIoU of real 3D-FRONT scenes is not reported. Some seed images come from the 3D-FRONT image collection, but their proportion and overlap with evaluation scenes are not stated.

Q3 (W3)

Could you report the PIoU of real 3D-FRONT scenes? Also, what proportion of the seed images comes from the 3D-FRONT image collection, and do these overlap with the scenes used for evaluation?

R3

Approximately 11% of our seed images come from a subset of the released 3D-FRONT image collection, and the remaining 89% are Internet images. We do not render additional 3D-FRONT views ourselves. We agree that the seed-source protocol should be explicit. Among the 45,000 seed images, 40,000 (89%) are Internet images and 5,000 (11%) come from the 3D-FRONT image collection. The evaluation images are strictly separated from the training images.

We also report PIoU = 0.58 for real 3D-FRONT scenes, computed with the same category set, box convention, and preprocessing as for generated scenes. PIoU is an evaluation-only metric: it is not used as an RL reward and does not determine ViTrim filtering. Although both ViTrim and PIoU involve 3D boxes, ViTrim's spatial-IoU signal measures consistency across video frames, whereas PIoU measures pairwise box overlap in the final generated scene. We interpret PIoU as a box-overlap proxy rather than a complete physical-validity metric, since valid functional relations such as a chair partly under a table can produce nonzero overlap. We separate this limitation from the conclusions supported by FID, KID, SCA, and CKL.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Consolidated response overview.

Dear Reviewers and Area Chair,

Thank you for the careful and constructive reviews. We appreciate the reviewers' recognition of the practical importance of reducing dependence on human 3D-layout annotations (Reviewer 6TXn), the value of constructing layouts from generated videos with quality-aware curation (Reviewer Ktia), and the structured ablations and practical transparency in the submitted study (Reviewer oitY). We are grateful that the reviews also identified the precise places where the submission should be clearer about causal attribution, evaluation scope, and the boundary of the representation.

Paper overview and contribution

SceneCast addresses the following supervision bottleneck. Scalable Internet images and generated videos can provide many indoor observations, but the reconstructed layouts are partial and noisy, whereas conventional layout generators learn from complete human-annotated 3D scenes. SceneCast constructs supervision for a reusable global layout policy from these otherwise insufficient observations: ViTrim selects geometrically consistent video prefixes, and code-grounded LLM feedback supplies the missing global completion and spatial-plausibility signal during policy refinement.

New analyses and clarifications

In response to the reviews, we add: (i) row-wise reward and termination protocols, a count-based stopping control, an ATISS+LLM-RL transfer control, and an LLM-refresh-interval ablation; (ii) seed-source accounting, a source-scene overlap audit, and a real-scene PIoU reference; (iii) the asset canonicalization rule, the box-level scope of the overlap metric, and the requirements for extending the representation to style, material, and explicit arrangements; and (iv) an adapted PhyScene comparison under a common evaluator, with precise task/reproducibility boundaries for InstructScene, SceneWeaver, and OptiScene.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.