acceptodds
Under review as a conference paper at ICLR 2027

From Plausible Scenes to Executable Worlds: Script-based 3D Indoor Scene Generation

Abstract

Script-based 3D indoor scene generation aims to construct 3D environments that faithfully realize the behaviors described in textual narratives. Existing scene generation methods, however, primarily focus on visual plausibility and semantic consistency, leaving an important question largely unexplored: can the generated scene actually support the prescribed behaviors? We argue that a behaviorally meaningful scene should constitute an executable world, where scripted actions are feasible not only individually but also as a coherent sequence. Based on this perspective, we identify three levels of executable requirements: spatial executability, which requires characters to reach the locations needed for consecutive actions; functional executability, which requires scene objects to geometrically and functionally support their intended interactions; and sequential executability, which requires successive actions to remain coherently executable over time. These requirements motivate us to move beyond one-shot scene generation toward an explicit generation-diagnosis-repair paradigm. To study it, we first construct FilmScene, a character behavior-centric dataset containing script-3D scene pairs, with scene-grounded interaction objects and action chains, together with real-world out-of-distribution test sets. Building upon this benchmark, we propose Script-to-Scene (S2S), a closed-loop framework that first generates a plausible scene, explicitly diagnoses violations of executable constraints, and then performs targeted scene repair while preserving valid scene structure. By combining spatial reachability analysis, functional compatibility assessment, behavioral sequence verification, S2S progressively transforms plausible scenes into executable worlds via iterative local scene editing. Extensive experiments on FilmScene and OOD test sets demonstrate that S2S improves both overall scene quality and the ability of generated environments to support script-driven character behaviors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D indoor scene generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2mefOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies script-based 3D indoor scene generation and introduces FilmScene, containing approximately 11k paired scripts and scenes. It proposes Script-to-Scene (S2S), which generates an initial scene, diagnoses violations of spatial, functional, sequential, and physical requirements, and iteratively repairs the implicated scene elements.

Critical strengths
  1. FilmScene provides a useful resource by pairing narrative scripts with indoor scenes and interaction annotations. Its emphasis on action-related objects and behavior chains supports research beyond conventional scene-description alignment.
  2. S2S combines complementary diagnostics and targeted editing into a coherent pipeline. The integration of reachability, affordance, visual, and geometric checks is practically motivated and represents substantial engineering effort.
  3. The evaluation covers multiple baselines, OOD scenarios, component ablations, and human judgments. The reported improvements in functional support and script compliance provide evidence that the integrated system helps align scenes with prescribed activities.
Critical weaknesses
  1. The “Executable Worlds” framing is broader than the evidence presented, which mainly establishes geometric and semantic support for scripted interactions. The paper should clarify the scope of executability and distinguish such support from verified completion of state-dependent action sequences.
  2. The pipeline largely combines established generation, critique, and scene-editing mechanisms. Its main methodological distinction appears to be the script-specific diagnostics, whose benefits over existing agentic refinement approaches need clearer demonstration.
  3. The rationale for choosing Qwen3-8B is unclear. A stronger off-the-shelf generator might reduce the need for task-specific fine-tuning and iterative repair, so the quality–cost trade-off motivating this design should be explained.
  4. SAGE [1] and SceneSmith [2] are important missing comparisons given their overlap in agentic scene generation, refinement, and physical validation. SceneReVis [3] and Edit-As-Act [4] are also relevant to iterative diagnosis and structure-preserving editing and should be discussed. Comparisons should clarify differences in models and assets and complement Table 8’s runtimes with model-call counts, token usage, and a clearly defined end-to-end timing scope.
  5. The current metrics do not establish complete script executability, particularly for state-dependent interactions such as unlocking a drawer before retrieving an object. The SC evaluator and scoring protocol remain unspecified; independent validation of complete action chains and the proportion of scenes satisfying all constraints would better support the claim.
  6. The sequential ablation also removes general distance, orientation, and occlusion checks, so it does not isolate reasoning about action order. The “minimal repair” claim needs direct measurements of edit magnitude and preservation of valid structure, together with clarification of whether diagnostic feedback and computational budgets are matched in the comparison against full regeneration.
  7. Please specify the data splits and the sizes of the two OOD subsets. The manuscript also needs proofreading to correct typos such as “assurrance” → “assurance” (p. 5, line 243) and “executablility” → “executability” (p. 7, line 358).

My current recommendation is weak reject. I appreciate the dataset and engineering contribution, but the evidence for complete script executability and the added value over closely related systems is not yet convincing. I am open to raising my score if the author response provides clarification or evidence that convincingly addresses these central concerns.

[1] Xia, Hongchi, et al. SAGE: Scalable Agentic 3D Scene Generation for Embodied AI. CVPR 2026.

[2] Pfaff, Nicholas, et al. SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes. ICML 2026.

[3] Zhao, Yang, et al. SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-turn RL. arXiv:2602.09432, 2026.

[4] Noh, Seongrae, et al. Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing. CVPR 2026.

Questions for authors

Please refer to the weaknesses above.

Reviewer_57ZSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors formulate Script-based 3D Indoor Scene Generation from the perspective of behavioral executability, identifying three script-induced requirements: spatial (reachability between consecutive interaction locations), functional (objects geometrically support their actions), and sequential (successive actions remain jointly feasible). They construct FilmScene, an 11k script–scene dataset with grounded action chains plus two OOD test sets, and propose S2S, a generate-diagnose-repair loop combining A* reachability checks, a trained affordance classifier, and a multi-view VLM diagnoser with targeted scene repair.

Critical strengths
  1. The distinction between a visually plausible scene and an executable one is a genuine gap in text-to-3D scene generation, and the three-level decomposition is intuitive and well illustrated by the examples in Sec. 1 and Figure 3.

  2. The diagnosis signals are appropriately heterogeneous rather than all delegated to an LLM: A* on an occupancy grid for reachability, a learned affordance classifier for functional fit, and a VLM over 16 rendered views for sequence-level obstruction. This is a sensible division of labor.

Critical weaknesses
  1. The paper proposes three executability constraints, a closed-loop diagnose-repair cycle, and an affordance classifier, yet the only quantitative result is the single system comparison in Table 1. It is therefore impossible to tell how much each of Cspa, Cfun, Cseq contributes, whether the loop improves monotonically over iterations, or whether the learned affordance classifier outperforms a simple geometric heuristic. For a paper whose contribution is the decomposition itself, this is the single most important missing experiment.

  2. Table 1 appears to be incomplete. Reading across, the rows for StageDesigner, SceneWeaver, and S2S (fine-tuned) each contain one fewer entry than LayoutGPT, HoloDeck, LayoutVLM, AnyHome, and S2S (zero-shot), which suggests a missing value in the FilmScene Executability block (most likely SC). If so, the claim in Sec. 5.2 that "the fine-tuned version obtains the best Functional Support and Script Compliance" is not supported by a reported number for SC. Please verify the table.

  3. Under the most natural column alignment, the fine-tuned variant is worse than the zero-shot variant on Functional Support (84.75 vs. 89.72), and also below AnyHome (88.04). This directly contradicts the narrative that fine-tuning provides "a stronger starting point for refinement." Either the alignment is wrong (see W2) or the fine-tuning result needs to be explained rather than asserted.

Questions for authors

No

Reviewer_edZeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Script-based 3D Indoor Scene Generation (SCR-SG), arguing that scenes generated from narrative scripts should be "executable worlds" rather than merely plausible layouts. Executability is decomposed into three requirements. Spatial: consecutive interaction locations are mutually reachable. Functional: object geometry supports the intended action. Sequential: the ordered action chain remains coherent.

Critical strengths
  1. A meaningful reframing of the objective. The paper moves the target from "the scene looks plausible" to "the scene supports the scripted behavior sequence". This direction is relevant to film, game, and embodied content creation and has received little attention in recent LLM-based layout work. The spatial / functional / sequential decomposition gives a useful vocabulary for describing failures (Fig. 3 illustrates them well). This framing is the main reason I see potential in the work.

  2. Resource potential and a practical interface. FilmScene ties each script to an action chain grounded in specific object instances, interaction points, and A* paths (Fig. 9–10 even give per-step walk distances and stand-off offsets). That is richer supervision than existing text–scene pairs, and the OOD sets point toward realistic use cases. If FilmScene is released with clear splits and licenses, it could support follow-up work independently of S2S. The structured defect log (Fig. 13) is interpretable, and the pipeline is inexpensive relative to agentic baselines (~24 s vs. ~580 s for SceneWeaver, Table 8).

Critical weaknesses
  1. Executability is never independently tested; the evaluation largely mirrors the diagnoser. Nothing is executed: no agent, motion synthesis, or physics simulation is run. Instead, the metrics reuse the method's own checks:
  • RC is the same A* test as the spatial diagnoser (Eq. 2).
  • CR/UOR are the same overlap/floating rules as the physical check.
  • FS is a VLM judgement, like the sequential diagnoser.
  • SC's evaluator is unspecified.

Baselines get none of this feedback. Table 3 makes the effect visible: removing any single check drops its matching metric below baselines that never had it, e.g., w/o Functional gives FS 82.47 vs. StageDesigner 89.18, and w/o Sequential gives SC 78.41 vs. LayoutVLM 82.71. The generator-only result is never reported. The fine-tuned generator learns FilmScene's closed vocabulary, which favors FID/OC. No test-set sizes, variances, or CIs are given, although many margins are 1–2 points. As it stands, the central claim cannot be separated from "fine-tuning in-domain and optimizing against the metrics".

  1. The executability mechanisms are much weaker than claimed, and the tables contradict each other.
  • Sequential diagnosis is a VLM prompt (Table 5) that checks cabinet/chair orientation and a >3 m distance threshold. It does not reason about object states, step order, or agent orientation, and Fig. 13 shows a false positive at 2.6 m.
  • The affordance classifier sees only box size and world-frame yaw, and the "negatives" in Eq. 7 receive labels near 1.
  • Minimal repair (Eq. 3) is never solved or measured, and the claimed add operation is missing from the repair prompt (Table 6).
  • The headline numbers are internally inconsistent. Table 8 reports 91.08/84.75 as S2S without iterative refinement, while Table 2 reports the same numbers as Full with it.
  • Table 1 markings err in the method's favor: OOD FS 83.18 is underlined, but StageDesigner scores 83.42.

Novelty over generate–verify–refine pipelines (SceneWeaver, LayoutVLM) is also under-argued. In addition, closely related prior work is neither cited nor discussed: activity-centric scene synthesis [1], scene evolution driven by sampled action sequences [2], and scene synthesis with reachability/interactivity guidance [3].

References (not cited in the submission): [1] Fisher, Matthew, et al. "Activity-centric scene synthesis for functional 3D scene modeling." ACM Transactions on Graphics (TOG) 34.6 (2015): 1-13. [2] Rui Ma, Honghua Li, Changqing Zou, Zicheng Liao, Xin Tong, and Hao Zhang. 2016. Action-driven 3D indoor scene evolution. ACM Trans. Graph. 35, 6, Article 173 (November 2016), 13 pages. https://doi.org/10.1145/2980179.2980223 [3] Yang, Yandan, et al. "Physcene: Physically interactable 3d scene synthesis for embodied ai." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

Questions for authors
  1. Which configuration produced Table 1? Please reconcile Table 2 (w/o Iterative = 90.12/83.58, Full = 91.08/84.75) with Table 8 (single-pass = 91.08/84.75, iterative = 91.64/85.42).
  2. Please report generator-only results (FT and zero-shot), and baselines with your diagnose–repair loop applied. Without these, the gains cannot be attributed to diagnosis–repair rather than to fine-tuning and checks aligned with the metrics.
  3. How is SC computed (model, prompt, inputs)? At test time, does the diagnoser use ground-truth FilmScene action chains or chains extracted from the script? If ground truth, S2S has privileged information that the baselines lack.
Reviewer_5edMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

this paper proposed a benchmark to test executability of generated scenes i.e. if a generated scene can support a certain behavior of actors in the scene. that requires a prescribed location is reachable, action object pairs should support the action on the object, a ordered chain of behavior is doable in the scene. then proposed a method to improve executability of the scene by repeatedly testing the executability and refine to fix the failures.

Critical strengths
  1. Executability is an interesting and novel evaluation angle for scene generation. The paper contributes a behavior-grounded dataset and a combination of spatial, functional, and action-chain constraints to support scene generation executability studies and future research in related directions.
  2. Experimental results demonstrate that the proposed framework improved the executability of generated scenes.
  3. The method and dataset could potentially be useful for scene generation in virtual production and robotics simulation
Critical weaknesses
  1. The functional requirement check is not sufficient. It is checked by a classifier, and the negative examples are generated by perturbing object dimensions, with labels decreasing according to the distance from the original dimensions. It does not consider other aspects of functionality, such as geometry, articulation, graspability, and actor-specific reach
  2. The sequential executability is largely based on the distance of consecutive objects, but nearby objects do not guarantee a valid behavior sequence. Changing object states could make a previously accessible object no longer accessible. For example, open the dishwasher -> walk to the sink could be blocked by the open dishwasher door.
  3. Actual execution remains untested. The three requirements are surrogate criteria for executability. The paper does not report simulator rollouts or whole-script execution success rates.
  4. Some implementation details are missing: train/validation/test counts, out-of-distribution test set size
Questions for authors
  1. what is the size of train/val/test split and out-of-distribution test set size?
  2. Some reference format is incorrect. Eg. line 157 Fu et al. (2021a) and line 176 Achiam et al. (2023)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.