- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SceneWise, a framework for indoor 3D scene generation that introduces a learned decision-consequence world model and a historical attribution model. The world model predicts the downstream consequences of candidate scene-editing decisions, including changes in placement feasibility, constraint margins, potential failures, and affected objects. These predictions are used to rank candidate actions before execution-based verification. When scene generation fails, the attribution model ranks historical decisions according to whether tested revisions have been effective, and the system performs local regeneration to repair the scene while minimizing unnecessary changes. The paper also introduces SceneWiseDB, a dataset containing 100,000 decision-consequence and matched-intervention records. Experiments on 94 room-level prompts, 100 tasks with coupled spatial and functional requirements, and additional repair and generalization settings suggest that SceneWise improves complete requirement satisfaction and reduces editing operations compared with several existing scene-generation methods.
Critical strengths
- A learned world model for predicting downstream decision consequences. SceneWise introduces a decision-consequence world model that predicts structured outcomes of candidate scene-editing decisions, including placement-feasibility changes, constraint-margin changes, and potential failures. This goes beyond checking whether an action is immediately valid and provides a principled way to assess its potential downstream effects.
- A learned historical attribution model for failure recovery. The paper introduces a historical attribution model that ranks previous decisions according to whether tested revisions can effectively resolve a failure.
Critical weaknesses
Insufficiently specified training data generation pipeline. SceneWiseDB is central to training both the decision-consequence world model and the historical attribution model. However, although Appendix B describes how alternative decisions are executed in copies of partial scenes and how execution outcomes are converted into supervision, the upstream source of the offline generation trajectories remains unclear. Specifically, the paper does not clearly identify which agent or scene-generation system produces these trajectories, how the initial source scenes are generated and sampled, or how candidate decisions are proposed during the data collection process. This is an important limitation because the distribution and diversity of the source trajectories directly affect what the learned models can capture. Without understanding the upstream generation process, it is difficult to assess the coverage of SceneWiseDB, determine whether the learned models are biased toward a particular generation policy, or reproduce the proposed dataset and training pipeline. I encourage the authors to provide a more complete description of the source-scene generation system, trajectory collection procedure, and candidate-sampling strategy.
Questions for authors
- Construction of the 70,000 decision-consequence records. Could the authors clarify how the 70,000 decision-consequence records are generated? In particular, which agent or scene-generation system produces the initial offline trajectories, and how are source scenes, partial-scene checkpoints, and alternative candidate decisions sampled? A more detailed description of the complete data-generation pipeline would help assess the dataset's coverage and reproducibility.
- Clarification of the mathematical notation. Could the authors simplify or clarify the notation in Sections 3.2–3.4, particularly the candidate-selection and execution-verification equations? For example, explicitly distinguishing predicted consequences and risks from execution-measured consequences and risks, and providing a step-by-step explanation of the roles of (K), (B), and (H), would make the proposed framework easier to understand and reproduce.
- Comparison with GPT-6 Astra. Could the authors compare SceneWise with a strong general-purpose LLM agent, such as GPT-6 Astra, using the same scene-editing tools, task inputs, and execution-verification budget? Such a comparison would help clarify whether the proposed learned consequence model and historical attribution mechanism provide benefits beyond those of a capable LLM agent with iterative feedback and search.
- Potential data leakage between training and evaluation. Although SceneWiseDB is partitioned by source-scene groups to prevent overlap across training, validation, and test splits, the separation between the training data and the downstream evaluation tasks remains unclear. Could the authors clarify whether the 94 SceneEval prompts and 100 coupled-requirement tasks are entirely independent of the source scenes used to construct SceneWiseDB? In particular, were any checks performed to identify overlapping or highly similar scene layouts, asset configurations, or task requirements? Further clarification would help rule out potential data leakage and strengthen confidence in the reported results.
4 comments
Authors' Response to Reviewer SmVn (continue)
Q3. Comparison with GPT-6 Astra
We compared SceneWise with a GPT-6 Astra agent using the same scene-editing tools, task inputs, and execution-verification budget of 60 calls per attempt. Both tables below report means with 95% confidence intervals.
Generation on 100 tasks with coupled spatial and functional requirements
Table R1. Comparison with GPT-6 Astra on 100 coupled-requirement tasks.
| Method | SR ↑ (%) | CSR ↑ (%) | EditOps ↓ | Edits ↓ |
|---|---|---|---|---|
| GPT-6 Astra agent | 79.0 ± 7.3 | 92.6 ± 1.5 | 7.6 ± 1.1 | 4.2 ± 0.6 |
| SceneWise | 84.0 ± 6.4 | 94.1 ± 1.8 | 4.8 ± 0.7 | 3.2 ± 0.5 |
SceneWise increased complete requirement satisfaction from 79.0% to 84.0% while reducing edit operations from 7.6 to 4.8 per attempt. This suggests that consequence-guided generation helps SceneWise satisfy coupled requirements with fewer editing operations.
Repair on 100 shared failed-scene checkpoints
Table R2. Comparison with GPT-6 Astra on 100 shared repair checkpoints.
| Method | SR ↑ (%) | CSR ↑ (%) | SR (E ≤ 5) ↑ (%) | Edits ↓ | Calls ↓ | Time ↓ (min/attempt) |
|---|---|---|---|---|---|---|
| GPT-6 Astra agent | 82.7 ± 5.8 | 94.7 ± 1.7 | 69.0 ± 7.9 | 4.2 ± 0.6 | 21.4 ± 1.6 | 11.2 ± 0.8 |
| SceneWise | 86.0 ± 6.1 | 96.2 ± 1.5 | 78.0 ± 7.1 | 2.6 ± 0.4 | 16.0 ± 1.4 | 8.4 ± 0.7 |
SceneWise successfully repaired the scene while changing at most five existing objects in 78.0% of attempts, compared with 69.0% for GPT-6 Astra. It also modified fewer checkpoint objects on average (2.6 vs. 4.2) and used fewer verification calls per attempt (16.0 vs. 21.4). These results suggest that historical attribution and local repair help SceneWise target effective revisions while preserving more of the existing scene.
Q4. Potential data leakage between training and evaluation
Neither evaluation set was used to construct SceneWiseDB. We found no exact or near-duplicate matches for any of the 94 SceneEval prompts or 100 coupled-requirement tasks (see table below). We compared each task with both template-generated and imported SceneWiseDB source prompts based on required object categories, quantities, spatial relations, and functional constraints. We also compared the layouts and asset configurations of the generated evaluation scenes with those of the source scenes.
Table R3. Overlap between evaluation tasks and SceneWiseDB source scenes.
| Aspect checked | SceneEval prompts | Coupled-requirement tasks |
|---|---|---|
| Task specifications | 0/94 | 0/100 |
| Scene layouts | 0/94 | 0/100 |
| Asset configurations | 0/94 | 0/100 |
response
Thank you for the detailed rebuttal. The authors’ clarifications address my concerns regarding data construction and potential data leakage between training and evaluation. The additional comparison with GPT-6 Astra further supports the effectiveness of the proposed method. I therefore support acceptance of the paper.
Thank you for your feedback
We are glad that your concerns have been addressed. Thank you for your positive feedback. We will incorporate your suggestions into the revised manuscript.
Authors' Response to Reviewer SmVn
Thank you for recognizing the methodological strengths of our work. We appreciate your constructive feedback and respond to each of your concerns below.
Q1. Construction of the 70,000 decision-consequence records
We sampled checkpoints from selected source scenes and executed alternative decisions to measure their immediate and delayed consequences, yielding 70,000 records after deduplication. We describe this process step by step below.
Source-scene generation and selection. Our scene-generation pipeline built on SceneSmith [1] generated new scenes and offline trajectories from structured prompts. We selected completed scenes restorable with all required assets, cycling through room types during expansion.
Trajectories and partial-scene checkpoints. From the offline generation trajectories, we selected checkpoints before successful tool calls. We also moved up to four independently movable furniture objects per restored scene, saving up to three checkpoints before moves with a subsequent action.
Alternative candidate decisions. At each checkpoint for a move, we constructed 25 candidates. We kept the recorded move as a control and sampled 24 alternative positions from a stratified grid covering the room interior and areas near its boundaries, keeping the object and its orientation fixed.
Execution and consequence measurement. We executed each candidate in an independent scene copy under the same continuation policy. We measured its immediate and delayed consequences over a horizon of at most three decisions, including the candidate itself.
Record construction. One decision–consequence record consists of a partial-scene context, a candidate decision, and the consequences measured from its executed continuation. Different candidates evaluated from the same partial scene count as separate records.
[1] Pfaff, N., Cohn, T., Zakharov, S., Cory, R., & Tedrake, R. (2026, February). Scenesmith: Agentic generation of simulation-ready indoor scenes. In Forty-third International Conference on Machine Learning.
Q2. Clarification of the mathematical notation in Sections 3.2–3.4
1. Predicted versus execution-measured consequences and risks. Eq. (1) defines the predicted and measured consequences of candidate decision d at step t:
Ĉₜᴴ(d) = Wᵩ(Sₜ, Qₜ, Gₜ, P, d), Cₜᴴ(d) = (ΔF, ΔR, V, A).The hat distinguishes predictions from execution-derived measurements. The four components are Placement Feasibility Changes (ΔF), Constraint Margin Changes (ΔR), Potential Failures (V), and Affected Objects (A).
Eq. (3) defines the risk function applied to either set of consequences:
R(C) = α R_F(C) + β R_R(C) + γ R_V(C).The three terms penalize decreases in placement feasibility, decreases in comparable constraint margins, and failure events, respectively. Thus,
R(Ĉₜᴴ(d))is the predicted risk, andR(Cₜᴴ(d))is the execution-measured risk.2. Roles of K, B, and H in candidate selection and execution verification. Eq. (4) contains two expressions. The first shortlists candidates by predicted risk, and the second selects the next decision by execution-measured risk.
K — propose candidates. The Designer proposes K candidate decisions.
Uₜᵛᵃˡⁱᵈcontains those that pass the immediate collision, boundary, and hard-constraint checks.B — shortlist candidates. The first expression in Eq. (4) is
𝒦ₜ = TopB_{d ∈ Uₜᵛᵃˡⁱᵈ} [−R(Ĉₜᴴ(d))].TopBreturns up to B candidates with the highest scores inside the brackets. Because the score is negative predicted risk, the shortlist𝒦ₜcontains the candidates with the lowest predicted risks.H — verify each shortlisted branch. We execute each candidate and follow the continuation policy π for up to H − 1 further decisions. Thus, H includes the candidate itself.
𝒦ₜᵒᵏcontains candidates whose branches complete the task or reach the horizon without failure. The second expression in Eq. (4) isd*ₜ₊₁ = argmin_{d ∈ 𝒦ₜᵒᵏ} R(Cₜᴴ(d)).argminreturns the candidate with the lowest execution-measured risk. We commit only that decision,d*ₜ₊₁. If all shortlisted branches fail, we invoke scene repair.In Section 3.4, the world model likewise predicts the consequences of proposed replacements, and matched interventions measure their effects.