- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents an approach MUSE: an agentic 3D scene authoring tool, with arguments for incremental local support to editing facilitated by a structured memory system of three different types of storage: tracking scene diagnostics/requirement status (working memory), persistent 3D scene memory and a third Skill memory mainly consisting of recurring decompositions. The work follows the popular line of incremental 3D scene editing/authoring task with a division of subtasks to the LLM/VLM. An important argument the paper proposes is to use local or compact support on the scene domain for every edits, i.e., the editing assumes non-target scene elements are protected where protection is decided based on certain rules, e.g., reference. Finally the paper proposes a scene editing dataset consisting of 145 scene construction examples, with a number of instructions per scene totalling 1584.
Critical strengths
-
In cases where scene authoring is done by incremental editing by a user, use of an "external" organized memory as proposed is justified.
-
The results and video show clearly the problem and its implementation.
Critical weaknesses
-
Contribution in the context of the results: The main contribution of the work is the addition of memory and a local editing of scene based on a rule set. While the first can be somewhat justified, the somewhat rigid editing of scenes is not very interesting. When looking at the supplementary video, the paper's proposal boils down to local "blender add/remove and move/remove" interpreter. It does not show at all the value of the complex mechanism proposed with several process and memory submodules. E.g., what happens if the author asks "Add a table infront of the bed, but some objects and walking path already exist in front of the bed". A video example could have included such cases.
-
The method itself is not very inspiring. Local support should be need-based and not as strict as advocated by the paper. I did not gain any new knowledge after reading through the method. For example, when adding an object, an intelligent agent should not protect an object just by reference. Another problem is that the scene authoring vocabulary looks extremely simple, e.g., "left/right, move/remove" etc. One could build a custom compiler for these simple words once extracted.
-
The memory structure as a main contribution is not satisfactory given that the structured memory is in fact a simple reorganization of the prompt.
Questions for authors
- The paper does not describe expertise used in scene-authoring dataset. Who created the dataset? Does the data satisfy the most important criteria of scene authoring: i.e., is it architecturally meaningful (interior architecture)? How was utility, aesthetics, suitability decided for "the instruction to placement" annotation?
6 comments
Response to Reviewer yh9e (2/2): Structured Memory and AuthorBench Construction
Q3. Is structured memory simply a reorganization of the prompt?
A3. At the planner interface, explicit state is indeed serialized into the prompt; the maintained state also drives execution guards, re-verification and stopping.
The history-only control in Table 4 (App. C.3.2) keeps all other components and budgets, but its planner receives the complete action/feedback history instead of explicit planner state and selects the next requirement itself. MUSE still raises construction Macro from 88.2 to 93.5 and editing Macro from 73.8 to 81.2, so how the state is organized matters for planning.
Outside the prompt, code updates Working Memory from combined verification results: deterministic checks take precedence for geometric and counting constraints, and model feedback covers the remaining semantic and visual aspects. Its records of requirement status and involved objects drive protection guards, link changed objects to the requirements to recheck, and decide whether execution continues or stops (Secs. 3.3–3.4).
R3-Table 2. Editing ablations (%, Table 4). PR: preservation rate; UCR: unintended change rate.
| Setting | Macro ↑ | PR ↑ | UCR ↓ |
|---|---|---|---|
| Full MUSE | 81.2 | 99.9 | 0.6 |
| History-only | 73.8 | 99.2 | 1.1 |
| History-only, no protection | 70.1 | 90.1 | 7.1 |
Even under history-only planning, disabling protection lowers PR to 90.1% and raises UCR to 7.1%. Removing requirement tracking or Scene Memory also lowers editing Macro to 76.4 and 73.2 (Table 4). Scene Memory carries identities, supports and functional groups across instructions (Figure 5), and Skill Memory provides fixed decomposition cues inspired by HSM [1] (Table 22).
Q4. Who created AuthorBench, and how are utility, aesthetics and suitability assessed?
A4. We wrote the AuthorBench tasks from the functional requirements of each room type, and two interior-design students independently reviewed them; scene quality is assessed separately from the benchmark checks.
As described in App. B.1, the 145 construction tasks cover six room types and are drafted from room-specific templates; each specifies object counts, supports, spatial relations and functional zones, and becomes an instruction with checks. Our editing tasks follow the scene-grounded, instruction-driven editing setting of Edit-As-Act [2], with our own task templates, complexity tiers and evaluation checks; the 240 test tasks start from 47 Edit-As-Act and 193 Imaginarium [3] scenes, the latter arranged by 20 professional artists. Two reviewers independently checked every task for consistency and realizability, revising or removing disputed cases (App. B.1); we will add their interior-design background.
Annotations do not prescribe a single placement: each check states a condition (support, a relation to a reference object or membership in a functional zone), and any layout satisfying all conditions is accepted. The checks thus measure instruction fulfillment, while utility, layout suitability and aesthetics are rated separately.
In the human study (Table 5), 50 participants, 16 with architecture or design backgrounds, rate instruction satisfaction, preservation and physical plausibility. We also rated the same 40 randomly sampled outputs with gpt-6.1-sol, five times each from the instruction and renderings only, adding functionality, layout suitability and aesthetic coherence.
R3-Table 3. Editing quality without AuthorBench checks (1–5). First three rows: human / model; others: model only.
| Dimension | Edit-As-Act | SceneReVis | ReSpace-E | MUSE |
|---|---|---|---|---|
| Instruction satisfaction | 2.5 / 3.2 | 3.2 / 3.7 | 3.6 / 4.1 | 4.5 / 4.7 |
| Preservation | 4.1 / 4.2 | 3.5 / 3.2 | 3.8 / 3.6 | 4.8 / 4.7 |
| Physical plausibility | 4.3 / 4.1 | 3.8 / 3.4 | 3.5 / 2.9 | 4.6 / 4.5 |
| Functionality | 3.8 | 4.5 | 3.7 | 4.5 |
| Layout suitability | 4.0 | 3.4 | 3.7 | 4.7 |
| Aesthetic coherence | 4.1 | 3.9 | 3.6 | 4.6 |
MUSE leads on five model-rated dimensions and ties SceneReVis on functionality, and human and model raters rank the four methods identically on every shared dimension. We will note expert assessment of architectural and ergonomic compliance as future work in App. E and document task design and reviewer backgrounds in App. B.1.
We hope these responses address the reviewer's concerns and welcome further questions. If they resolve the main issues, we would be grateful if the reviewer would consider updating the score.
References
[1] Pun et al. HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation. 3DV, 2026.
[2] Noh et al. Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing. CVPR, 2026.
[3] Zhu et al. Imaginarium: Vision-Guided High-Quality 3D Scene Layout Generation. ACM TOG, 2025.
Dear Reviewer yh9e,
We have uploaded a revised PDF with all changes marked in blue. New tables are numbered R1–R7 so that the table numbers cited in our response stay unchanged. Changes related to your comments:
- Secs. 3.3–3.4 and Table 8: protection is now described as derived from anchors, preservation requirements, and stable regions, correcting the earlier wording.
- Sec. 3.1: how each memory layer is updated and used.
- App. A.3.3 (Table R2): the four obstructed-placement cases.
- Sec. 4 and App. B.1: task design, annotator backgrounds, and scene provenance. App. B.1 also clarifies that the 47 Edit-As-Act and 193 Imaginarium test cases come from 8 and 52 distinct source scenes, respectively.
- Sec. 5.3 and App. B.4 (Table R4): ratings of functionality, layout suitability, and aesthetic coherence.
As the discussion period closes on November 18, we would be grateful to know whether our response and the revision address your concerns, and we are happy to clarify anything further.
Response to Reviewer yh9e (1/2): Obstructed Placement, Protection and Operation Vocabulary
We thank the reviewer for the concrete scenario and questions. Q1–Q3 address Weaknesses 1–3, and Q4 addresses the question on AuthorBench.
Q1. How does MUSE handle occupied target locations and walking paths, and why is its control mechanism needed?
A1. We agree that the video mainly shows simple operations. The obstructed-placement cases below show how MUSE detects unmet requirements and repairs the same object.
We ran the reviewer's scenario through the full pipeline in four new bedroom cases: add a table (1.2 × 0.6 × 0.75 m) in front of the bed while keeping a path from the door to the bedside. Table size and orientation, the target region and the path endpoints are fixed; cases B–D share the initial scene and differ in permissions or explicit rearrangement instructions.
Case B shows the difference between a successful operation and a satisfied request. Relational placement found no valid candidate and fell back to placement beside the bed; the action reported success, but the table lay outside the requested region and blocked the path endpoint. Inspector feedback detected both violations, and Working Memory kept the table's identity while marking the region and path requirements as unfinished. MUSE then adjusted the bench and repositioned the existing table instead of creating a second one, after which size, location, footprint-overlap, room-boundary and 0.8 m path checks passed, with the bed, nightstands and wardrobe unchanged (App. A.2.4, A.3.3).
Case C shows that users can also specify the rearrangement and its order. In case D, the fixed bench overlaps every feasible placement in the target region, so MUSE reports the conflict and leaves the scene unchanged rather than violating the fixed-furniture request.
Q3 provides aggregate evidence for these mechanisms through matched ablations. We will add these cases to App. A.3.3 and update the supplementary video to show the first attempt, feedback, repair, final checks and the stop in the infeasible case.
Q2. Why constrain edits locally, and why use MUSE for a simple operation vocabulary?
A2. Using an object as a spatial reference does not by itself prevent its movement; protection follows preservation requests and scene structure.
We agree that protection should reflect the task. In "add a side table beside the sofa", referring to the sofa does not fix its position. Working Memory tracks satisfied requirements and role-to-object bindings, while execution protection is derived from scene anchors, preservation requirements and stable regions (functional groups tracked across updates), with current edit targets excluded. Anchors and preservation requirements prevent deletion or replacement; stable regions and preservation requirements prevent movement, rotation or scaling, so a sofa inside a stable region can be protected. Both sets are rebuilt for each instruction (App. A.2.6). We will clarify this distinction in Sec. 3.3, correcting the statements that satisfied bindings automatically extend protection and that the transform-protected set contains the protected set, and document anchor selection, stable-region updates and protection-set derivation in App. A.2.4.
Protection does not trade completion for locality: removing the protection sets from full MUSE lowers editing Macro from 81.2 to 75.3 and raises UCR from 0.6% to 9.7% (Table 4).
The operation vocabulary is small by design; the difficulty lies in choosing the intended instance, interpreting spatial relations, checking the resulting scene and avoiding side effects. In Figure 6 (App. A.3.3), adding a rug creates the object but fails the under-table relation; feedback then directs a correction to the same rug, followed by checks of the relation and preservation. Existing editing systems reach 5.4–11.2% All-Goal on AuthorBench (Table 2), and a general-purpose agent given MUSE's scene operations reaches 65.0% on 80 editing tasks, versus 81.25% for MUSE.