acceptodds
Under review as a conference paper at ICLR 2027

MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction

Abstract

Text-driven 3D scene generation is a promising technique for digital content creation, embodied AI simulation, and interactive design, yet practical workflows often require refining, extending, or correcting existing scenes while preserving non-target content. Existing methods can produce realistic and structurally plausible scenes, but they generally lack editability with requirement-level state tracking, so part-level failures often lead to full-scene regeneration or manual intervention. To tackle this challenge, we formulate controllable 3D scene authoring as incremental requirement satisfaction, unifying construction and editing. We present MUSE, a memory-grounded multi-agent framework in which an Architect compiles instructions into structured requirements, a Sculptor executes local scene operations, and an Inspector verifies each step while updating Working, Scene, and Skill Memory. To evaluate requirement-level controllability and preservation-aware editing, we introduce AuthorBench, comprising 145 constrained construction cases and a 1,584-case editing pool with external structured checks. Against the strongest baseline, MUSE improves construction All-Goal success from 37.9% to 80.7% and surface-constraint fulfillment from 35.0% to 93.3%; on a stratified 240-case editing split, it achieves 76.7% All-Goal success, a 99.9% preservation rate, and a 0.6% unintended change rate. Human evaluations and navigation-proxy tests further support stronger intent alignment and spatial stability, while ablations validate the memory design. Together, these results establish MUSE as an effective framework for controllable 3D scene authoring.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Generation, Language-Guided Scene Editing, Multi-Agent Systems, Structured Memory, Constraint Satisfaction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yh9eOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents an approach MUSE: an agentic 3D scene authoring tool, with arguments for incremental local support to editing facilitated by a structured memory system of three different types of storage: tracking scene diagnostics/requirement status (working memory), persistent 3D scene memory and a third Skill memory mainly consisting of recurring decompositions. The work follows the popular line of incremental 3D scene editing/authoring task with a division of subtasks to the LLM/VLM. An important argument the paper proposes is to use local or compact support on the scene domain for every edits, i.e., the editing assumes non-target scene elements are protected where protection is decided based on certain rules, e.g., reference. Finally the paper proposes a scene editing dataset consisting of 145 scene construction examples, with a number of instructions per scene totalling 1584.

Critical strengths
  1. In cases where scene authoring is done by incremental editing by a user, use of an "external" organized memory as proposed is justified.

  2. The results and video show clearly the problem and its implementation.

Critical weaknesses
  1. Contribution in the context of the results: The main contribution of the work is the addition of memory and a local editing of scene based on a rule set. While the first can be somewhat justified, the somewhat rigid editing of scenes is not very interesting. When looking at the supplementary video, the paper's proposal boils down to local "blender add/remove and move/remove" interpreter. It does not show at all the value of the complex mechanism proposed with several process and memory submodules. E.g., what happens if the author asks "Add a table infront of the bed, but some objects and walking path already exist in front of the bed". A video example could have included such cases.

  2. The method itself is not very inspiring. Local support should be need-based and not as strict as advocated by the paper. I did not gain any new knowledge after reading through the method. For example, when adding an object, an intelligent agent should not protect an object just by reference. Another problem is that the scene authoring vocabulary looks extremely simple, e.g., "left/right, move/remove" etc. One could build a custom compiler for these simple words once extracted.

  3. The memory structure as a main contribution is not satisfactory given that the structured memory is in fact a simple reorganization of the prompt.

Questions for authors
  1. The paper does not describe expertise used in scene-authoring dataset. Who created the dataset? Does the data satisfy the most important criteria of scene authoring: i.e., is it architecturally meaningful (interior architecture)? How was utility, aesthetics, suitability decided for "the instruction to placement" annotation?
6 comments
AuthorsOpenReview ↗

Response to Reviewer yh9e (1/2): Obstructed Placement, Protection and Operation Vocabulary

We thank the reviewer for the concrete scenario and questions. Q1–Q3 address Weaknesses 1–3, and Q4 addresses the question on AuthorBench.


Q1. How does MUSE handle occupied target locations and walking paths, and why is its control mechanism needed?

A1. We agree that the video mainly shows simple operations. The obstructed-placement cases below show how MUSE detects unmet requirements and repairs the same object.

We ran the reviewer's scenario through the full pipeline in four new bedroom cases: add a table (1.2 × 0.6 × 0.75 m) in front of the bed while keeping a path from the door to the bedside. Table size and orientation, the target region and the path endpoints are fixed; cases B–D share the initial scene and differ in permissions or explicit rearrangement instructions.

R3-Table 1. Editing with spatial constraints.

ConditionExecutionOutcome
A: Free space; furniture fixedAdd table; 1 roundGoals met; furniture unchanged
B: Bench occupies target; rearrangement permittedFirst placement fails checks; MUSE adjusts the bench and repositions the same table; 3 roundsGoals met after repair
C: Explicit bench adjustment and orderAdjust bench, then add table; 2 roundsGoals met
D: Bench occupies target; furniture fixedReport conflict and stopScene unchanged; addition unmet

Case B shows the difference between a successful operation and a satisfied request. Relational placement found no valid candidate and fell back to placement beside the bed; the action reported success, but the table lay outside the requested region and blocked the path endpoint. Inspector feedback detected both violations, and Working Memory kept the table's identity while marking the region and path requirements as unfinished. MUSE then adjusted the bench and repositioned the existing table instead of creating a second one, after which size, location, footprint-overlap, room-boundary and 0.8 m path checks passed, with the bed, nightstands and wardrobe unchanged (App. A.2.4, A.3.3).

Case C shows that users can also specify the rearrangement and its order. In case D, the fixed bench overlaps every feasible placement in the target region, so MUSE reports the conflict and leaves the scene unchanged rather than violating the fixed-furniture request.

Q3 provides aggregate evidence for these mechanisms through matched ablations. We will add these cases to App. A.3.3 and update the supplementary video to show the first attempt, feedback, repair, final checks and the stop in the infeasible case.


Q2. Why constrain edits locally, and why use MUSE for a simple operation vocabulary?

A2. Using an object as a spatial reference does not by itself prevent its movement; protection follows preservation requests and scene structure.

We agree that protection should reflect the task. In "add a side table beside the sofa", referring to the sofa does not fix its position. Working Memory tracks satisfied requirements and role-to-object bindings, while execution protection is derived from scene anchors, preservation requirements and stable regions (functional groups tracked across updates), with current edit targets excluded. Anchors and preservation requirements prevent deletion or replacement; stable regions and preservation requirements prevent movement, rotation or scaling, so a sofa inside a stable region can be protected. Both sets are rebuilt for each instruction (App. A.2.6). We will clarify this distinction in Sec. 3.3, correcting the statements that satisfied bindings automatically extend protection and that the transform-protected set contains the protected set, and document anchor selection, stable-region updates and protection-set derivation in App. A.2.4.

Protection does not trade completion for locality: removing the protection sets from full MUSE lowers editing Macro from 81.2 to 75.3 and raises UCR from 0.6% to 9.7% (Table 4).

The operation vocabulary is small by design; the difficulty lies in choosing the intended instance, interpreting spatial relations, checking the resulting scene and avoiding side effects. In Figure 6 (App. A.3.3), adding a rug creates the object but fails the under-table relation; feedback then directs a correction to the same rug, followed by checks of the relation and preservation. Existing editing systems reach 5.4–11.2% All-Goal on AuthorBench (Table 2), and a general-purpose agent given MUSE's scene operations reaches 65.0% on 80 editing tasks, versus 81.25% for MUSE.

AuthorsOpenReview ↗

Response to Reviewer yh9e (2/2): Structured Memory and AuthorBench Construction

Q3. Is structured memory simply a reorganization of the prompt?

A3. At the planner interface, explicit state is indeed serialized into the prompt; the maintained state also drives execution guards, re-verification and stopping.

The history-only control in Table 4 (App. C.3.2) keeps all other components and budgets, but its planner receives the complete action/feedback history instead of explicit planner state and selects the next requirement itself. MUSE still raises construction Macro from 88.2 to 93.5 and editing Macro from 73.8 to 81.2, so how the state is organized matters for planning.

Outside the prompt, code updates Working Memory from combined verification results: deterministic checks take precedence for geometric and counting constraints, and model feedback covers the remaining semantic and visual aspects. Its records of requirement status and involved objects drive protection guards, link changed objects to the requirements to recheck, and decide whether execution continues or stops (Secs. 3.3–3.4).

R3-Table 2. Editing ablations (%, Table 4). PR: preservation rate; UCR: unintended change rate.

SettingMacro ↑PR ↑UCR ↓
Full MUSE81.299.90.6
History-only73.899.21.1
History-only, no protection70.190.17.1

Even under history-only planning, disabling protection lowers PR to 90.1% and raises UCR to 7.1%. Removing requirement tracking or Scene Memory also lowers editing Macro to 76.4 and 73.2 (Table 4). Scene Memory carries identities, supports and functional groups across instructions (Figure 5), and Skill Memory provides fixed decomposition cues inspired by HSM [1] (Table 22).


Q4. Who created AuthorBench, and how are utility, aesthetics and suitability assessed?

A4. We wrote the AuthorBench tasks from the functional requirements of each room type, and two interior-design students independently reviewed them; scene quality is assessed separately from the benchmark checks.

As described in App. B.1, the 145 construction tasks cover six room types and are drafted from room-specific templates; each specifies object counts, supports, spatial relations and functional zones, and becomes an instruction with checks. Our editing tasks follow the scene-grounded, instruction-driven editing setting of Edit-As-Act [2], with our own task templates, complexity tiers and evaluation checks; the 240 test tasks start from 47 Edit-As-Act and 193 Imaginarium [3] scenes, the latter arranged by 20 professional artists. Two reviewers independently checked every task for consistency and realizability, revising or removing disputed cases (App. B.1); we will add their interior-design background.

Annotations do not prescribe a single placement: each check states a condition (support, a relation to a reference object or membership in a functional zone), and any layout satisfying all conditions is accepted. The checks thus measure instruction fulfillment, while utility, layout suitability and aesthetics are rated separately.

In the human study (Table 5), 50 participants, 16 with architecture or design backgrounds, rate instruction satisfaction, preservation and physical plausibility. We also rated the same 40 randomly sampled outputs with gpt-6.1-sol, five times each from the instruction and renderings only, adding functionality, layout suitability and aesthetic coherence.

R3-Table 3. Editing quality without AuthorBench checks (1–5). First three rows: human / model; others: model only.

DimensionEdit-As-ActSceneReVisReSpace-EMUSE
Instruction satisfaction2.5 / 3.23.2 / 3.73.6 / 4.14.5 / 4.7
Preservation4.1 / 4.23.5 / 3.23.8 / 3.64.8 / 4.7
Physical plausibility4.3 / 4.13.8 / 3.43.5 / 2.94.6 / 4.5
Functionality3.84.53.74.5
Layout suitability4.03.43.74.7
Aesthetic coherence4.13.93.64.6

MUSE leads on five model-rated dimensions and ties SceneReVis on functionality, and human and model raters rank the four methods identically on every shared dimension. We will note expert assessment of architectural and ergonomic compliance as future work in App. E and document task design and reviewer backgrounds in App. B.1.

We hope these responses address the reviewer's concerns and welcome further questions. If they resolve the main issues, we would be grateful if the reviewer would consider updating the score.

References

[1] Pun et al. HSM: Hierarchical Scene Motifs for Multi-Scale Indoor Scene Generation. 3DV, 2026.

[2] Noh et al. Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing. CVPR, 2026.

[3] Zhu et al. Imaginarium: Vision-Guided High-Quality 3D Scene Layout Generation. ACM TOG, 2025.

Dear Reviewer yh9e,

We have uploaded a revised PDF with all changes marked in blue. New tables are numbered R1–R7 so that the table numbers cited in our response stay unchanged. Changes related to your comments:

  • Secs. 3.3–3.4 and Table 8: protection is now described as derived from anchors, preservation requirements, and stable regions, correcting the earlier wording.
  • Sec. 3.1: how each memory layer is updated and used.
  • App. A.3.3 (Table R2): the four obstructed-placement cases.
  • Sec. 4 and App. B.1: task design, annotator backgrounds, and scene provenance. App. B.1 also clarifies that the 47 Edit-As-Act and 193 Imaginarium test cases come from 8 and 52 distinct source scenes, respectively.
  • Sec. 5.3 and App. B.4 (Table R4): ratings of functionality, layout suitability, and aesthetic coherence.

As the discussion period closes on November 18, we would be grateful to know whether our response and the revision address your concerns, and we are happy to clarify anything further.

Reviewer_3iMyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MUSE is a multi-agent framework for text-driven 3D scene construction and editing that decomposes instructions into requirements, executes local object operations, and verifies outcomes while protecting non-target content. Its contributions include requirement-level tracking and object bindings, structured working/scene/skill memory, and AuthorBench, comprising 145 construction cases and a 1,584-case editing pool with a 240-case evaluation split. Validation includes baseline comparisons, component ablations, external benchmarks, human evaluations, short sequential-editing tests, and navigation-proxy experiments, demonstrating improved requirement satisfaction and scene preservation under the evaluated settings.

Critical strengths

Overall , I think the following aspects are the sounds of the study:

  1. Explicit requirement tracking and protection-aware editing. MUSE connects user requirements to object identities, verification outcomes, and execution constraints, providing a concrete mechanism for completing edits while preserving non-target content. The controlled history-only ablation supports the value of structured state.
  2. Evaluation of both task completion and preservation. AuthorBench evaluates individual requirements and unintended changes rather than relying solely on visual quality. External geometric checks, human evaluation, and additional benchmarks provide complementary evidence, making the demonstrated editing improvements convincing.
Critical weaknesses

My major concerns are as follows:

  1. Recent agentic works has not been discussed and compared so the advance is insufficiently established. Scenethesis already performs iterative, object-specific optimization of scale and pose, while other agentic systems support feedback-driven scene modification. and cited approaches such as SAGE and SceneWeaver are absent from the experiments. Although the ablations support structured state tracking, they do not establish its advantage over competitive agentic alternatives, leaving the central novelty claim insufficiently substantiated.

  2. The evidence does not adequately support persistent, reliable authoring across interactions. Working Memory and protection sets reset with each instruction, so earlier requirements remain enforced only when explicitly incorporated into the new program. The controlled sequential evaluation covers only two successive edits and measures completion through internal run status rather than external All-Goal checks. Because maintaining user intent across edits is central to the motivation, these limitations leave the manuscript’s principal claimed capability only partially demonstrated.

Questions for authors

I have following questions and hope the author can provide futher clarification.

  1. I am confused about the advances of the study . The difference between one-step generaion vs. agentic framework is clear for me. But among recent agentic works (SAGE, scenethesis, SceneSmith etc), what capability or measurable benefit does this representation provide?
  2. Stable object IDs support precise execution after selection, but the critical step is determining which instance the user intends. The paper provides no dedicated evaluation across repeated categories, identical assets, ambiguous descriptions, or references that depend on previous edits. Its failure audit explicitly includes incorrect target selection. How are competing instance matches resolved, and what happens when an instruction is ambiguous?
  3. Most spatial evaluation uses structured geometry, so camera reversal need not reverse a relation’s truth value. Nevertheless, the study does not fully explain how it chooses between room-relative, object-relative, and viewer-relative interpretations of “left,” “right,” “front,” and “behind.” Its physical checks rely on bounding boxes, footprints, and support heights; these do not by themselves establish mesh-level collision freedom or stability under gravity. What reference frame governs each relation, and how is linguistic ambiguity resolved consistently during planning and evaluation?
  4. The cross-backend experiments and history-only base model ablation are useful, but there is no evidence establishs whether MUSE’s specialized architecture remains necessary with a strong contemporary general-purpose agent such as GPT astra.
11 comments
AuthorsOpenReview ↗

Response to Reviewer 3iMy (1/3): Comparison with Recent Agentic Systems

We thank the reviewer for the expert questions. Q1 addresses Weakness 1 and Question 1, Q2 addresses Weakness 2, and Q3–Q5 address Questions 2–4.


Q1. What capabilities or measurable benefits does MUSE's representation offer over recent agentic systems?

A1. MUSE links each requirement to object IDs, verification results and execution protection. It outperforms SceneSmith on AuthorBench construction and improves over a history-only planner under matched tools.

Once a user has a satisfactory scene, local edits offer a natural way to create new versions while reusing its assets, layout and spatial relations. MUSE therefore uses the same requirement-driven operations to construct and to edit scenes (Secs. 1 and 3.1).

SAGE [1], SceneSmith [2] and SceneWeaver [4], already discussed in Sec. 2, use tools and iterative refinement, with SceneWeaver adding self-reflective planning and physics-aware execution; Scenethesis [3] optimizes object scales and poses. Object-level refinement is therefore not unique to MUSE; what MUSE adds is linking requirement state to verification and protection (Secs. 3.1–3.4).

R2-Table 1. Reported mechanisms and evaluation coverage. ✓: explicitly reported; ✗: not explicitly reported in the reviewed sources.

Mechanism / evaluationSAGESceneWeaverScenethesisSceneSmithMUSE
Scene construction✓✓✓✓✓
Object-level modification / refinement✓✓✓✓✓
Requirement state linked to object IDs and verification✗✗✗✗✓
Explicit execution guards for non-target preservation✗✗✗✗✓
Evaluation over successive user edit instructions✗✗✗✗✓

SceneSmith keeps TODO items and stage memory, and SceneWeaver keeps reflection memory, both recording generation progress. MUSE instead binds each user requirement to object IDs and verification status, which in turn drive execution protection.

For an experimental comparison, we ran the original SceneSmith implementation, whose released code supports our indoor construction setting, with its default GPT-5.2 backend on all 145 AuthorBench construction tasks; LayoutGPT and MUSE use MiMo-V2.5. SAGE and SceneWeaver are compared through their reported mechanisms (R2-Table 1).

R2-Table 2. Construction on AuthorBench (%). Higher is better.

MethodObj.Rel.Surf.ZoneMacroAll-Goal
LayoutGPT98.483.835.079.074.137.9
SceneSmith97.187.994.575.388.775.2
MUSE99.696.093.385.293.580.7

MUSE leads by 4.8 Macro and 5.5 All-Goal points, while SceneSmith is slightly higher on surface constraints (94.5 vs. 93.3). Because the backbones differ, this is a system-level comparison. The matched history-only control (Table 4; App. C.3.2) isolates explicit requirement state: with the same model, tools, verifier, protection and budget, it raises construction Macro from 88.2 to 93.5, editing Macro from 73.8 to 81.2, and non-target preservation from 99.2% to 99.9%.

Reuse also matters in practice. Initial construction can be costly: SceneSmith reports 3 h 26 min and USD 13.98 per room on its own evaluation [2, App. O.5]. In navigation curriculum generation (Sec. 5.3, Table 6), incremental editing raises adjacent-stage free-space IoU from 0.649 to 0.978 and difficulty monotonicity from 0.394 to 0.850 compared with independent rebuilding.

We will add Scenethesis and expand the comparison in Sec. 2, add the SceneSmith results to Sec. 5.1 and clarify the role of reuse in Sec. 5.3. We also plan to study whether scene variants improve robot policy training.

References

[1] Xia et al. SAGE: Scalable Agentic 3D Scene Generation for Embodied AI. CVPR, 2026.

[2] Pfaff et al. SceneSmith: Agentic Generation of Simulation-Ready Indoor Scenes. 2026.

[3] Ling et al. Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation. 2025.

[4] Yang et al. SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent. NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer 3iMy (2/3): Persistent Authoring and Target Identification

Q2. Can MUSE complete new instructions while preserving valid earlier requirements?

A2. We agree that earlier requirements are not automatically re-enforced. Over five edits, MUSE nevertheless completes more instructions and retains more earlier constraints than history-only.

Working Memory and protection sets are rebuilt for each instruction, so an earlier requirement is explicitly enforced only when it is part of the current program (Sec. 3.1; App. A.2.6). Scene Memory carries object identities, geometry and support relations forward; Figure 5 shows a bench replaced by a bookshelf, books and a vase added, and both moved with the shelf. Users can also state which relations should persist, e.g., "Move the bookshelf beside the dresser, keeping the books and vase on the shelves."

Table 24 (App. C.4.1) of the submission covers 50 sequences of construction plus two edits, scored by internal run status. To address both limitations, we extended these sequences to five edits and reran both methods with external scoring. Full MUSE and history-only receive the same instructions, start from identical construction scenes, share models, tools and budgets, and each continues from its own previous output. The evaluator checks the current instruction and tracks historical constraints; its records are hidden from both methods, and requirements explicitly revised or revoked later are excluded.

R2-Table 3. External evaluation over five edits. Chain success requires all five current instructions to pass.

MetricFull MUSEHistory-only
Round-1 All-Goal42/50 (84%)40/50 (80%)
Round-5 All-Goal36/50 (72%)31/50 (62%)
Decline, round 1 to 512 points18 points
All-Goal across all rounds192/250 (76.8%)174/250 (69.6%)
Chains passing all five edits22/50 (44%)15/50 (30%)
Round-5 historical retention96%92%
Round-5 historical satisfaction90%84%

Retention is the fraction of previously satisfied, still-active constraints that remain satisfied; satisfaction is the fraction of all still-active historical constraints that are satisfied, both pooled over tasks. MUSE completes more edits, declines less in current-instruction success and retains more earlier constraints. Because both methods share scene state and execution protection, this gap reflects structured planner state, including requirement status and object bindings. We will report the protocol with per-round and per-chain results in App. C.4.1 and clarify state updates in Sec. 3.1 and App. A.2.6; App. E already states the carry-over limitation.


Q3. How does MUSE identify the intended instance and handle ambiguous instructions?

A3. MUSE matches object descriptions and supported scene roles, then executes through stable IDs. It does not yet flag ambiguity or ask for clarification.

Matching uses Scene Memory metadata (App. A.2.5) and role bindings in Working Memory (Table 8), such as "the left nightstand". In the current implementation, indistinguishable candidates are resolved by a fixed deterministic order, and there is no dedicated mechanism for references to earlier actions such as "the one moved earlier". App. E already notes that missing or ambiguous references can prevent target grounding.

The failure audit in App. D.2 quantifies this error source: among 29 audited move, replace and rotate cases, one failed move grounded the wrong same-category target, and four of five failed replacements missed or misbound the target. In a new eight-case diagnostic with manually specified structured instructions, default matching selected the intended target in 6/8 cases: 2/2 for appearance descriptions, 2/2 for supported roles and 2/4 for general spatial descriptions. When the correct ID was supplied before binding, all eight edits moved only the target and passed its position checks. Errors therefore arise in matching rather than execution, with general spatial descriptions as the weakest case.

Beyond the submission, an interactive prototype lets users select an instance in the floor plan or object list, read its ID and include it in the next instruction. Since MUSE already reports blocked actions instead of forcing them (Table 10), the next step is to detect ambiguous matches and request clarification before editing. We will document matching in App. A.2.5, report the new diagnostic in App. D.2, describe the prototype and expand the limitation in App. E.

AuthorsOpenReview ↗

Response to Reviewer 3iMy (3/3): Reference Frames, Physical Checks and General-Purpose Agents

Q4. How are spatial reference frames shared across planning and evaluation, and what do the physical checks verify?

A4. MUSE assigns a reference frame by relation type and uses the same directional definitions in execution and scoring. Its physical checks assess static geometry.

As listed in Table 13 (App. B.2.2), relation checks use subject and object boxes, the room frame and the reference object's facing direction, and relation placement uses the same facing direction (App. A.3.2). Object-relative left, right, front and behind therefore follow the reference object's recorded horizontal orientation, so "in front of the desk" means the desk's local front. Room relations such as "against the east wall" use room coordinates, and positions along a wall use the wall segment's local frame.

In a new controlled diagnostic, execution, internal verification and external scoring agreed on all 16 checks: moving only the recorded camera to the opposite side leaves the chair in front of and to the right of the desk, whereas rotating the desk by 180° places the chair behind and to its left.

Support checks use footprint overlap and height alignment. Collision checks first filter candidate pairs with axis-aligned bounding boxes (AABB), then test the transformed asset meshes with Trimesh/FCL; missing or degenerate meshes use box proxies, with AABB overlap as the fallback if mesh checking is unavailable. Table 13 lists only the box stage, so we will describe the full procedure in App. B.2.2. Two limitations remain: viewer-relative wording such as "to the desk's left as seen from here" follows the object-relative convention, and, as App. E.1 notes, stability under gravity is not evaluated. We will also add this diagnostic and specify tolerances in App. A.3.2.


Q5. Is a specialized scene-authoring architecture still useful as general-purpose agents improve?

A5. Yes, mainly for editing. With the same scene tools, a strong general-purpose agent matches MUSE's construction All-Goal but reaches 65.0% versus 81.25% in editing, where requirement state and protection matter most. MUSE also improves with stronger backbones, so the two are complementary.

As general-purpose agents improve, direct comparisons help establish which design choices remain useful. We compared Astra + Codex (gpt-6-astra, medium) with and without MUSE's scene tools on 40 construction and 80 editing tasks; the tool variant adds scene operations but not MUSE's planning, memory or scheduling. Both variants use code and file tools, identical public inputs and the same goal checker, and MUSE's outputs are re-scored on the same tasks. Because most construction inputs differ from MUSE's, we compare against MUSE mainly on editing.

R2-Table 4. Quality, text-token cost and mean runtime per task. Quality: 40 construction and 80 editing tasks; MUSE* cost and time: full benchmark (Table 27). Text-token costs only, at Standard rates with cache discounts (Astra) or MiMo-V2.5 rates (MUSE*).

TaskMethodMacro-GF ↑All-Goal ↑USD ↓Seconds ↓
BuildAstra + Codex83.49%50.00%0.20742.45
Build+ MUSE tools86.35%77.50%0.625156.17
BuildMUSE*92.63%77.50%0.020108.24
EditAstra + Codex84.16%66.25%0.289141.60
Edit+ MUSE tools82.14%65.00%0.711296.38
EditMUSE*85.34%81.25%0.012479.53

Three tool-assisted editing runs produced no valid output and count as failures; among the 77 valid outputs, 52 pass (67.5%), still well below MUSE.

Scene tools give the general agent most of what construction needs, raising its All-Goal from 50.0% to 77.5%. They do not help editing (66.25% → 65.00%), where the agent must track which requirements are met and which objects must stay unchanged; on nearly identical inputs (60/80 identical, the rest differing only in wall metadata), MUSE reaches 81.25%. The matched history-only control identifies structured planner state as a key factor, adding 5.3 construction and 7.4 editing Macro points under the same model, tools and budget (Table 4). Astra without tools is the fastest; adding tools raises its text cost per task from USD 0.207 to 0.625 in construction and from 0.289 to 0.711 in editing.

Specialized structure and stronger base models are complementary. As reported in Table 21 (App. C.3.1), replacing Qwen3-VL-72B with Qwen3-VL-235B-FP8 while keeping the rest of the pipeline unchanged raises MUSE's editing Macro from 69.5 to 78.1, and structured authoring traces could in turn supply training data for base models. We will add this comparison to Sec. 5.3 and App. C.3.2.

We hope these responses address the reviewer's concerns and welcome further questions. If they resolve the main issues, we would be grateful if the reviewer would consider updating the score.

Reviewer_5m8DOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors formulate controllable 3D scene authoring as incremental requirement satisfaction over a persistent scene state, unifying construction and preservation-aware editing. They present MUSE, a multi-agent framework in which an Architect compiles instructions into typed requirements, a Sculptor executes protection-constrained local operations, and an Inspector verifies each step while maintaining Working, Scene, and Skill Memory. They also introduce AuthorBench, a requirement-level benchmark with 145 construction cases and a 1,584-case editing pool.

Critical strengths
  1. The reformulation from one-shot scene synthesis to requirement-level incremental authoring is well motivated, and the explicit protection sets give a clean mechanism for preservation-aware editing that prior dialogue-history-based systems lack.

  2. The empirical study is unusually thorough for this line of work: cross-backend transfer over five VLMs (including a 7B model), component ablations (Table 4), two external benchmarks (Table 3), human evaluation, and a downstream navigation-curriculum test.

Critical weaknesses
  1. AuthorBench is co-developed with MUSE (Sec. 4), and the benchmark's check taxonomy (object / relation / surface / zone / preserve / forbid) is almost isomorphic to MUSE's requirement types (hard / edit / preserve / forbid). Even granting that the benchmark was frozen first, the baselines are being scored under a decomposition that only MUSE natively emits. Could the authors report results on a benchmark whose check taxonomy was fixed independently of the method, or at minimum quantify how much of the gain survives when checks are re-expressed in a method-agnostic form?

  2. The headline construction gain is dominated by one check family. MUSE reaches 93.3% on surface constraints versus 35.0% for the best baseline, but after removing surface-family checks the All-Goal gap shrinks from 42.8 points (80.7 vs. 37.9) to 7.6 points (87.6 vs. 80.0) (Table 1). Since the Sculptor has dedicated support-aware placement and snap-to-surface tooling that the baselines do not, this reads less like a framework-level advantage and more like a tooling advantage. The claim in Sec. 5.1 that "the improvement extends beyond surface placement" is technically true but considerably weaker than the abstract suggests.

  3. The editing baselines appear to be adapted in a way that makes the comparison uninformative. LayoutGPT-E obtains PR 25.1% and UCR 95.7% (Table 2), i.e. it overwrites essentially the whole scene, which suggests the adapter regenerates the scene rather than editing it. A method that cannot edit is not a meaningful editing baseline. How were LayoutGPT-E and ReSpace-E adapted, and is there a variant that at least attempts localized modification?

  4. In Table 2, Ret. (= Complex/Easy GF) exceeds 100% for MUSE (104.8) and Edit-As-Act (102.3), meaning Complex-tier performance is better than Easy-tier. This is counterintuitive and is never discussed. Is this a tier-definition artifact, a small-sample effect on the 240-case split, or an asymmetry in how checks are generated per tier?

  5. Skill Memory consists of ten manually authored motif cards (Sec. 3.2). The ablation shows removing it costs 4.8 Macro and 6.3 Complex points (Table 4), so it carries real weight. The paper argues the cards contain no case identifiers, but the relevant question is distributional: if AuthorBench's recurring motifs are drawn from the same set of common indoor patterns the cards encode, the prior is effectively tuned to the benchmark. A held-out evaluation on motifs not covered by the ten cards would make this much more convincing.

  6. The single-pass ablation (Macro 93.5 → 14.5) is not informative about MUSE's specific design, since removing iteration would collapse essentially any agentic pipeline. The informative comparisons are the history-only variants, and those show a more modest 5.3 / 7.4 point drop — this is the number that should anchor the claim that structured memory beats chronological history.

  7. The central claim is persistent state across instructions, but the main editing evaluation uses 1,584 single-instruction cases. The only multi-turn evidence is the 100 three-stage navigation curricula (Table 6), which the authors note are scored with a grid-based proxy rather than a trained policy. A multi-instruction editing protocol, measuring drift and preservation over several sequential edits, would directly test the paper's main hypothesis.

  8. No cost analysis is reported. MUSE runs three agents with per-requirement verification and a global feasibility pass, against largely single-pass baselines. Token consumption, wall-clock time, and number of LLM calls per scene should be reported alongside the quality metrics.

Questions for authors

No

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5m8D (1/3): Benchmark Alignment and Sources of Construction Gains

We thank the reviewer for the careful reading and constructive suggestions. Q1–Q8 follow the order of the eight Critical Weaknesses.


Q1. Benchmark–method alignment and independent evaluation

A1. We agree that AuthorBench's check families align with MUSE's requirement types. External benchmarks, an independent agentic system and ratings made without these checks provide complementary evidence.

AuthorBench checks derive from fixed specifications and source-scene metadata rather than MUSE outputs (Sec. 4; App. B.1), and the submission also reports two external benchmarks (Table 3; App. C.2). On all 63 E2A-Bench editing tasks, scored by the benchmark's own evaluator and averaged per room type, MUSE obtains 70.4/87.3/94.0 in Instruction Fulfillment, Scene Coherence and Physical Plausibility, versus 69.1/86.6/91.6 reported for Edit-As-Act, i.e., higher on all three. For construction, we adapt SceneEval's 94 single-room cases by translating its official count and spatial-relation annotations into natural-language requirements for every method. MUSE scores 55.4/83.8/25.3/79.7 in count (CNT), support (SUP), collision (COLsc, lower is better) and accessibility (ACC), versus 42.7/61.3/73.0/90.0 for Holodeck, leading on the first three while Holodeck leads on accessibility.

New in this response, we ran SceneSmith, an independently developed agentic system, with its native GPT-5.2 backend on all 145 AuthorBench construction tasks. It reaches 88.7 Macro, 94.5 Surface GF and 75.2 All-Goal, versus 93.5/93.3/80.7 for MUSE with MiMo-V2.5. AuthorBench therefore credits a strong system built without MUSE's representation, and MUSE still leads.

In a new assessment without AuthorBench checks, we rated the human study's 40 outputs, drawn from 10 randomly sampled editing cases per method (App. B.4), with gpt-6.1-sol, five times each on 1–5 scales. The judge sees only the instruction and the initial and final renderings; method names, internal states and checks are withheld.

R1-Table 1. Editing quality without AuthorBench checks. First three rows: human (Table 5) / model; remaining rows: model only.

DimensionEdit-As-ActSceneReVisReSpace-EMUSE
Instruction satisfaction2.5 / 3.23.2 / 3.73.6 / 4.14.5 / 4.7
Preservation4.1 / 4.23.5 / 3.23.8 / 3.64.8 / 4.7
Physical plausibility4.3 / 4.13.8 / 3.43.5 / 2.94.6 / 4.5
Functionality3.84.53.74.5
Layout suitability4.03.43.74.7
Aesthetic coherence4.13.93.64.6

MUSE leads on five model-rated dimensions and ties SceneReVis on functionality, and human and model raters rank the four methods identically on every shared dimension. We will document the rating protocol in App. B.4 and the external adaptations in App. C.2.


Q2. How much of the construction gain comes from surface handling and tools?

A2. Support-aware tools are a substantial part of MUSE's construction capability; explicit planner state adds a consistent further gain within the same execution framework.

We agree that the headline gap needs qualification: without surface checks, the All-Goal advantage over LayoutGPT falls from 42.8 to 7.6 points (87.6 vs. 80.0; Table 1). The history-only control in Table 4 (App. C.3.2) separates the two sources. It keeps MUSE's Architect output, model, assets, tools, Inspector, geometric repair, guards and budget, but its planner receives the scene, the requirements and the chronological action/feedback history instead of explicit planner state.

R1-Table 2. Construction results (%). History-only and full MUSE share tools and guards.

MethodMacro ↑Surface GF ↑All-Goal ↑
LayoutGPT74.135.037.9
History-only88.286.073.1
Full MUSE93.593.380.7

Moving from LayoutGPT to the iterative history-only system with MUSE's tools raises Surface GF from 35.0 to 86.0 and All-Goal from 37.9 to 73.1, combining tooling, planning and feedback. Within the shared framework, explicit planner state then adds 5.3 Macro, 7.3 Surface GF and 7.6 All-Goal points. We will qualify the abstract's construction claim with the surface-excluded comparison, separate these sources of improvement in Sec. 5.1, and lead Sec. 5.3 with the matched history-only comparison.

AuthorsOpenReview ↗

Response to Reviewer 5m8D (2/3): Editing Baselines, Tier Retention and Skill Memory

Q3. Is LayoutGPT-E an appropriate editing baseline, and do other methods attempt local changes?

A3. We agree that LayoutGPT-E is a regeneration reference rather than a native editor. Edit-As-Act, SceneReVis and ReSpace-E do edit the existing scene.

As described in App. B.3 (Table 14), LayoutGPT-E receives the instruction and the serialized source scene and generates a complete layout, without a local-editing interface. The other three methods run their native editing pipelines: ReSpace-E edits its structured scene representation, Edit-As-Act uses its goal-regressive action loop, and SceneReVis uses its released multi-turn revision pipeline; only their final scenes are converted for evaluation.

R1-Table 3. Editing completion and preservation (%, Table 2). PR: preservation rate; UCR: unintended change rate.

MethodAll-Goal ↑PR ↑UCR ↓
LayoutGPT-E (regeneration)36.225.195.7
Edit-As-Act5.484.917.8
SceneReVis10.492.65.9
ReSpace-E11.2100.01.9
MUSE76.799.90.6

All-Goal requires every goal check to pass, so completion and preservation must be read together. ReSpace-E returns evaluable outputs for all 240 tasks and preserves all non-target content (PR 100.0%), but completes 11.2% of tasks. MUSE combines 76.7% completion with 99.9% PR and 0.6% UCR. Tables 14–15 further show that Edit-As-Act, run on MiMo-V2.5 like all AuthorBench baselines, is weak on surface- and zone-heavy edits and returns evaluable scenes for only 212 of 240 tasks. We will list LayoutGPT-E separately as a regeneration reference in Table 2 and expand App. B.3 on editing interfaces, scene conversion and object identities.


Q4. Why does Complex/Easy Ret. exceed 100%?

A4. The tiers differ in operation mix as well as compositional complexity, so GF need not decrease monotonically across tiers for a given system.

As reported in Table 17 (App. C.1), MUSE's Easy/Complex GF is 80.0%/83.9% and Edit-As-Act's is 21.7%/22.2%. Each tier contains 60 distinct tasks. The Easy tier includes 16 scaling tasks, on which MUSE (MiMo-V2.5) passes only 3 of 16 scaling checks. Removing scaling checks changes MUSE's Easy/Complex GF to 90.0%/85.9%, reversing the order, while every task retains at least one goal check.

Because GF gives partial credit within each task and is then averaged over tasks (App. B.2.1), the tiers compare different mixes of operations and checks, not the same edits with complexity increased in isolation. For MUSE, the scaling diagnostic identifies operation mix as the main cause; with 60 tasks per tier, the 3.9-point difference corresponds to about two tasks. App. D.2 (Table 28) and App. E already identify scaling as the weakest operation (48.8% GF); we will add the per-tier operation breakdown to App. C.1 and clarify how Ret. should be read in App. B.2.1.


Q5. Is Skill Memory tailored to AuthorBench's recurring motifs?

A5. The cards encode generic decomposition patterns rather than benchmark cases. Their gains appear only where a card is retrieved, and without them MUSE still reaches 88.7 construction and 71.4 editing Macro.

The ten fixed cards give category-level decomposition and role-binding cues for common indoor patterns. They contain no case IDs, reference layouts or case-specific target values, and remain unchanged during evaluation (Sec. 3.2; App. A.2.2). Table 22 (App. C.3.2) of the submission groups the Skill Memory ablation by whether a card is retrieved:

R1-Table 4. Macro GF (%) grouped by card retrieval (Table 22). Delta is full MUSE minus the variant without Skill Memory.

TaskSubsetCasesFullw/o SkillDelta
ConstructionTriggered13593.087.9+5.1
ConstructionNon-triggered10100.099.3+0.7
EditingTriggered17387.273.6+13.6
EditingNon-triggered6765.865.7+0.1

Cards are retrieved in 93% of construction and 72% of editing cases because AuthorBench rooms contain these common indoor patterns. Where no card is retrieved, Skill Memory has no effect (+0.1 in editing), consistent with a decomposition prior for recurring patterns rather than memorized cases. Without the cards, MUSE's construction Macro (88.7) remains well above LayoutGPT's (74.1). A held-out-motif evaluation would test generalization to unseen patterns directly; we will state this in App. E and clarify the grouping in App. C.3.2.

AuthorsOpenReview ↗

Response to Reviewer 5m8D (3/3): History-Only Control, Sequential Editing and Cost

Q6. History-only controls as the main evidence for structured memory

A6. We agree that the matched history-only control should anchor the claim about explicit planner state.

With the same models, tools, verifiers, guards and budget, MUSE improves construction Macro by 5.3 points and editing Macro by 7.4 (Table 4; App. C.3.2). We will lead the ablation discussion in Sec. 5.3 with these results and use the single-pass comparison only to explain the role of iteration.


Q7. Can MUSE complete new instructions and preserve earlier results across sequential edits?

A7. In a new five-round evaluation, MUSE achieves higher current-instruction completion and higher retention of earlier constraints than history-only.

Table 24 (App. C.4.1) of the submission covers 50 sequences with two edits, scored by internal run status. We extended these sequences to five edits and reran both methods with external scoring. Full MUSE and history-only receive the same instructions, start from identical construction scenes, share models, tools and budgets, and each continues from its own previous output. The evaluator checks the current instruction and tracks historical constraints; its records are hidden from both methods, and constraints explicitly revised or revoked by a later instruction are excluded.

R1-Table 5. External evaluation over five edits. Chain success requires all five current instructions to pass; historical metrics pool check counts over tasks.

MetricFull MUSEHistory-only
Round-1 All-Goal42/50 (84%)40/50 (80%)
Round-5 All-Goal36/50 (72%)31/50 (62%)
Decline, round 1 to 512 points18 points
All rounds192/250 (76.8%)174/250 (69.6%)
Chains passing all five edits22/50 (44%)15/50 (30%)
Round-5 historical retention96%92%
Round-5 historical satisfaction90%84%

Both methods decline monotonically across rounds, but MUSE declines more slowly, completes more edits and retains more earlier constraints. Retention is the fraction of previously satisfied, still-active constraints that remain satisfied; satisfaction covers all still-active historical constraints. Scene Memory carries the scene structure forward, while Working Memory and protection sets are rebuilt for each instruction, so an earlier requirement is explicitly enforced only when it is part of the current program (Sec. 3.1; App. A.2.6), a limitation already stated in App. E. We will report the protocol with per-round and per-chain results in App. C.4.1.


Q8. How many model calls and tokens does MUSE use, and how long does each scene take?

A8. MUSE uses 11.6 LLM requests per construction scene and 6.5 per editing scene, at an estimated text-token cost of about USD 0.020 and 0.012.

These costs are reported in Table 27 (App. C.5) of the submission; we summarize them here alongside quality, measured on the same runs.

R1-Table 6. MUSE quality and per-scene cost. Token means are approximate totals divided by case counts; costs apply MiMo-V2.5 text rates to recorded text tokens.

MetricConstruction (145)Editing (240)
All-Goal ↑80.7%76.7%
Mean LLM requests per scene11.626.48
Mean input / output tokens per scene~133.7k / 4.6k~76.0k / 3.4k
Mean / median end-to-end time108 / 85 s480 / 296 s
Estimated text-token cost per sceneUSD 0.020USD 0.012

Requests cover requirement compilation, planning and verification; end-to-end time also includes scene processing and the final feasibility pass. Editing starts from populated scenes and spends time on loading, grounding, collision checking and local verification, so fewer requests need not mean less wall time. MUSE's multimodal models also receive images, whose tokens are not included in the cost estimate.

Compared with single-pass baselines such as LayoutGPT, which makes one generation call per scene, MUSE spends about 12 calls per construction scene on requirement compilation, verification and repair. This overhead costs about USD 0.02 in text tokens and under two minutes per construction scene on average, while editing time is dominated by scene processing rather than model calls; App. E already notes this latency and plans selective verification and caching. We will place this summary next to the main results in Secs. 5.1–5.2.

Comments on the paper

1 comment
Area_Chair_tY9kOpenReview ↗

Discussion phrase will end soon

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.