MUSE: Agentic 3D Scene Authoring via Memory-Grounded Incremental Requirement Satisfaction

Text-driven 3D scene generation is a promising technique for digital content creation, embodied AI simulation, and interactive design, yet practical workflows often require refining, extending, or correcting existing scenes while preserving non-target content. Existing methods can produce realistic and structurally plausible scenes, but they generally lack editability with requirement-level state tracking, so part-level failures often lead to full-scene regeneration or manual intervention. To tackle this challenge, we formulate controllable 3D scene authoring as incremental requirement satisfaction, unifying construction and editing. We present MUSE, a memory-grounded multi-agent framework in which an Architect compiles instructions into structured requirements, a Sculptor executes local scene operations, and an Inspector verifies each step while updating Working, Scene, and Skill Memory. To evaluate requirement-level controllability and preservation-aware editing, we introduce AuthorBench, comprising 145 constrained construction cases and a 1,584-case editing pool with external structured checks. Against the strongest baseline, MUSE improves construction All-Goal success from 37.9% to 80.7% and surface-constraint fulfillment from 35.0% to 93.3%; on a stratified 240-case editing split, it achieves 76.7% All-Goal success, a 99.9% preservation rate, and a 0.6% unintended change rate. Human evaluations and navigation-proxy tests further support stronger intent alignment and spatial stability, while ablations validate the memory design. Together, these results establish MUSE as an effective framework for controllable 3D scene authoring.