acceptodds
Withdrawn from ICLR 2027

I-OctoScene: Multi-agent 3D Scene Synthesis with Multimodal Interactive Instructions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D Scene Synthesis (3DSS) has emerged as a key enabler of embodied AI, providing diverse and controllable 3D environments for training and evaluation. Although recent approaches have achieved notable progress in 3DSS, they are still constrained by closed and monolithic paradigms, limiting synthesis personalization and quality. To address these issues, we propose I-OctoScene, an Interactive Multi-Agent 3DSS framework that generates 3D indoor scenes through multi-agent collaboration guided by progressive external interactions. Specifically, I-OctoScene introduces a Progressive Multimodal Interactive Refinement module (PMIR) to support the interpretation and integration of external feedback, enabling controllable scene generation aligned with personalized user requirements. In addition, an Intent-Aligned Tool Coordination mechanism (IATC) leverages Adaptive Tooling Skills (ATSs) to systematically orchestrate tool usage, facilitating complex external intent resolution and coordinated tool collaboration, ultimately improving synthesis quality. Extensive quantitative and visualization experiments across diverse room-type settings demonstrate the superiority of our proposed method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-agent, interactive learning, 3D scene synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TpWvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the study proposed I-OctoScene. I-OctoScene is a multi-agent framework for interactive 3D indoor scene synthesis and editing, incorporating progressive feedback through text, clicks, and annotated images. Its contributions include multimodal feedback integration, structured tool coordination through predefined Adaptive Tooling Skills, and a benchmark assessing scene quality and interaction compliance. Validation includes comparisons on common and open-vocabulary room types, component ablations, a 10-participant human evaluation, and runtime/token-cost analysis, with reported improvements in scene quality and user alignment.

Critical strengths

After reading the manuscript, there are several aspects that I think is well addressed:

  1. Practical interactive authoring: The framework supports progressive instructions through text, clicks, and annotated images, enabling users to refine scenes as their requirements evolve.
  2. Baseline coverage: Comparisons include established scene-generation methods and the directly relevant SceneWeaver under shared model, asset, and tool settings, providing a useful assessment of scene quality.
  3. Complementary empirical validation: Component ablations and human evaluation support the framework’s effectiveness, with human preferences providing evidence beyond model-based scoring.
Critical weaknesses

My concerns are as follows:

  1. The paper does not sufficiently demonstrate that local edits preserve overall scene consistency. Although agents receive global scene information, the experiments do not adequately test whether new edits maintain realistic object sizes, usable space, and earlier user requirements. Handling of infeasible requests, such as adding another bed to an already crowded room, also remains unclear. This limits confidence in the system’s reliability during practical, repeated editing.
  2. The evaluation is too forgiving to show whether edits are correct. The scoring instructions allow high scores when the system merely attempts an edit or produces something roughly matching the request. The final evaluation also ignores specific details and coordinates. Therefore, the reported scores do not convincingly show that the system correctly performs the user’s requested changes. Do the proposed tools still provide meaningful improvements when used with a more capable foundation model, such as GPT-6 Astra?
  3. I would love to see discussion related to the failure cases.
  4. Relevant agentic methods, such as Scenethesis and SAGE, should be discussed, with a clear explanation of how the proposed approach differs from them.
Questions for authors
  1. Although the planner receives 3D layout metadata, how are support, containment, and under-object relationships geometrically verified? How does the method evaluate the cases where top-down view is insufficient to determine whether the requested relationship is satisfied?
  2. How would the multi-agent and ATS ablations with a stronger contemporary backbone such as GPT 6? Do the proposed modules still improve performance?
  3. Please report end-to-end runtime and failure rates over all attempted runs and add failure case disucssion.
  4. How does the system handle the user request such as inserting a second bed into a bedroom with insufficient usable space? Specify how the system prioritizes user intent, realistic dimensions, collision avoidance, accessibility, and preservation of existing objects, and what it communicates when all requirements cannot be satisfied.
Reviewer_z8b8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces I-OctoScene, an interactive multi-agent framework for indoor 3D scene synthesis through multi-agent collaboration. The framework incorporates Progressive Multimodal Interactive Refinement (PMIR) to support user feedback and Intent-Aligned Tool Coordination (IATC) to organize tool execution through predefined skills. Experiments demonstrate improvements in scene quality and interaction compliance over existing approaches.

Critical strengths
  1. Well-Structured Multi-Agent Framework. The separation of planning, execution, and evaluation into specialized agents provides a clear and interpretable workflow for scene synthesis.
  2. Flexible Multimodal Interaction. The framework supports progressive scene refinement through textual instructions, visual annotations, and interactive controls, improving user controllability.
Critical weaknesses
  1. Limited Methodological Novelty. The proposed framework largely reuses the scene generation and editing tools from prior work SceneWeaver, with its main additions consisting of role-specific agent prompts and predefined tool workflows (ATSs). Consequently, the technical novelty beyond reorganizing existing tools into a multi-agent framework remains limited.
  2. Insufficient Baseline Comparisons. The paper lacks comparisons with stronger agentic frameworks, such as SceneCode, SceneSmith, and GPT with Codex. Moreover, computational budgets, including token consumption and tool calls, are not strictly controlled across methods, making it difficult to attribute improvements to the proposed framework.
  3. Insufficient Validation for Embodied AI. Although the paper motivates scene synthesis for embodied AI, the experiments primarily evaluate visual quality, object collisions, and instruction following. Physical stability, robot reachability, and downstream robotic task performance are not evaluated, leaving the practical applicability of generated scenes insufficiently demonstrated.
Questions for authors
  1. Why do the authors use GPT-4.1 rather than a more powerful foundation model such as GPT-6?
  2. Why the interaction capability is directly included in the benchmark score? It's unfair, since it inherently disadvantages baselines that do not support progressive interaction.
Reviewer_RTaaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents I-OctoScene, an interactive multi-agent framework for indoor 3D scene synthesis from existing assets. Specialized agents interpret user feedback, plan edits, execute tools, and evaluate results over successive rounds. Progressive Multimodal Interactive Refinement (PMIR) supports language, control inputs, and visual annotations, while Intent-Aligned Tool Coordination (IATC) uses predefined Adaptive Tooling Skills (ATSs) to organize tool execution. The paper introduces an interaction evaluation framework and reports comparisons across common and open-vocabulary room types, component ablations, human assessment, and cost analysis.

Critical strengths
  1. The paper addresses a useful task: interactive 3D scene construction that allows users to refine a scene through language, clicks, and visual annotations over multiple rounds.
  2. The experiments cover diverse room types and include component ablations, human preference evaluation, and cost analysis. The results demonstrate improvements in several scene quality measures and encouraging examples of richer layouts.
  3. The paper is clearly written. The appendix provides detailed agent prompts, tool descriptions, and skill definitions that help explain the implementation.
Critical weaknesses
  1. The methodological novelty appears limited. The framework combines role-based agents, multimodal feedback, and manually defined skills, with its main contribution lying in the integration of these established mechanisms for interactive 3D scene synthesis.
  2. The interaction evaluation relies heavily on permissive LLM judgments. Approximate compliance or visible attempts can receive high scores, and final-demand evaluation omits precise editing details. The human study primarily evaluates rendered scenes rather than success on precise interactive tasks, and its small sample of 10 participants limits the robustness and representativeness of the findings.
  3. The evidence for multi-agent benefits is limited to GPT-4.1 without budget-matched controls. The single-agent ablation does not establish equal inference budgets or access to feedback and skill instructions. Gains may partly reflect additional resources, and their generality to other or stronger models is untested.
Questions for authors
  1. Could the authors compare the full system with a single agent given the same tools, skill instructions, and feedback opportunities under a matched model-call or token budget? Does the advantage persist with other or stronger models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.