acceptodds
Under review as a conference paper at ICLR 2027

VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

Abstract

Constructing an interactive 3D open world from a user query is important for gaming, simulation, and embodied AI. However, existing methods are primarily evaluated on idealized simple queries and lack an open-source framework, making it difficult to systematically analyze the bottlenecks and how can overcome them. To this end, we propose VIBEWORLDING, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first build VWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we develop VIBEWORLDING-GYM, a joint multimodal RL post-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, and (2) a rubric-based verifier that combines physical feasibility (e.g., collision) and intent fulfillment verification, supporting both fair model evaluation and scalable multimodal RL reward service. Our experiments show that current frontier MLLMs are far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-source MLLMs to become comparable to closed-source frontiers: our VibeWorlder-8B is comparable to frontier MLLMs, while our flagship VibeWorlder-30B-A3B attains a Pass@1 comparable to the strongest evaluated models. We release our data, code, and models to facilitate research for end-to-end 3D world construction: https://anonymous.4open.science/r/VibeWorlding-Gym/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal agents, 3D agent

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_jkCLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VibeWorlding, a multimodal agent framework for constructing 3D open worlds. The authors introduce VWE-Bench, a benchmark comprising 2,616 3D assets, 323 seed worlds, and 6,828 queries. They further develop VibeWorlding-Gym, which integrates a unified 3D tool environment, a dual-constraint verifier, and SFT and GRPO training to improve agents' 3D scene construction capabilities. Experimental results demonstrate that the proposed approach significantly improves the performance of open-source MLLMs, enabling them to achieve performance comparable to several frontier closed-source models.

Critical strengths
  1. The research problem is meaningful. This work integrates open-ended 3D scene construction and multi-turn editing into a unified framework. The proposed benchmark and interactive environment provide a valuable platform for systematically evaluating the 3D understanding and manipulation capabilities of multimodal agents.

  2. The experimental evaluation is relatively comprehensive. The authors compare multiple frontier MLLMs and conduct ablation studies, human evaluations, and generalization experiments to validate the effectiveness of their approach. The performance improvements achieved through reinforcement learning are substantial.

Critical weaknesses
  1. The methodological novelty could be further clarified. The proposed approach mainly combines existing techniques, including SFT, GRPO, and multimodal tool use, and shares certain similarities with related works such as SceneReVis. The authors should better highlight the key technical differences and unique contributions of their method.

  2. The evaluation reliability has certain limitations. Physical feasibility is primarily assessed using AABB-based approximate collision detection, which may not accurately capture complex 3D collisions. Additionally, Verified and Unverified queries employ different scoring schemes, and the rationale for combining them into the Overall Pass@1 metric requires further justification.

Questions for authors
  1. Compared with existing reinforcement learning approaches for 3D scene construction, such as SceneReVis, what are the key innovations of this work in terms of training strategies and multimodal feedback mechanisms?

  2. Would the physical feasibility evaluation results remain consistent if more precise 3D collision detection methods were employed? Additionally, could the authors further justify the combination of different scoring schemes for Verified and Unverified queries into the Overall Pass@1 metric?

1 comment
Reviewer_jkCLOpenReview ↗

Minor Suggestions for Evaluation Reporting

Thank you again for the revisions. After reviewing the updated manuscript, I have two minor suggestions that could further improve the presentation and experimental analysis.

  1. In Table 13, bootstrap confidence intervals are provided for most models, but the interval for VibeWorlder-30B-A3B is still missing. It would be helpful to report its bootstrap interval and, if possible, a paired comparison with GPT-5.5.
  2. Since the evaluation set contains substantially more refinement queries than construction queries, the Overall Mean Reward may obscure differences between these two tasks. Reporting separate aggregate results for construction and refinement would provide a clearer picture of the model's strengths and limitations.

Additionally, the Pass@1 label in Figure 5 could be updated to reflect the revised metric definition. These are minor suggestions and do not affect my current recommendation of Weak Acceptance.

Reviewer_YNsqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VibeWorlding introduces a benchmark and training environment for agents that construct and edit 3D scenes using asset tools, textual scene maps, and rendered views. The authors combine supervised fine-tuning with GRPO and evaluate instruction following and physical plausibility through geometric checks and a multimodal judge. Experiments report substantial post-training gains over the base models.

Critical strengths
  1. The unified environment brings asset retrieval, scene editing, rendering, and evaluation into one workflow. This is a useful foundation for systematic training and comparison of scene-construction agents.

  2. The verifier explicitly penalizes unrequested edits. This addresses a practical failure mode that a task-completion score alone could overlook.

Critical weaknesses
  1. The evaluation definitions and reported scores need to be reconciled before the headline results can be assessed reliably.

    1. Table 3’s Overall scores do not follow from the reported Verified and Unverified scores under the stated 66/188-query split. GPT-5.5’s component scores imply 56.3%, whereas Overall is reported as 57.3%. More directly, SceneAssistant’s Overall of 13.8% exceeds both its Verified score of 12.9% and its Unverified score of 12.8%. These discrepancies exceed rounding error. The exact aggregation rule and per-query scores are needed to interpret the reported comparisons.

    2. The reward description also differs between the main text and appendix. Section 4.3 describes Unverified rewards as binary, while Appendix A.4.3 awards partial credit to unsuccessful construction episodes. This shaping term could help explain the aggregate-score discrepancies if it also enters evaluation, but that would differ from the hard-pass definition in Section 5.2. The authors should clearly separate training rewards from evaluation scores and specify which rule produced each reported result.

    3. The physical-feasibility scores require stronger independent validation. Appendix A.4.2 states that the main collision checker quantizes asset extents to whole metres and does not recompute them from the current orientation. Table 7 provides useful threshold sensitivity analysis, but does not establish agreement with actual mesh intersections or support relations. The human study does not independently validate this aspect either: Table 10 explicitly states that its Physical column mirrors the automatic collision reference.

  2. The methodological novelty is not clearly established against the closest prior work.

    Section 7 describes SceneReVis as relying solely on textual scene states. However, Sections 3.1–3.2 of SceneReVis explicitly include rendered visual observations and use them during multi-turn GRPO. Joint textual and visual feedback therefore does not establish the claimed distinction from that work. The authors should correct this characterization and explain what is new in their task formulation, benchmark, verifier, or training design. This matters because the current positioning makes it difficult to assess the methodological contribution beyond the useful integration of existing components. SceneReVis

Questions for authors
  1. Evaluation scores and rewards

    1. How is Overall in Table 3 computed from the 66 verified and 188 unverified queries? For example, GPT-5.5’s component scores imply 56.3%, whereas Overall is reported as 57.3%. Please provide the aggregation rule and per-query scores.

    2. What reward was used for unsuccessful construction episodes during RL? Section 4.3 describes binary rewards, while Appendix A.4.3 introduces partial credit. Does this partial credit also enter the reported evaluation scores?

    3. Are the alternative-judge results in Table 13 averaged over repeated judgments? Please clarify the denominators and averaging procedure, since several reported percentages cannot arise from a single binary evaluation of 188 rollouts.

  2. Physical validity

    1. Beyond the study in Table 10, do you have independent validation of the collision and support checks against actual asset geometry or human annotations made without seeing the automatic checker’s output?
  3. Methodological contribution

    1. What is the main methodological distinction from SceneReVis, given that its Sections 3.1–3.2 already describe rendered visual observations during multi-turn GRPO? SceneReVis
  4. Visual results of the generated 3D scenes

    1. Which models and checkpoints produced the construction and refinement examples in Figures 6 and 7?

    2. From the existing evaluation runs, could you show a few matched examples of the actual 3D scenes produced by the base, SFT, and RL versions of the same backbone, alongside a strong frontier baseline on the same requests? Please use consistent viewpoints and mark the requested changes and any remaining errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.