VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?

Constructing an interactive 3D open world from a user query is important for gaming, simulation, and embodied AI. However, existing methods are primarily evaluated on idealized simple queries and lack an open-source framework, making it difficult to systematically analyze the bottlenecks and how can overcome them. To this end, we propose VIBEWORLDING, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first build VWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we develop VIBEWORLDING-GYM, a joint multimodal RL post-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, and (2) a rubric-based verifier that combines physical feasibility (e.g., collision) and intent fulfillment verification, supporting both fair model evaluation and scalable multimodal RL reward service. Our experiments show that current frontier MLLMs are far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-source MLLMs to become comparable to closed-source frontiers: our VibeWorlder-8B is comparable to frontier MLLMs, while our flagship VibeWorlder-30B-A3B attains a Pass@1 comparable to the strongest evaluated models. We release our data, code, and models to facilitate research for end-to-end 3D world construction: https://anonymous.4open.science/r/VibeWorlding-Gym/.