- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- Paper introduces a benchmark for evaluating multimodal AI agents on realistic 3D creation, reconstruction, editing tasks etc in Blender.
- Dataset shares 223 tasks across 9 categories and 16 real-world application scenarios, including architecture, product design, animation,etc etc. Tasks use text, image, video ect.
- Agents use Blender tools and Python code to iteratively construct, render, inspect, and refine 3D scenes. Final artifacts are evaluated using renddered views, program inpsection, task-specific binary rubrics, other visual-quality scores.
- Main results: Evaluates 13 frontier models, with GPT-6-Astra achieving the highest overall score (79.9), Opus-5 (73.0), GPT-5.6-Sol (67.7). However, even GPT-6-Astra fuly satsisfies all rubric criteria on only 42.2% of tasks.
- Key points shared: (a) Stronger performance does not necessarily require more tool calls or visual inspections. GPT-6-Astra outperforms Opus-5 and DeepSeek-V4.1-Flash while using substantially fewer output tokens and tool calls.
- Key points shared: (b) Agents struggle with 3D spatial coherence, physical constraints, fine-grained geometry, effective self-correction etc. They often produce visually convincing renders using 2D shortcuts or verify incomplete constraints, resulting in structurally incorrect 3D artifacts.
Critical strengths
- Evaluates realistic 3D creation workflows using 223 tasks across 9 categories, with text, image, video, and structured specification inputs covering real-world applications.
- Evaluation framework is neat with multi-view rendering, program scene inspection, task-specific binary rubrics, visual-quality scores to assess both visual.
- Extensive evaluation of multiple frontier models
- Identifies meaningful limitations such as view-dependent 2D shortcuts, physical inconsistencies, and ineffective self-verificatio etc.
Critical weaknesses
- Limited novelty: With the rapid development of agentic systems, the proposed pipeline largely combines existing components (tool-using agents, multi-view rendering, LLM-as-Judge, rubric eval) without introducing much new techniques or surprising research insights.
- More engineering than research contribution: The approach resembles a well-engineered software evaluation pipeline rather than a novel research methodology. Its technical contributions and broader scientific insights appear insufficiently substantial for a venue like ICLR.
- Limited benchmark scale and significance: The dataset contains only 223 tasks, with limitd cobveage across categories. Given the limited methodological novelty, the dataset's scale, diversity may not be sufficiently compelling to establish a major standalone benchmark contribution.
Questions for authors
Please try to address the weaknesses raised above and share your thoughts on what novel research insights the paper offers. There is no harm in developing a technical product, which could be equally impactful, or investing more effort in expanding the dataset to strengthen its value as a benchmark.
Final Summary for the Area Chair: Key Contributions, Recognized Strengths, and Our Rebuttal Evidence (1/2)
Dear Area Chair,
As the discussion period draws to a close, we would like to summarize the main highlights of BlenderWorld, the strengths already recognized by the reviewers, and the new evidence introduced during the rebuttal to assist your assessment of the research value and contribution of this work.
1. The work addresses an important evaluation gap already recognized by the reviewers.
BlenderWorld evaluates the complete capability of completing 3D modeling and delivering editable project files starting from multimodal references and production requirements. Its value lies not only in generating rendered images, but also in verifying whether the artifacts possess accurate geometry, structure, physical relationships, and delivery attributes.
This contribution was recognized from multiple perspectives in the initial reviews:
These assessments indicate a clear foundation of community value: the paper provides a reusable task suite and evaluation protocol that enable researchers to examine 3D delivery capabilities beyond image appearance. Our main experiments cover 223 tasks, 9 categories, 22 subcategories, 16 application scenarios, and 2,651 requirements, encompassing 2,899 full workflow runs across 13 agents.
2. The paper provides traceable research findings rather than just a performance ranking.
Combining output artifacts with execution traces allows us to explain precisely where models fail.
For example, representing a human character with a textured plane can produce a convincing submitted view, yet the character disappears from the side. This reveals the gap between satisfying a visual objective under a specific camera and constructing true 3D structure, and demonstrates that multi-view and native project inspection can uncover issues that remain hidden in a single image.
In another case, a model treated the combined height of 102 mm of wood and a 48 mm caster as satisfying a requirement for a 150 mm wooden leg. While the arithmetic itself was valid, the target of verification had drifted away from the actual task. This shows that improving self-verification requires preserving the exact meaning of constraints rather than simply increasing tool calls or check counts.
Requirement satisfaction and visual quality can also diverge. On Art and Animation, GPT-6-Astra achieves a rubric score of 89.6 compared to a visual quality score of 64.0; in the main experiments, the strongest agent satisfies all rubrics on only 42.2% of tasks. These results help distinguish incomplete delivery, structural errors, and weak visual quality, offering concrete problems for future research on constraint preservation, 3D consistency, and effective self-verification. Reviewers 5VmA, eRft, and TDQu all noted the value of our failure analysis in their listed strengths, and Reviewer B7z3 explicitly recognized the ability of native inspection to uncover hidden errors.