acceptodds
Under review as a conference paper at ICLR 2027

BlenderWorld: Benchmarking Visual Agents for 3D Creation from Real-World References

Abstract

Recent advances in multimodal large language models (MLLMs) have enabled them to act as autonomous agents, creating and editing 3D models via code or direct software manipulation. In practical workflows, these models must infer object shapes, materials, spatial layouts, and lighting from diverse references while strictly following instructions to reconstruct, modify, or design 3D content. However, existing benchmarks provide limited coverage of such open-ended requests across real-world scenarios. To bridge this gap, we introduce BlenderWorld, the first comprehensive benchmark dedicated to evaluating frontier models on realistic 3D creation tasks. BlenderWorld comprises 223 tasks spanning architecture, products, natural environments, and scientific visualization, driven by text, image, video, and structured specification inputs. To rigorously assess these complex deliverables, we establish a unified evaluation protocol: we render the submitted Blender projects from multiple viewpoints and evaluate them against 2,651 task-specific binary rubrics alongside holistic visual quality scores. Benchmarking thirteen frontier models reveals that GPT-6-Astra achieves the highest overall fused score of 79.9, followed by Opus-5 (73.0) and GPT-5.6-Sol (67.7). Ultimately, our findings reveal a critical bottleneck: while current models can parse broad instructions, they still profoundly struggle with fine-grained 3D spatial coherence and accurate structural reproduction in Blender, leaving substantial headroom for future 3D-native agentics.

Primary area:
datasets and benchmarks
Keywords:
Visual Agents, Multimodal Large Language Models, 3D Modeling

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_TDQuOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  • Paper introduces a benchmark for evaluating multimodal AI agents on realistic 3D creation, reconstruction, editing tasks etc in Blender.
  • Dataset shares 223 tasks across 9 categories and 16 real-world application scenarios, including architecture, product design, animation,etc etc. Tasks use text, image, video ect.
  • Agents use Blender tools and Python code to iteratively construct, render, inspect, and refine 3D scenes. Final artifacts are evaluated using renddered views, program inpsection, task-specific binary rubrics, other visual-quality scores.
  • Main results: Evaluates 13 frontier models, with GPT-6-Astra achieving the highest overall score (79.9), Opus-5 (73.0), GPT-5.6-Sol (67.7). However, even GPT-6-Astra fuly satsisfies all rubric criteria on only 42.2% of tasks.
  • Key points shared: (a) Stronger performance does not necessarily require more tool calls or visual inspections. GPT-6-Astra outperforms Opus-5 and DeepSeek-V4.1-Flash while using substantially fewer output tokens and tool calls.
  • Key points shared: (b) Agents struggle with 3D spatial coherence, physical constraints, fine-grained geometry, effective self-correction etc. They often produce visually convincing renders using 2D shortcuts or verify incomplete constraints, resulting in structurally incorrect 3D artifacts.
Critical strengths
  • Evaluates realistic 3D creation workflows using 223 tasks across 9 categories, with text, image, video, and structured specification inputs covering real-world applications.
  • Evaluation framework is neat with multi-view rendering, program scene inspection, task-specific binary rubrics, visual-quality scores to assess both visual.
  • Extensive evaluation of multiple frontier models
  • Identifies meaningful limitations such as view-dependent 2D shortcuts, physical inconsistencies, and ineffective self-verificatio etc.
Critical weaknesses
  • Limited novelty: With the rapid development of agentic systems, the proposed pipeline largely combines existing components (tool-using agents, multi-view rendering, LLM-as-Judge, rubric eval) without introducing much new techniques or surprising research insights.
  • More engineering than research contribution: The approach resembles a well-engineered software evaluation pipeline rather than a novel research methodology. Its technical contributions and broader scientific insights appear insufficiently substantial for a venue like ICLR.
  • Limited benchmark scale and significance: The dataset contains only 223 tasks, with limitd cobveage across categories. Given the limited methodological novelty, the dataset's scale, diversity may not be sufficiently compelling to establish a major standalone benchmark contribution.
Questions for authors

Please try to address the weaknesses raised above and share your thoughts on what novel research insights the paper offers. There is no harm in developing a technical product, which could be equally impactful, or investing more effort in expanding the dataset to strengthen its value as a benchmark.

Reviewer_B7z3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BlenderWorld benchmarks agents on 223 Blender tasks involving reconstruction, modification, and design from text, visual references, and specifications. Agents must deliver editable projects and requested media. Evaluation combines task-conditioned renders, project inspection, task-specific requirements, and visual-quality ratings. Experiments with thirteen agents compare performance and analyze structural and visual failures.

Critical strengths
  1. The benchmark covers a useful range of tasks and production requirements. Requiring editable projects alongside rendered outputs makes it a valuable complement to benchmarks focused on individual objects or visual similarity.
  2. The evaluation addresses errors that appearance alone can conceal. For example, native inspection catches an incorrect timber length that the agent's own check misses. The human comparisons and weighting analyses also support the usefulness of the overall scores.
Critical weaknesses

My main concern is the validation of native-project evaluation. The human study in Appendix E.1 measures overall artifact preference, rather than whether individual structural requirements are judged correctly (relative vs absolute). The examples demonstrate useful capabilities, but their reliability across varied requirements remains unclear. Since these checks are central to the benchmark, this limits my confidence in the reported complete-success rates and structural diagnoses.

Questions for authors
  1. Is there a representative audit of individual requirement judgments, including false passes and false failures?
  2. What stopping rules and time, turn, budget, or token limits governed generation for each system?
Reviewer_eRftOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BLENDERWORLD, a benchmark for evaluating Multimodal LLM agents on realistic 3D tasks combining reconstruction and reference based design across 9 tasks and 16 categories using blender as existing benchmarks are limited . Benchmark comprises tasks categories and application scenarios covering architecture, products, natural environments, vehicles, scientific visualization and more. Tasks are driven by text, image, video and structured specs like real world briefs. Generation agents iteratively build editable projects while evaluator agent select task relevant views and inspect scene structure, task specific binary rubrics asses compliance with each brief while visual quality scores access rendered result. The eval protocol is a key contribution with evidence based routed dual judge system with Agent as Judge, VLM as Judge, binary rubric criteria and 8 dimension visual quality scores with a fused score.13 frontier models are evaluated on the benchmark and the paper concludes GPT-6 Astra achieves the highest score while identifying that only ~42% tasks passes on 100% of rubric criteria. Paper also provides failure analysis and reveals that current agents relay on 2D visual shortcuts and struggle with fine grained 3D spatial coherence.

Critical strengths

BLENDERWORLD fills a real gap and timely as most existing benchmarks are limited and focus on narrow tasks. The paper grounds its motivation in latest development as MLLMs+ harness evolves into autonomous agents working on long running tasks

Eval dataset is robust with broad category coverage and application grounded spec grounded in real world eval. ex emphasis on structural relationship for construction brief

Multi dimensional eval protocol is robust with evidence routed rubric judging as key contribution

  • Agent as judge provides a programatic inspection of blender scene state including editability
  • VLM as judge with rendered views evaluating visual inspection from rendered views

This addresses the fundamental problem of evaluating 3D outputs without ground truth.The agent as judge bypassing visual shortcuts is particularly valuable

The paper provides a very detailed rubric design - Dual track validation, multi model cross checking with quorum of judges, human review correlation and validation and source linked requirements with binary fail/pass condition

In addition to quantitative metrics paper provided strong qualitative failure analysis

  • missing dynamic interaction - agents satisfy object presence but missed on physics
  • view point inconsistency - people disappearing from side view identifying 2D visual shortcuts
  • Fine grained geometric distortion - lion cub with finger like paws
  • Contact constraint violation - Floating objects
Critical weaknesses

Judge validity and reliability

  • Significant concern is the validity of VLM as judge and dimension scoring with modest judge agreement with overall pairwise agreement ~82 and Pearson with human pref of ).714 and spearman 0.74 and on small gap cases(<10) agreement drops to 68. while these are reasonable, judge is used for primary scoring for all binary rubrics and 8 dimension scores which need further discussion

Dimension scores are subjective with VLM as judge, even with anchors inter judge variability across outputs is substantial. Paper should discuss variance across dimension score across multiple runs whether 1-10 scale is appropriate(vs likert scale or pair wise comparison). How are dimension scores calibrated against task difficulty.

Judge bias Risk

  • GPT-5.6-terra is used for AaR rendering and rubric judging. This creates potential bias where the judge may favor outpouts that align with its own rendering preference or training distribution. The paper should explicitly address this

GPT family uses codex CLI where as other models use claude code. This might introduce potential degradation as models and harness are co-designed. Does this affect tool calling behavior, would the results be different across different harnesses?

No ablation showing impact of:

  • Removing agent as judge
  • changing the no. of views
  • removing spec files
  • use different types

without these its difficult to asses score variance is due to benchmark design or model capability gap

Questions for authors
  • What is the inter-judge variability across dimension score? how much does model ranking change with different models as judge?
  • Have you tested with different harness for GPT and vice versa? How did you ensure that harness was not a factor in GPT astra performing the best?
  • With single rollouts, how confident are you in the top 3 rankings? are the rankings statistically significant?
  • how are dimensions calibrated across task types?
  • What was the reasoning effort used across the models and does that impact the eval results. Are the all the models have same defaults for reasoning? Paper reports mean turns, tokens and tools but does not discuss how reasoning effort impacts it?
Reviewer_5VmAOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a more practical and broad benchmark for settings where a multimodal large language model (MLLM) autonomously creates or manipulates 3D models. Existing benchmarks have been proposed with a narrow scope, each targeting a specific task. In contrast, BlenderWorld organizes 9 categories and 22 subcategories, providing tasks that measure how well MLLMs act across diverse scenarios, and proposes evaluating them under a unified evaluation protocol that includes task-conditioned multi-view rendering and task-specific binary rubrics. The authors evaluate recent MLLMs on BlenderWorld and further analyze current failure modes in 3D modeling, such as reliance on 2D visual shortcuts and failure to account for physical factors.

Critical strengths

S1. The paper motivates the importance and necessity of the benchmark well, and I agree with it. It is a good benchmark for comprehensively evaluating whether a model can create and edit 3D models well.

S2. Beyond a simple performance comparison, the in-depth analysis of where current MLLMs struggle was good.

Critical weaknesses

W1. I think the various analyses the authors provide are useful and good. However, isn’t it possible that the cause of failure lies in how the task is conveyed rather than in the model itself? For example, if the brief were given in more detail, or if the parts that require fine-grained 3D structure were specified explicitly in the spec, a good portion of these failures might be resolved. It seems necessary to separate “the agent model lacks the capability” from “the task was not conveyed in enough detail for the model to carry it out.”

W2. The entire evaluation pipeline relies on a single judge model: GPT-5.6-terra handles view selection, VLM-as-Judge (rubric and dimension scores), and Agent-as-Judge. The reported alignment with human preferences suggests it is a reasonable choice. Still, I am a bit concerned about self-preference bias, since some of the evaluated agents come from the same model family as the judge. It would help to know whether the rankings hold up under a different judge, for example a Claude-family model. Would the authors be able to re-run part of the evaluation with one or two alternative judges and report how consistent the rankings are?

Minor Question

Q1. Given the timing, there seems to be no reason to include GPT-6 Astra while excluding Claude Fable. If there is a particular reason, please justify it.

Questions for authors

See weaknesses above.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Final Summary for the Area Chair: Key Contributions, Recognized Strengths, and Our Rebuttal Evidence (1/2)

Dear Area Chair,

As the discussion period draws to a close, we would like to summarize the main highlights of BlenderWorld, the strengths already recognized by the reviewers, and the new evidence introduced during the rebuttal to assist your assessment of the research value and contribution of this work.

1. The work addresses an important evaluation gap already recognized by the reviewers.

BlenderWorld evaluates the complete capability of completing 3D modeling and delivering editable project files starting from multimodal references and production requirements. Its value lies not only in generating rendered images, but also in verifying whether the artifacts possess accurate geometry, structure, physical relationships, and delivery attributes.

This contribution was recognized from multiple perspectives in the initial reviews:

  • Reviewer 5VmA affirmed the importance and necessity of the benchmark, noting that it is well suited for a comprehensive evaluation of 3D creation and editing capabilities.
  • Reviewer eRft explicitly agreed that the work fills a genuine gap and commended the per-requirement evidence routing and native project inspection, particularly for their ability to expose visual shortcuts.
  • Reviewer B7z3 recognized the coverage of tasks and production requirements, noting that requiring both editable project files and rendered outputs is a valuable complement to single-object or visual similarity evaluations.
  • Reviewer TDQu highlighted the realistic 3D workflows, multimodal inputs, evaluation framework, multi-model experiments, and meaningful failure modes.

These assessments indicate a clear foundation of community value: the paper provides a reusable task suite and evaluation protocol that enable researchers to examine 3D delivery capabilities beyond image appearance. Our main experiments cover 223 tasks, 9 categories, 22 subcategories, 16 application scenarios, and 2,651 requirements, encompassing 2,899 full workflow runs across 13 agents.

2. The paper provides traceable research findings rather than just a performance ranking.

Combining output artifacts with execution traces allows us to explain precisely where models fail.

For example, representing a human character with a textured plane can produce a convincing submitted view, yet the character disappears from the side. This reveals the gap between satisfying a visual objective under a specific camera and constructing true 3D structure, and demonstrates that multi-view and native project inspection can uncover issues that remain hidden in a single image.

In another case, a model treated the combined height of 102 mm of wood and a 48 mm caster as satisfying a requirement for a 150 mm wooden leg. While the arithmetic itself was valid, the target of verification had drifted away from the actual task. This shows that improving self-verification requires preserving the exact meaning of constraints rather than simply increasing tool calls or check counts.

Requirement satisfaction and visual quality can also diverge. On Art and Animation, GPT-6-Astra achieves a rubric score of 89.6 compared to a visual quality score of 64.0; in the main experiments, the strongest agent satisfies all rubrics on only 42.2% of tasks. These results help distinguish incomplete delivery, structural errors, and weak visual quality, offering concrete problems for future research on constraint preservation, 3D consistency, and effective self-verification. Reviewers 5VmA, eRft, and TDQu all noted the value of our failure analysis in their listed strengths, and Reviewer B7z3 explicitly recognized the ability of native inspection to uncover hidden errors.

AuthorsOpenReview ↗

Final Summary for the Area Chair: Key Contributions, Recognized Strengths, and Our Rebuttal Evidence (2/2)

3. The rebuttal substantially strengthens the evidence for evaluation reliability and our main results.

Compared with the initial submission, we added the following verification experiments to address key concerns:

Initial ConcernRebuttal AdditionKey Takeaway
Does overall human preference agreement ensure that individual structural requirements are judged accurately?Three human auditors inspected requirement judgments across 50 GPT-6-Astra outputs; false passes and false rejections accounted for 1.7% and 1.2% of all audited judgments, respectively.Reliability verification is grounded at the level of specific requirements rather than relying solely on overall preference.
Does the leading ranking depend on a GPT family judge?We re-evaluated all 13 agents using an Opus-4.8 judge; the top five order remained unchanged, with Spearman 0.984 and Kendall 0.949.Under fixed evidence, the primary ranking is supported by an alternative judge; shifts in absolute scores and changes in rank order should be evaluated separately.
Are Dimension scores prone to random fluctuation?We scored the same set of 223 outputs three times, yielding means of 72.42, 72.37, and 72.43, with a range of 0.06.Provides direct evidence for the repeatability of aggregate scores on fixed artifacts.
Does the strong performance of GPT-6-Astra appear only under Codex?On the same carefully sampled 50-task subset, Claude Code achieved 85.36 compared with 79.60 under the original Codex setup.Strong performance persists under a different agent scaffold, supporting cross-framework consistency.
Do reasoning settings affect the results?With low reasoning effort, GPT-6-Astra scored 76.43, still outperforming the Opus-5 baseline of 73.17 on the same subset.Configuration changes can affect scores, but reducing reasoning effort did not eliminate the strong performance of GPT-6-Astra.

We also clarified the shared scoring anchors, maximum available reasoning effort, the 8-hour generation limit, and other stopping rules, and explained which task information and verification targets must be preserved across component ablations.

4. Reviewer follow-up confirms that key concerns have been effectively addressed.

In follow-up discussion, Reviewer 5VmA commended our trace analysis, explicitly stated that W2 and Q1 are resolved, noted that the response answers most of their questions, and maintained a positive rating.

As of our most recent check on November 19, Reviewers B7z3, eRft, and TDQu have not yet posted follow-up comments, so we do not yet know their assessment of the newly added evidence.

We hope that the realistic and valuable evaluation setting, the research findings already recognized by the reviewers, and the direct validation and diagnostic experiments added during the rebuttal provide strong grounds for considering the paper for acceptance. We respectfully ask that the final decision weigh the initial reviews together with the fuller body of evidence presented in the rebuttal.

Thank you very much for your time and effort in coordinating the reviews, guiding the discussion, and completing the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.