- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces 3D Code Factory, an automated framework for generating procedural 3D coding tasks with multimodal evaluation rubrics, and 3D Live Bench, a benchmark covering Three.js and Blender environments. The framework combines scene-state inspection and rendered-image evaluation to assess geometric correctness and visual quality. Experiments across multiple models and agent scaffolds reveal substantial gaps between execution success, rubric compliance, and visual quality. The authors also demonstrate automated task scaling and preliminary improvements from trajectory-based fine-tuning.
Critical strengths
-
Important problem: It addresses a meaningful challenge in evaluating procedural 3D code, where multiple valid implementations make reference-based evaluation unreliable.
-
Well-motivated framework: The paper combines scene-state and image-based verification through fact-anchored task and rubric generation.
-
Comprehensive agent comparison: The paper evaluates multiple models, rendering scaffolds, and 3D environments, revealing interesting performance and efficiency trade-offs.
-
Potential scalability: The paper demonstrates automated generation of 488 validated tasks and explores their use for agent fine-tuning.
Critical weaknesses
-
Questionable evaluation reliability: Despite claiming verifiable evaluation, the framework relies heavily on LLM judgments, even for geometric constraints, without sufficient independent ground-truth validation.
-
Limited benchmark representativeness: The main benchmark contains only 36 tasks, while the 488 extended tasks are primarily validated through automated LLM-based checks, with limited evidence of generalizability.
-
Insufficient functional verification: Static scene-state inspection and rendered images cannot reliably establish correct dynamic behavior, mechanical functionality, or temporal consistency.
-
Unconvincing fine-tuning improvements: Despite increased execution rates, absolute rubric compliance after fine-tuning remains extremely low, with evaluations on only 20 held-out tasks.
Questions for authors
-
How does the multimodal evaluator compare against independently human-verified ground truth for geometric and structural criteria? Could deterministic geometric checks replace some LLM-based judgments?
-
How representative are the 36 core tasks, and are the reported model differences statistically significant?
-
How does the framework verify actual dynamic or mechanical functionality rather than static geometric plausibility? Could temporal simulation, collision testing, or animation-based verification be incorporated?
4 comments
-
W/Q2a (continued): How representative are the 36 core tasks, and are the reported model differences statistically significant?
-
We evaluated all 17 configurations on 27 added tasks, expanding the benchmark to 63 tasks across 31 Structure, 17 Dynamics, and 15 Material tasks. Model separations remain consistent (RC and VQ with 95% bootstrap confidence intervals in brackets):
Model Scaffold RC [95% CI] VQ [95% CI] GPT-6-Astra Codex .808 [.760, .851] .665 [.648, .682] GPT-5.6-Sol Codex .749 [.700, .795] .578 [.556, .599] GPT-5.6-Sol OpenClaw .650 [.580, .715] .545 [.507, .580] DeepSeek-V4.1-Flash OpenClaw .458 [.375, .540] .397 [.334, .457] Astra exceeds Sol in VQ under Codex (adjusted p = .0013). Sol exceeds DeepSeek-V4.1-Flash under OpenClaw (adjusted p = .0051 for RC and .0026 for VQ).
-
-
W/Q2b: The 488 extended tasks are primarily validated through automated LLM-based checks, with limited evidence of generalizability.
-
Trajectories from synthesized tasks improve performance on tasks excluded from training.
The 488 extended tasks support task expansion and trajectory collection. Cross-model comparisons use the 63 expert-reviewed core tasks. Running Qwen3.5-122B-A10B-FP8 across these 488 tasks in engine sandboxes yields 1,333 nonempty trajectories and 715 valid deliverables. Training solely on top-quartile trajectories yields consistent SFT gains across 8B, 9B, and 35B models under both scaffolds on the disjoint 36 tasks (average RC gain of 2.10×).
-
-
W/Q3: How does the framework verify actual dynamic or mechanical functionality rather than static geometric plausibility? Could temporal simulation, collision testing, or animation-based verification be incorporated?
-
The Dynamics domain primarily evaluates procedural construction of motion-related scenes and articulated mechanisms in specified operating states. Its rubrics inspect mounting relationships, component poses, reach, contact, and clearances, with analytic checks across configurations where specified, such as gangway slope and roller travel across fluctuating water levels.
While multi-frame simulation, collision tracking, and rigid-body physics can indeed be executed via the underlying Blender and Three.js engines, our primary focus is on whether agents can procedurally build valid kinematic mechanisms in code. Establishing correct attachment hierarchies, joint axes, and functional clearances is the foundational prerequisite before simulation can take place, and this assembly capability remains the primary bottleneck for current agents. Verifying mechanisms at designated operational states also provides a lightweight and scalable benchmark without the heavy runtime overhead or instability of physics solvers. Dynamic simulation is a natural and technically feasible extension that can be layered on top as procedural agents mature.
-
-
W/Q4: Despite increased execution rates, absolute rubric compliance after fine-tuning remains extremely low, with evaluations on only 20 held-out tasks.
-
SFT improves all three metrics across the three models and both scaffolds.
In initial experiments, fine-tuning text-only Qwen3-8B on 20 held-out tasks yielded low compliance because it lacked vision and had weak 3D modeling skills, frequently causing syntax and code errors. We therefore re-ran fine-tuning using vision-capable Qwen3-VL-8B alongside Qwen3.5-9B and Qwen3.6-35B, evaluating under Codex and OpenCode on the 36 tasks (task IDs disjoint from training).
-
From 715 valid deliverables on the 488 tasks, selecting the top VQ quartile per runtime and scaffold yields 170 training trajectories (7,111 turns) and 37 for validation. We fine-tuned Qwen3-VL-8B (LoRA), Qwen3.5-9B (full-parameter), and Qwen3.6-35B-A3B (LoRA) for one epoch (888 steps).
-
Average gains in RC (2.10×) and VQ (1.71×) exceed the gain in Exec (1.49×). RC after SFT ranges from .062 to .188 on 8B and 9B and reaches .449 on 35B OpenCode:
Model Scaffold Setting Exec (%) ↑ RC ↑ VQ ↑ 8B Codex Base 19.4 .020 .034 8B Codex + SFT 50.0 .111 .076 8B OpenCode Base 27.8 .035 .044 8B OpenCode + SFT 41.7 .062 .102 9B Codex Base 63.9 .099 .114 9B Codex + SFT 75.0 .154 .166 9B OpenCode Base 58.3 .122 .102 9B OpenCode + SFT 63.9 .188 .142 35B Codex Base 69.4 .177 .173 35B Codex + SFT 94.4 .288 .231 35B OpenCode Base 61.1 .202 .175 35B OpenCode + SFT 97.2 .449 .311 Average gain - - 1.49× 2.10× 1.71×
-
I appreciate the authors' response that addressed some of my concerns partially. Some other concerns still remain, e.g., the evaluation reliability and the limited improvements. Hence, I will retain my rating unchaged.
Thank you for your follow-up. Regarding the two concerns you highlighted, we would greatly appreciate a brief clarification on what specific gaps remain in light of the new evidence in our rebuttal:
-
Evaluation reliability: In our rebuttal, we explained why hardcoded deterministic scripts fail under diverse agent naming, mesh decompositions, and scene hierarchies (e.g., folding desk and orrery), necessitating LLM interpretation over extracted geometric State. More importantly, our independent human validation (8 annotators, 446 labels) directly confirms its reliability: 90.5% agreement (κ = 0.729) on geometric State criteria and 95.9% (κ = 0.851) on non-tie criteria, alongside high cross-judge rank stability (Spearman ρ = 0.929–0.976).
-
Fine-tuning improvements: Addressing the initial text-only 8B result on 20 tasks, our new SFT experiments on 36 disjoint tasks across three vision-language models (8B/9B/35B) and two scaffolds yield an average 2.10× RC gain and 1.71× VQ gain: both substantially exceeding the 1.49× Exec gain. On Qwen3.6-35B-A3B (OpenCode), RC more than doubles from 0.202 to 0.449 (with Exec rising from 61.1% to 97.2%) using only 170 trajectories trained for one epoch, demonstrating clear generalization from our synthesized tasks.
We would be grateful for any concrete pointers before the discussion period closes.
W/Q1: How does the multimodal evaluator compare against independently human-verified ground truth for geometric and structural criteria? Could deterministic geometric checks replace some LLM-based judgments?
For geometric and structural criteria (our State channel), eight annotators provided 199 Pass/Fail judgments across 108 criteria, achieving 90.5% agreement (κ=.729) with the three-judge majority vote (Luna, Gemini, DeepSeek), compared to 86.6% (κ=.566) on perceptual Image criteria and 88.3% (κ=.645) overall:
Agreement reaches 95.9% (κ=.851) on criteria without human ties. Image disagreements stem from perceptual judgments under lighting or partial occlusion, while State discrepancies reflect coordinate conventions (Z-up vs. Y-up in Blender) or concept-to-entity mapping.
Deterministic Python checks readily compute numeric metrics once target entities and reference points are established, but hardcoded checking scripts struggle when agents adopt diverse object naming, mesh decompositions, and coordinate conventions. State extracts raw geometric data, while rubrics specify functional parts and relations.
Geometric decomposition: In the folding desk, for example, the rubric measures the distance between cabinet-side and desk-side pivot centers. Different agents decompose the support arm into separate webs, flanges, and eyelets, while DeepSeek places object origins at (0,0,0) with coordinates stored in mesh vertices; naively querying
object.locationor a bounding box measures the wrong quantity.Scene hierarchies: Similarly, in the orrery, Gemini nests
moon_arm_pivothierarchically while Luna places parts flat under the scene root, requiring the verifier to interpret assembly relations across diverse scene structures.Because rigid programmatic rules struggle to accommodate such implementation diversity, we use the LLM to interpret semantic components, structural assemblies, and parameters from State, and evaluate whether the rubric requirements are met.
W/Q2a: How representative are the 36 core tasks, and are the reported model differences statistically significant?
We expanded the benchmark from 36 to 63 tasks by adding 27 tasks and evaluated all 17 configurations on these additions.
Procedural 3D tasks are inherently composite. Rather than testing a single scalar property, each task couples spatial, mechanical, and visual constraints, verified by an average of ~29 fine-grained rubric criteria. Across the 63 tasks, 1,829 criteria (684 state, 1,145 image) systematically cover eight core modeling capabilities: spatial metric reasoning (60/63 tasks), layout & circulation (57/63), terrain & volumes (52/63), kinematics & mechanisms (48/63), hierarchy & attachment (46/63), materials & surfaces (60/63), lighting & atmosphere (40/63), and composition & legibility (46/63). Complementing functional rubric compliance, Visual Quality (VQ) independently evaluates rendered perceptual quality across nine dimensions covering geometry, scene completeness, materials, lighting, composition, detail, visual coherence, aesthetics, and image quality.
Under one-hour budgets, the original 36 tasks cleanly separate agents from 8B models to frontier models:
Both the 8B–122B and 122B–Astra differences are statistically significant for RC and VQ (Bonferroni-adjusted p = .0014 under Codex).