- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
3DCodeVerse combines a cross-runtime executable 3D program corpus (CAD, Blender, URDF, GLSL, Three.js, OpenSCAD and others), a multi-agent harness with planning, execution, spatial inspection and visual feedback, and open-model fine-tuning on the corpus. It reports large execution-rate gains from SFT/DPO (e.g., 0% to 84.9% for a 9B model and 51.9% to 92.9% for a 27B model on 3DCodeBench), paired object-quality improvements for the complete workflow, and a 90% win rate against Claude Design in pairwise human evaluation when both use Claude Opus 5.5. The resource integration and conversion validation are potentially valuable; the manuscript, however, contains inconsistent headline results and compares workflows with very different generation budgets.
Critical strengths
S1. A broad executable 3D-code resource with concrete conversion validation. The collection spans CAD, Blender, URDF, GLSL and Three.js, and it validates conversions rather than only aggregating sources: mean DeepCAD voxel IoU is 0.985 and 6,878/6,882 Articraft conversions pass the stated geometry criteria. A cross-dialect, provenance-tracked executable corpus is a genuine resource contribution even where the underlying mechanisms are not new.
S2. Most consistent results show substantial gains, and limitations are disclosed. Execution improves markedly after training (0% to 84.9% for 9B and 51.9% to 92.9% for 27B on 3DCodeBench), and paired intervals support object-quality gains for the complete workflow. The appendix openly reports representation duplication, quality tiers, unmatched budgets, limited ablations, oracle state and negative results, which makes the claims auditable.
Critical weaknesses
W1. Headline results are internally inconsistent and cannot be audited from the paper. Table 2 reports 74.5% Three.js execution for 9B +SFT, whereas the corresponding 270K run in Table 20 reports 32.5%, and the other five columns of that row (84.9/90.3/98.5/86.0/56.0) match Table 2 exactly, so the 270K run is the intended configuration. With a single-seed evaluation on 40 tasks and binary execution outcomes, rates should move in 2.5-point steps, so 74.5% also needs an explanation of its denominator or aggregation. Table 3 reports OpenSCAD F=.416 for 27B SFT+DPO against .314 in Table 20 (the other columns agree), which changes the negative-transfer conclusion. The abstract's quality/executability wording also blurs units: the 2,028,286 export rows multiply captions, representations and shader passes and are not independent programs (the catalogue has 269,597 records including variants), and the mapping from each headline to its 60/30/100-prompt subset and run snapshot is incomplete. Tables should be regenerated from a single manifest of checkpoints, mixtures, prompt IDs, seeds, runtimes and per-example predictions.
W2. The comparisons do not isolate the benefit of the harness, and novelty is not positioned against the closest mechanisms. Native baselines generate in a single pass without tools or feedback; generation cost is about $0.035 versus $3.312 per task (roughly 95x, 3.312/0.035 = 94.6), and Claude Design usage is not measured. The evidence therefore shows that a more expensive complete workflow improves quality, not that it beats budget-matched execute-repair, best-of-N or tool-enabled agents; none of the paired artifact-quality ablations in Table 26 separates from zero, and the scene/graphics studies use only 7-11 paired prompts with a negative mean for Sonnet graphics. The 90% preference result needs raw prompt/rater counts, W/L/T, tie handling and prompt-bootstrap intervals. SceneCraft (planning, spatial constraints, visual critique), BlenderLLM (script data, SFT, filtered self-improvement), 3DCodeBench (API migration, geometric validation, multi-turn feedback) and VIGA (editable programs with visual feedback) already cover the core mechanisms, so the defensible contribution is the cross-dialect resource and validated conversion; the independently added programs/sources and a modern-API adaptation baseline are needed to show what remains beyond them.
Questions for authors
- Which denominators, runs or checkpoints explain the Table 2/3/20 discrepancies and the 74.5% Three.js rate with N=40? Can corrected tables and a unified manifest be provided?
- What are the raw evaluation units and W/L/T counts behind the 90% preference result, and which prompt subset and version underlie each headline?
- Does the complete harness outperform ordinary execute-repair, SceneCraft-style workflows or best-of-N under equal generation cost and tool access?
- Relative to 3DCodeBench, BlenderLLM and the source datasets, which independently added programs, conversions and training gains remain, and how much geometric improvement survives a modern-API baseline?