acceptodds
Desk rejected at ICLR 2027

3D Code Factory: Scaling Verifiable 3D Tasks for Coding Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent rapid progress in coding agents has increased the demand for scalable task generation to advance agent capabilities, which requires diverse, executable tasks equipped with reliable automated verifiers. For procedural 3D modeling, fixed-reference comparisons can penalize semantically valid alternative implementations, while rendered images alone cannot fully verify spatial relationships and precise measurements. To address this problem, we introduce 3D Code Factory, an automated framework that synthesizes executable 3D coding tasks paired with multimodal evaluation rubrics from seed scenarios, decomposing verification into scene-state checks for spatial relations and rendered-image checks for visual appearance. Building on this framework, we construct 3D Live Bench to evaluate coding agents on 36 core tasks in Three.js and Blender under different scaffolds, assessing deliverables across three metrics: execution success, rubric compliance, and visual quality. Extensive evaluations reveal substantial room for improvement in complex 3D modeling: despite high program execution rates, agents struggle to satisfy coupled task requirements, with peak visual quality and rubric compliance achieved by different model–scaffold configurations. During iterative refinement, localized repairs often introduce regressions, invalidating or neglecting previously satisfied criteria. Furthermore, scaffold design fundamentally alters how base-model capabilities translate into task completion, while prolonged edit–render loops substantially inflate token expenditure with diminishing returns.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, 3D generation, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xEsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces 3D Code Factory, an automated framework for generating procedural 3D coding tasks with multimodal evaluation rubrics, and 3D Live Bench, a benchmark covering Three.js and Blender environments. The framework combines scene-state inspection and rendered-image evaluation to assess geometric correctness and visual quality. Experiments across multiple models and agent scaffolds reveal substantial gaps between execution success, rubric compliance, and visual quality. The authors also demonstrate automated task scaling and preliminary improvements from trajectory-based fine-tuning.

Critical strengths
  • Important problem: It addresses a meaningful challenge in evaluating procedural 3D code, where multiple valid implementations make reference-based evaluation unreliable.

  • Well-motivated framework: The paper combines scene-state and image-based verification through fact-anchored task and rubric generation.

  • Comprehensive agent comparison: The paper evaluates multiple models, rendering scaffolds, and 3D environments, revealing interesting performance and efficiency trade-offs.

  • Potential scalability: The paper demonstrates automated generation of 488 validated tasks and explores their use for agent fine-tuning.

Critical weaknesses
  • Questionable evaluation reliability: Despite claiming verifiable evaluation, the framework relies heavily on LLM judgments, even for geometric constraints, without sufficient independent ground-truth validation.

  • Limited benchmark representativeness: The main benchmark contains only 36 tasks, while the 488 extended tasks are primarily validated through automated LLM-based checks, with limited evidence of generalizability.

  • Insufficient functional verification: Static scene-state inspection and rendered images cannot reliably establish correct dynamic behavior, mechanical functionality, or temporal consistency.

  • Unconvincing fine-tuning improvements: Despite increased execution rates, absolute rubric compliance after fine-tuning remains extremely low, with evaluations on only 20 held-out tasks.

Questions for authors
  • How does the multimodal evaluator compare against independently human-verified ground truth for geometric and structural criteria? Could deterministic geometric checks replace some LLM-based judgments?

  • How representative are the 36 core tasks, and are the reported model differences statistically significant?

  • How does the framework verify actual dynamic or mechanical functionality rather than static geometric plausibility? Could temporal simulation, collision testing, or animation-based verification be incorporated?

4 comments
  • W/Q1: How does the multimodal evaluator compare against independently human-verified ground truth for geometric and structural criteria? Could deterministic geometric checks replace some LLM-based judgments?

    • For geometric and structural criteria (our State channel), eight annotators provided 199 Pass/Fail judgments across 108 criteria, achieving 90.5% agreement (κ=.729) with the three-judge majority vote (Luna, Gemini, DeepSeek), compared to 86.6% (κ=.566) on perceptual Image criteria and 88.3% (κ=.645) overall:

      CriteriaHuman labelsAgreement (%)κ
      All44688.3.645
      State19990.5.729
      Image24786.6.566

      Agreement reaches 95.9% (κ=.851) on criteria without human ties. Image disagreements stem from perceptual judgments under lighting or partial occlusion, while State discrepancies reflect coordinate conventions (Z-up vs. Y-up in Blender) or concept-to-entity mapping.

    • Deterministic Python checks readily compute numeric metrics once target entities and reference points are established, but hardcoded checking scripts struggle when agents adopt diverse object naming, mesh decompositions, and coordinate conventions. State extracts raw geometric data, while rubrics specify functional parts and relations.

      Geometric decomposition: In the folding desk, for example, the rubric measures the distance between cabinet-side and desk-side pivot centers. Different agents decompose the support arm into separate webs, flanges, and eyelets, while DeepSeek places object origins at (0,0,0) with coordinates stored in mesh vertices; naively querying object.location or a bounding box measures the wrong quantity.

      Scene hierarchies: Similarly, in the orrery, Gemini nests moon_arm_pivot hierarchically while Luna places parts flat under the scene root, requiring the verifier to interpret assembly relations across diverse scene structures.

      Because rigid programmatic rules struggle to accommodate such implementation diversity, we use the LLM to interpret semantic components, structural assemblies, and parameters from State, and evaluate whether the rubric requirements are met.

  • W/Q2a: How representative are the 36 core tasks, and are the reported model differences statistically significant?

    • We expanded the benchmark from 36 to 63 tasks by adding 27 tasks and evaluated all 17 configurations on these additions.

    • Procedural 3D tasks are inherently composite. Rather than testing a single scalar property, each task couples spatial, mechanical, and visual constraints, verified by an average of ~29 fine-grained rubric criteria. Across the 63 tasks, 1,829 criteria (684 state, 1,145 image) systematically cover eight core modeling capabilities: spatial metric reasoning (60/63 tasks), layout & circulation (57/63), terrain & volumes (52/63), kinematics & mechanisms (48/63), hierarchy & attachment (46/63), materials & surfaces (60/63), lighting & atmosphere (40/63), and composition & legibility (46/63). Complementing functional rubric compliance, Visual Quality (VQ) independently evaluates rendered perceptual quality across nine dimensions covering geometry, scene completeness, materials, lighting, composition, detail, visual coherence, aesthetics, and image quality.

    • Under one-hour budgets, the original 36 tasks cleanly separate agents from 8B models to frontier models:

      ModelScaffoldExec (%) ↑RC [95% CI]VQ [95% CI]
      Qwen3-VL-8BCodex19.4.020 [.006, .037].034 [.012, .060]
      Qwen3.5-9BOpenCode58.3.122 [.070, .180].102 [.065, .142]
      Qwen3.6-35B-A3BCodex69.4.177 [.104, .258].173 [.120, .230]
      Qwen3.5-122B-A10B-FP8Codex66.7.266 [.179, .357].197 [.134, .262]
      Qwen3.5-122B-A10B-FP8OpenCode75.0.360 [.262, .461].233 [.171, .295]
      Gemini-3.8-FlashOpenCode100.0.767 [.693, .831].580 [.541, .612]
      GPT-5.6-LunaCodex100.0.814 [.757, .864].601 [.576, .625]
      GPT-5.6-SolCodex100.0.848 [.814, .880].614 [.591, .636]
      GPT-6-AstraCodex100.0.876 [.830, .916].680 [.660, .699]

      Both the 8B–122B and 122B–Astra differences are statistically significant for RC and VQ (Bonferroni-adjusted p = .0014 under Codex).

  • W/Q2a (continued): How representative are the 36 core tasks, and are the reported model differences statistically significant?

    • We evaluated all 17 configurations on 27 added tasks, expanding the benchmark to 63 tasks across 31 Structure, 17 Dynamics, and 15 Material tasks. Model separations remain consistent (RC and VQ with 95% bootstrap confidence intervals in brackets):

      ModelScaffoldRC [95% CI]VQ [95% CI]
      GPT-6-AstraCodex.808 [.760, .851].665 [.648, .682]
      GPT-5.6-SolCodex.749 [.700, .795].578 [.556, .599]
      GPT-5.6-SolOpenClaw.650 [.580, .715].545 [.507, .580]
      DeepSeek-V4.1-FlashOpenClaw.458 [.375, .540].397 [.334, .457]

      Astra exceeds Sol in VQ under Codex (adjusted p = .0013). Sol exceeds DeepSeek-V4.1-Flash under OpenClaw (adjusted p = .0051 for RC and .0026 for VQ).

  • W/Q2b: The 488 extended tasks are primarily validated through automated LLM-based checks, with limited evidence of generalizability.

    • Trajectories from synthesized tasks improve performance on tasks excluded from training.

      The 488 extended tasks support task expansion and trajectory collection. Cross-model comparisons use the 63 expert-reviewed core tasks. Running Qwen3.5-122B-A10B-FP8 across these 488 tasks in engine sandboxes yields 1,333 nonempty trajectories and 715 valid deliverables. Training solely on top-quartile trajectories yields consistent SFT gains across 8B, 9B, and 35B models under both scaffolds on the disjoint 36 tasks (average RC gain of 2.10×).

  • W/Q3: How does the framework verify actual dynamic or mechanical functionality rather than static geometric plausibility? Could temporal simulation, collision testing, or animation-based verification be incorporated?

    • The Dynamics domain primarily evaluates procedural construction of motion-related scenes and articulated mechanisms in specified operating states. Its rubrics inspect mounting relationships, component poses, reach, contact, and clearances, with analytic checks across configurations where specified, such as gangway slope and roller travel across fluctuating water levels.

      While multi-frame simulation, collision tracking, and rigid-body physics can indeed be executed via the underlying Blender and Three.js engines, our primary focus is on whether agents can procedurally build valid kinematic mechanisms in code. Establishing correct attachment hierarchies, joint axes, and functional clearances is the foundational prerequisite before simulation can take place, and this assembly capability remains the primary bottleneck for current agents. Verifying mechanisms at designated operational states also provides a lightweight and scalable benchmark without the heavy runtime overhead or instability of physics solvers. Dynamic simulation is a natural and technically feasible extension that can be layered on top as procedural agents mature.

  • W/Q4: Despite increased execution rates, absolute rubric compliance after fine-tuning remains extremely low, with evaluations on only 20 held-out tasks.

    • SFT improves all three metrics across the three models and both scaffolds.

      In initial experiments, fine-tuning text-only Qwen3-8B on 20 held-out tasks yielded low compliance because it lacked vision and had weak 3D modeling skills, frequently causing syntax and code errors. We therefore re-ran fine-tuning using vision-capable Qwen3-VL-8B alongside Qwen3.5-9B and Qwen3.6-35B, evaluating under Codex and OpenCode on the 36 tasks (task IDs disjoint from training).

    • From 715 valid deliverables on the 488 tasks, selecting the top VQ quartile per runtime and scaffold yields 170 training trajectories (7,111 turns) and 37 for validation. We fine-tuned Qwen3-VL-8B (LoRA), Qwen3.5-9B (full-parameter), and Qwen3.6-35B-A3B (LoRA) for one epoch (888 steps).

    • Average gains in RC (2.10×) and VQ (1.71×) exceed the gain in Exec (1.49×). RC after SFT ranges from .062 to .188 on 8B and 9B and reaches .449 on 35B OpenCode:

      ModelScaffoldSettingExec (%) ↑RC ↑VQ ↑
      8BCodexBase19.4.020.034
      8BCodex+ SFT50.0.111.076
      8BOpenCodeBase27.8.035.044
      8BOpenCode+ SFT41.7.062.102
      9BCodexBase63.9.099.114
      9BCodex+ SFT75.0.154.166
      9BOpenCodeBase58.3.122.102
      9BOpenCode+ SFT63.9.188.142
      35BCodexBase69.4.177.173
      35BCodex+ SFT94.4.288.231
      35BOpenCodeBase61.1.202.175
      35BOpenCode+ SFT97.2.449.311
      Average gain--1.49×2.10×1.71×

I appreciate the authors' response that addressed some of my concerns partially. Some other concerns still remain, e.g., the evaluation reliability and the limited improvements. Hence, I will retain my rating unchaged.

Thank you for your follow-up. Regarding the two concerns you highlighted, we would greatly appreciate a brief clarification on what specific gaps remain in light of the new evidence in our rebuttal:

  1. Evaluation reliability: In our rebuttal, we explained why hardcoded deterministic scripts fail under diverse agent naming, mesh decompositions, and scene hierarchies (e.g., folding desk and orrery), necessitating LLM interpretation over extracted geometric State. More importantly, our independent human validation (8 annotators, 446 labels) directly confirms its reliability: 90.5% agreement (κ = 0.729) on geometric State criteria and 95.9% (κ = 0.851) on non-tie criteria, alongside high cross-judge rank stability (Spearman ρ = 0.929–0.976).

  2. Fine-tuning improvements: Addressing the initial text-only 8B result on 20 tasks, our new SFT experiments on 36 disjoint tasks across three vision-language models (8B/9B/35B) and two scaffolds yield an average 2.10× RC gain and 1.71× VQ gain: both substantially exceeding the 1.49× Exec gain. On Qwen3.6-35B-A3B (OpenCode), RC more than doubles from 0.202 to 0.449 (with Exec rising from 61.1% to 97.2%) using only 170 trajectories trained for one epoch, demonstrating clear generalization from our synthesized tasks.

We would be grateful for any concrete pointers before the discussion period closes.

Reviewer_FJEeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces 3D Code Factory, a framework for automatically generating procedural 3D coding tasks and corresponding evaluation rubrics, and 3D Live Bench, a benchmark for evaluating coding agents in Three.js and Blender environments. The evaluation combines scene-state inspection with rendered-image assessment to measure requirement compliance and visual quality, alongside execution success. Experiments show that, despite high execution success rates, current coding agents often struggle to satisfy fine-grained task requirements. Iterative refinement can also introduce regressions, with local fixes violating previously satisfied constraints.

Critical strengths
  1. The task setting is interesting and relevant. The benchmark assesses the 3D modeling capabilities of current coding agents, highlighting their limitations and identifying directions for future improvement.

  2. The benchmark generation pipeline is impressive, using LLMs to generate both task descriptions and corresponding evaluation rubrics from shared scene facts, helping maintain consistency between task requirements and evaluation criteria.

  3. The paper provides useful analyses, including comparisons with human preferences, an exploration of trajectory-based fine-tuning, and examples showing how iterative refinement can introduce regressions by violating previously satisfied constraints.

Critical weaknesses
  1. The presentation could be improved. In particular, the text in Figures 1, 2, and 6 is too small to read comfortably. Moreover, the trajectory visualization in Figure 2 does not clearly convey the actions taken or how the scene evolves across steps.

  2. The evaluation may depend on the choice of VLM judge. Although the paper compares two judges, their rubric scores differ considerably. It would be helpful to check whether changing the judge affects model rankings and to validate rubric judgments with human annotations.

  3. The benchmark could be organized more systematically. Although it includes scene and capability categories, it lacks explicit difficulty levels and a clear account of how the tasks cover core 3D modeling skills. It is therefore unclear whether the 36 core tasks provide sufficiently broad and representative coverage.

Questions for authors

See weaknesses

2 comments
  • W/Q1: The text in Figures 1, 2, and 6 is too small; the trajectory visualization in Figure 2 does not clearly convey the actions taken or scene evolution.

    • We enlarged the text in Figures 1, 2, and 6. The successive renders in Figure 2 illustrate how the scene evolves during a rollout. The actual trajectory records comprise LLM trajectories, which capture model inputs and outputs, and ACP trajectories, which capture agent–client messages, tool calls, and execution results. We added these distinctions to Figure 2.
  • W/Q2a: Rubric scores differ considerably between judges; does changing the judge affect model rankings?

    • Rankings remain stable across the three judges. The earlier Luna–Gemini ablation omitted geometric evidence through State truncation; we restored the complete extracted scene state and rescored the affected cases.

      Across 288 outputs (eight configurations × 36 tasks), pairwise Spearman correlations are .929–.952 for RC and .952–.976 for VQ across Luna, Gemini, and DeepSeek (Figure 6(b,c)). Cross-judge score spreads remain small (mean range 0.039 RC, 0.045 VQ), and GPT-5.6-Luna is strictest overall without showing self-preference on OpenAI models. Each cell below shows RC (rank) / VQ (rank):

      Model (scaffold)DeepSeekGeminiLuna
      GPT-6-Astra (Codex).908 (1) / .674 (2).913 (2) / .652 (2).854 (2) / .689 (2)
      GPT-6-Astra (OpenClaw).889 (2) / .679 (1).922 (1) / .660 (1).873 (1) / .690 (1)
      GPT-5.6-Luna (Codex).793 (3) / .551 (3).802 (4) / .558 (4).793 (3) / .590 (3)
      Gemini-3.8-Flash (OpenCode).789 (4) / .551 (4).817 (3) / .573 (3).771 (5) / .577 (4)
      GPT-5.6-Luna (OpenCode).753 (5) / .517 (5).781 (5) / .549 (5).783 (4) / .575 (5)
      Claude-Opus-5 (Claude Code).617 (6) / .423 (7).648 (6) / .475 (7).601 (6) / .445 (6)
      Claude-Sonnet-5 (Claude Code).544 (8) / .402 (8).570 (8) / .444 (8).524 (8) / .396 (8)
      DeepSeek-V4.1-Flash (Codex).578 (7) / .430 (6).587 (7) / .502 (6).563 (7) / .434 (7)
  • W/Q2b: Validate rubric judgments with human annotations.

    • Eight annotators provided 446 Pass/Fail judgments on 232 criterion–output pairs, compared against the majority vote of Luna, Gemini, and DeepSeek:

      CriteriaHuman labelsAgreement (%)κ
      All44688.3.645
      State19990.5.729
      Image24786.6.566

      Agreement is notably higher for State criteria (90.5%, κ=.729) than Image criteria (86.6%, κ=.566). As a sensitivity analysis, aggregating judgments on the 195 criteria without human ties yields 95.9% agreement (κ=.851). Disagreements concentrate in Image criteria involving perceptual judgments (e.g., subtle lighting contrast or partial occlusion), while State discrepancies stem from coordinate conventions (Z-up vs. Y-up in Blender) or entity mapping.

  • W/Q3: The benchmark lacks explicit difficulty levels and a clear account of core 3D modeling skills, leaving the breadth and representativeness of the 36 core tasks unclear.

    • We expanded the benchmark from 36 to 63 tasks by adding 27 new tasks and evaluated all 17 configurations on these additions.

    • In open-ended 3D coding, difficulty cannot be captured by a single pre-assigned axis. Task difficulty emerges from coupled spatial, assembly, and visual constraints and varies across agent backbones and scaffolds. For example, the funicular station task simultaneously requires inclined tracks, level floors, platform alignment, and clear circulation; its difficulty comes from satisfying these constraints jointly. We therefore characterize difficulty empirically using each task's mean RC across the 17 evaluated configurations. Sorting by mean RC yields three equal-sized groups: Easier (mean RC .813), Middle (.622), and Harder (.408). Harder tasks feature more complex scene organization across multiple functional zones, disparate elevations, and internal structures, requiring agents to maintain global layout, spatial hierarchy, and overall consistency.

    • As for core modeling skills, the rubric criteria serve as the formal skill specification, stating concrete conditions decomposed into state-verified and image-verified checks. Across the 63 tasks, 1,829 criteria (684 state, 1,145 image) explicitly cover eight core modeling capabilities: terrain and volumetric construction (52/63 tasks), spatial metric reasoning (60/63), layout and circulation (57/63), hierarchy and attachment (46/63), kinematics and mechanisms (48/63), materials and surfaces (60/63), lighting and atmosphere (40/63), and composition and legibility (46/63). VQ separately assesses quality across nine dimensions: geometry, scene completeness, materials, lighting, composition, detail, coherence, aesthetics, and imaging quality.

Thanks for the response. I will keep my rate to weak acceptance.

Reviewer_CRTwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary
  • They proposes a framework 3D Code Factory for automatically generating executable 3D coding tasks paired with evaluation rubrics, enabling scalable benchmarking and evaluation of coding agents.
  • 3D Live Bench: Builds on 3D Code Factory to evaluate multiple coding agents across Blender and Three.js. Agents iteratively write and execute code, inspect scene states, review rendered images, refine their 3D models using tool feedback.
  • Generates task instructions and evaluation rubrics from shared scene facts. Evaluation combines scene-state checks (geometry, distances, hierarchy) with image-based checks (appearance, lighting, visibility).
  • Dataset has 36 core benchmark tasks and an extended set of 488 validated tasks, filtered from 1000 generated candidates across 500 scenarios.
  • Main results: Although 15/17 agent configurations achieve 100% execution success, rubric compliance remains lowish. The best visual quality and rubric compliance come from different configurations, highlighting the gap between visual appearance and task correctness.
  • Main findings: Agent scaffolds significantly affect performance and cost. Fine-tuning Qwen3-8B on 786 agent trajectories improves execution success (20% → 60% on Blender), but rubric compliance remains very low.
Critical strengths
  • Works on an important problem of evaluating open-ended 3D coding tasks.
  • Deriving instructions and evaluation rubrics from shared scene facts is a sensible approach, improving consistency between task requirements and evaluation criteria.
  • Multimodal evaluation, following common practices, combinning scene-state inspection with rendered-image evaluation enables assessment of both geometric correctness and visual quality.
  • Evaluates complete coding-agent workflows with iterative code execution, scene inspection and visual feedback across multiple models, scaffolds, and 3D environments.
Critical weaknesses
  • Limited novelty: The overall framework largely combines established techniques—LLM-based task synthesis, rubric generation, multimodal judging, and agent rollouts—without introducing a fundamentally new verification or learning algorithm.
  • Questionable verification reliability: Despite claiming verifiable tasks, geometric constraints are ultimately judged by an LLM using scene metadata. Bounding boxes and object hierarchies cannot reliably establish physical attachment, functional mechanisms, or geometric correctness.
  • More engineering than research contribution: The approach resembles a well-engineered software evaluation pipeline rather than a novel research methodology. Its technical contributions and broader scientific insights appear insufficiently substantial for a venue like ICLR.
  • Limited benchmark and training evidence: The main benchmark has only 36 tasks. Fine-tuning improves execution success, but rubric compliance remains extremely low, providing less insights on 3D imporvement, progress etc
Questions for authors

Please try to address the weaknesses raised above and share your thoughts on what novel research insights the paper offers. There is no harm in developing a technical product, which could be equally impactful, or investing more effort in expanding the dataset to strengthen its value as a benchmark.

2 comments
  • W/Q1: Limited novelty without a new verification or learning algorithm.

    • We respectfully disagree with the limited-novelty assessment. Fact-Anchored Synthesis jointly constructs task instructions and verification criteria from shared scene facts, so that different valid 3D implementations can be evaluated against the same explicit conditions.

    • In procedural 3D coding, execution success alone establishes neither spatial correctness nor rendered quality, while a single natural-language goal admits a large, under-constrained space of valid scenes and implementations. Existing benchmarks make verification tractable by narrowing this space externally, relying on curated procedural assets (3DCodeBench [1]), parametric 3D datasets (P3D-Bench [2]), or expert-written runtime contracts (WorldCoder-Bench [3]). Scaling open-ended tasks across different 3D engines without reference assets therefore remains difficult.

    • The core difficulty is defining objective, checkable correctness for open-ended 3D tasks without restricting valid implementations. When a single prompt admits many valid geometries and code structures, fixed reference assets restrict implementation openness, while per-task runtime contracts require manual, task-specific authoring that limits scalability. Conversely, unconstrained prompts leave correctness undefined and give a verifier nothing to check against. Solving this requires an intermediate specification that makes correctness checkable without collapsing the solution space to one implementation. Fact-Anchored Synthesis provides this layer, grounding task instructions and rubrics in shared, feasibility-checked observable facts F\mathcal{F}.

      Starting from text seed scenarios, our framework enables low-cost task expansion through automated task and rubric generation, reducing manual reference-asset curation and per-task checker authoring.

    • Our framework yields 488 audited tasks from 1,000 candidates and the expanded 63-task benchmark. SFT on the collected trajectories improves Exec, RC, and VQ across 8B, 9B, and 35B students under both Codex and OpenCode, with an average RC gain of 2.10×.

  • W/Q2: Bounding boxes and object hierarchies cannot reliably establish physical attachment, functional mechanisms, or geometric correctness.

    • We clarify that verification in our benchmark refers to the observable geometric, structural, and visual conditions explicitly specified by each rubric. Our evaluator assesses these conditions using extracted scene State and rendered views, with a declared evidence channel for each criterion.

      Our State checks assess observable geometric conditions specified by the rubric, including proximity within a stated tolerance, dimensional containment, and hierarchical grouping, using measured positions, bounds, and transforms. Properties requiring visual judgment, such as whether joints are visibly constructed and interfaces appear mechanically plausible, are assigned to Image criteria. Each criterion declares its required evidence channel, and the scope of each channel is defined in the evaluation specification.

      For example, the folding-desk rubrics check pivot-to-pivot distances and attachment-point placement using measured State. Image criteria separately check modeled hinge barrels, pivot joints, and mounting brackets, including pins or bolts passing through connected segments and visible fastening details. This evaluates the specified assembly through both spatial measurements and constructed interfaces, with criterion-level human agreement reported below.

    • To validate consistency within these declared channels, eight annotators provided 446 individual Pass/Fail judgments on 232 criterion–output pairs, compared against the majority vote of Luna, Gemini, and DeepSeek:

      CriteriaHuman labelsAgreement (%)κ
      All44688.3.645
      State19990.5.729
      Image24786.6.566

      Agreement is notably higher for State criteria (90.5%, κ=.729) than Image criteria (86.6%, κ=.566). As a sensitivity analysis, aggregating judgments on the 195 criteria without human ties yields 95.9% agreement (κ=.851). Disagreements concentrate primarily in Image criteria involving perceptual judgments (e.g., subtle material contrast under lighting or component visibility under partial occlusion), while State discrepancies stem primarily from coordinate-axis conventions (Z-up vs. Y-up in Blender) or concept-to-entity mapping in multi-component assemblies.

References

[1] Gao et al. (2026). 3DCodeBench: Benchmarking Agentic Procedural 3D Modeling Via Code.

[2] Yang et al. (2026). P3D-Bench: Benchmarking MLLMs for Parametric 3D Generation and Structural Reasoning.

[3] Lu et al. (2026). WorldCoder-Bench: Benchmarking Physically Grounded 3D World Synthesis.

  • W/Q3: More engineering than research contribution; what novel research insights does the paper offer?

    • We respectfully disagree that the study offers only engineering value. Section 5 presents empirical findings on how coding agents behave in procedural 3D environments:

    • Scaffold design changes execution and cost: In procedural 3D coding, completing a scene requires cycling between code generation, engine execution, runtime error handling, and visual inspection. Scaffolds significantly affect whether models enter this active execution loop or stall in read-only planning. Furthermore, per-task token usage and API cost vary substantially across scaffolds for the same base model (Sec. 5.2; Figure 6(a)).

    • Local visual repairs can break spatial constraints: In 3D modeling, geometry and visual appearance are tightly coupled. When agents edit code based on rendered visual feedback (e.g., to reduce visual overlap or improve visibility), local adjustments frequently break previously satisfied global constraints such as clearances, alignments, or coordinate frames (Sec. 5.2).

    • Visual quality and compliance can diverge: A scene can look visually compelling while leaving specific rubric requirements unfulfilled, and agents sometimes focus on cosmetic adjustments (such as lighting or materials) instead of missing functional parts. This demonstrates why 3D evaluation requires checking both internal scene state and rendered output (Sec. 5.2).

    • SFT sustains inspection and editing: SFT on verified trajectories helps open models across 8B, 9B, and 35B sizes learn multi-turn inspection and editing, yielding an average 2.10× gain in rubric compliance. The improvements address different bottlenecks across model sizes: helping 8B models produce executable code, teaching 9B models to act on visual feedback, and helping 35B models move from planning to active execution (Sec. 5.3; Table 3).

      These empirical findings identify concrete behavioral bottlenecks in multi-turn 3D coding agents. More broadly, our end-to-end pipeline (integrating scalable task synthesis, multi-turn engine execution, and checkable multimodal verification) transforms open-ended 3D modeling into a closed-loop environment with automated reward feedback. The same framework supports controlled comparisons of models and scaffolds and supplies trajectories for the SFT experiments reported here.

  • W/Q4: The main benchmark has only 36 tasks; fine-tuning improves execution success but rubric compliance remains extremely low, providing limited insight into 3D improvement.

    • We expanded the benchmark from 36 to 63 tasks and evaluated all 17 configurations on the added tasks.

    • For fine-tuning, we collected 1,333 nonempty trajectories from Qwen3.5-122B-A10B-FP8 on the 488 extended tasks under Codex and OpenCode, retained 715 valid deliverables, and reserved 37 for validation (1,140 turns).

      Selecting the top VQ quartile per runtime and scaffold yields 170 training trajectories (7,111 turns).

      We fine-tuned Qwen3-VL-8B (LoRA), Qwen3.5-9B (full-parameter), and Qwen3.6-35B-A3B (LoRA) for one epoch (888 steps), evaluating on the other 36 tasks.

    • SFT improves all three metrics across models and scaffolds. Gains in RC (2.10×) and VQ (1.71×) both exceed Exec (1.49×), with RC reaching .449 on 35B OpenCode:

      ModelScaffoldSettingExec (%) ↑RC ↑VQ ↑
      8BCodexBase19.4.020.034
      8BCodex+ SFT50.0.111.076
      8BOpenCodeBase27.8.035.044
      8BOpenCode+ SFT41.7.062.102
      9BCodexBase63.9.099.114
      9BCodex+ SFT75.0.154.166
      9BOpenCodeBase58.3.122.102
      9BOpenCode+ SFT63.9.188.142
      35BCodexBase69.4.177.173
      35BCodex+ SFT94.4.288.231
      35BOpenCodeBase61.1.202.175
      35BOpenCode+ SFT97.2.449.311
      Average gain--1.49×2.10×1.71×

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.