- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
- This work introduces UnityBench-Mini3D with 77 project tasks with 1178 tests.
- This work tackles a limitation of prior game development by extending to Unity and evaluates partial correctness of solutions.
- The authors evaluated 21 models across nine coding harnesses, alongside GUI-agent baselines and tool ablation
Critical strengths
- Useful expansion of game development benchmarks to Unity.
- Comprehensive evaluation across models and harnesses.
Critical weaknesses
- The paper offers limited novelty beyond extending evaluation to Unity. Prior benchmarks already test project-level game development using skill categories and executable tests, and the experiments largely repeat GameDevBench’s evaluation setup (e.g., across model, harness, difficulty, modality, etc). As such, the takeaways and additional insights provided by this work remain unclear. Further, the best model already has over 72% performance, indicating the problems are not necessarily that difficult.
- The paper does not establish whether Unity introduces harder or distinct challenges compared with other engines.
- The paper does not report agreement between authored milestones and expert assessments.
Questions for authors
Please address weaknesses above.
4 comments
Response to reviewer QAAd - W1-2
Response to W1 (continued)
| Harness | Model | UI | Anm. | Edtr. | Scn. | P.&S. | Phys. | Misc. | Overall | Easy | Med. | Hard |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Codely | Claude Opus 5 | 22.22 | 16.67 | 45.45 | 15.79 | 33.33 | 27.27 | 33.33 | 25.97 | 56 | 13.79 | 8.7 |
| Codely | GPT 5.6 Sol | 11.11 | 16.67 | 18.18 | 15.79 | 33.33 | 27.27 | 0 | 19.48 | 40 | 6.9 | 13.04 |
| Codely | Kimi K3 | 11.11 | 25 | 45.45 | 26.32 | 33.33 | 18.18 | 33.33 | 27.27 | 56 | 13.79 | 13.04 |
| Codely | DeepSeek V4 Pro | 0 | 16.67 | 36.36 | 15.79 | 25 | 18.18 | 0 | 18.18 | 48 | 6.9 | 0 |
| Codely | DeepSeek V4 Flash | 11.11 | 33.33 | 27.27 | 15.79 | 25 | 18.18 | 0 | 20.78 | 44 | 6.9 | 13.04 |
| Codely | MiniMax M3 | 22.22 | 16.67 | 9.09 | 0 | 8.33 | 9.09 | 0 | 9.09 | 24 | 0 | 4.35 |
| Codely | GLM 5.2 | 11.11 | 25 | 9.09 | 15.79 | 25 | 9.09 | 0 | 15.58 | 32 | 10.34 | 4.35 |
| Codely | Qwen 3.8 Max | 33.33 | 16.67 | 54.55 | 21.05 | 16.67 | 18.18 | 0 | 24.68 | 52 | 13.79 | 8.7 |
| Codely | Grok 4.6 | 33.33 | 25 | 36.36 | 21.05 | 33.33 | 18.18 | 33.33 | 27.27 | 64 | 13.79 | 4.35 |
| Codely | Hunyuan 3 | 0 | 16.67 | 0 | 15.79 | 0 | 9.09 | 0 | 7.79 | 12 | 10.34 | 0 |
| Codely | Seed 2.1 Pro | 0 | 8.33 | 9.09 | 0 | 16.67 | 9.09 | 0 | 6.49 | 12 | 3.45 | 4.35 |
| OpenCode | Kimi K3 | 22.22 | 25 | 27.27 | 21.05 | 25 | 27.27 | 33.33 | 24.68 | 48 | 20.69 | 4.35 |
| OpenCode | GPT 5.6 Sol | 22.22 | 33.33 | 45.45 | 21.05 | 16.67 | 27.27 | 0 | 25.97 | 48 | 13.79 | 17.39 |
| OpenCode | GLM 5.2 | 11.11 | 16.67 | 18.18 | 15.79 | 0 | 9.09 | 0 | 11.69 | 28 | 6.9 | 0 |
| OpenCode | DeepSeek V4 Flash | 0 | 8.33 | 9.09 | 5.26 | 0 | 9.09 | 0 | 5.19 | 12 | 0 | 4.35 |
| Codex | GPT 5.6 Sol | 22.22 | 25 | 36.36 | 15.79 | 50 | 36.36 | 33.33 | 29.87 | 60 | 17.24 | 13.04 |
| Codex | GPT 5.6 Terra | 22.22 | 25 | 45.45 | 15.79 | 33.33 | 9.09 | 0 | 23.38 | 48 | 13.79 | 8.7 |
| Codex | GPT 5.6 Luna | 22.22 | 33.33 | 27.27 | 10.53 | 16.67 | 18.18 | 0 | 19.48 | 40 | 17.24 | 0 |
| Codex | GPT 5.5 | 11.11 | 25 | 36.36 | 10.53 | 16.67 | 27.27 | 0 | 19.48 | 44 | 10.34 | 4.35 |
| Claude Code | Claude Opus 5 | 11.11 | 25 | 36.36 | 21.05 | 33.33 | 27.27 | 33.33 | 25.97 | 44 | 20.69 | 13.04 |
| Claude Code | Claude Sonnet 5 | 0 | 16.67 | 18.18 | 15.79 | 25 | 18.18 | 0 | 15.58 | 40 | 6.9 | 0 |
| Grok Build | Grok 4.6 | 33.33 | 33.33 | 27.27 | 21.05 | 33.33 | 9.09 | 33.33 | 25.97 | 64 | 13.79 | 0 |
| Grok Build | Grok 4.5 | 11.11 | 41.67 | 45.45 | 21.05 | 41.67 | 27.27 | 33.33 | 31.17 | 64 | 20.69 | 8.7 |
| Antigravity | Gemini 3.1 Pro | 0 | 16.67 | 9.09 | 5.26 | 8.33 | 9.09 | 0 | 7.79 | 16 | 6.9 | 0 |
| Antigravity | Gemini 3.7 Flash | 11.11 | 25 | 9.09 | 10.53 | 25 | 9.09 | 33.33 | 15.58 | 40 | 6.9 | 0 |
| Qwen Code | Qwen 3.8 Max | 11.11 | 33.33 | 45.45 | 26.32 | 25 | 18.18 | 0 | 25.97 | 52 | 20.69 | 4.35 |
| Qwen Code | Qwen 3.7 Max | 0 | 8.33 | 0 | 10.53 | 8.33 | 18.18 | 0 | 7.79 | 16 | 3.45 | 4.35 |
| Qwen Code | Qwen 3.7 Plus | 0 | 16.67 | 9.09 | 10.53 | 16.67 | 18.18 | 0 | 11.69 | 28 | 3.45 | 4.35 |
| Kimi Code | Kimi K3 | 22.22 | 41.67 | 36.36 | 15.79 | 16.67 | 18.18 | 0 | 23.38 | 52 | 10.34 | 8.7 |
| Kimi Code | Kimi K2.7 Code | 0 | 8.33 | 9.09 | 10.53 | 0 | 18.18 | 0 | 7.79 | 16 | 3.45 | 4.35 |
| DS Harness | DeepSeek V4 Flash | 0 | 25 | 36.36 | 15.79 | 25 | 18.18 | 0 | 19.48 | 40 | 10.34 | 8.7 |
| DS Harness | DeepSeek V4 Pro | 22.22 | 33.33 | 36.36 | 36.84 | 50 | 27.27 | 0 | 33.77 | 56 | 31.03 | 13.04 |
Response to reviewer QAAd - W2
W2. The paper does not establish whether Unity introduces harder or distinct challenges compared with other engines.
Response to W2.
Unity provides a distinct development setting through its GameObject-component model, Prefab relationships, and asset-reference system. Its C# scripts and text-serializable scenes and Prefabs support direct inspection and modification, while Unreal's principal asset formats are binary and Godot uses a node-and-resource model with text-based scene representations. Godot scenes commonly express hierarchy through readable parent paths, whereas Unity uses local fileIDs and asset GUIDs for serialized references and records Prefab instance changes as overrides. Agents editing Unity files must preserve these relationships, while Unreal's binary assets generally require editor tools or APIs. When relevant, Unity agents must also identify the UI and rendering systems used by each project, such as UI Toolkit or UGUI and Built-in or URP, and coordinate changes with asset imports and script reloads.
UnityBench-Mini3D grounds its milestone tests in these concrete development mechanisms. Prefab-editing tasks check whether changes preserve instance connections, inherited properties, and existing transforms, while vehicle-control tasks assess WheelCollider and Rigidbody configuration alongside acceleration and straight-line stability. In the Task 0004 case study (Appendix E.3\ in the paper), Grok 4.6 passed 9/9 checks in Codely using Unity APIs, but 0/9 in Grok Build after editing scene text, despite its own parser reporting no mismatches. This case illustrates why our scene tests inspect Unity's loaded object model to check task-specific requirements. Separately, removing Codely's dedicated engine tools reduced overall scores by 2.01-7.44 percentage points across three models (Section 4.4\ in the paper). Evaluation must account for persistence, script compilation, asset import, and differences between editor and runtime state. It must also control execution conditions for physics and rendering checks and distinguish task failures from editor, tool-connection, or environment errors. By examining both project configuration and resulting runtime behavior, our milestones reveal which Unity-specific requirements an agent satisfies and where its implementation remains incomplete, providing a complementary perspective to benchmarks in other engines.
Response to reviewer QAAd - W3
W3. The paper does not report agreement between authored milestones and expert assessments.
Response to W3.
Our validation combines expert consensus on milestone criteria with repeated experiments assessing evaluation stability and discriminatory ability. The construction and expert-review process is outlined in Section 3.1, with further details below.
Expert review, agreement, and revision. Six Unity developers constructed the task specifications, starting projects, golden references, and milestone tests. Five additional developers with LLM / agent experience joined the review, focusing on whether task descriptions and starting projects provided agents with sufficient, clear information to complete the tasks. Together, reviewers examined whether the acceptance criteria matched the stated requirements and intended skills, whether the starting resources were sufficient, and whether the validation approach contained omissions or could be bypassed.
Each week, designers prepared candidate projects and acceptance criteria for discussion at the following week's group meeting. Designers presented each task's rationale and validation logic. When disagreements arose, they explained their design intent, and reviewers discussed ambiguities and potential biases. If multiple reviewers continued to consider a design unsuitable, the task was redesigned. Candidates were revised and resubmitted until all eleven reviewers approved them, establishing consensus on the task requirements and their acceptance criteria. Experts who had not authored the tasks attempted them manually using only the starting projects and specifications. Other experts assessed reasonable score ranges based on task completion. We checked whether these ranges overlapped with the evaluator-derived ranges as a qualitative check of specification clarity and scoring alignment.
Milestone design and empirical validation. We decompose each task's objective into concrete requirements with observable acceptance criteria. Experts cross-review these criteria for consistency with the task objectives and potential ambiguity. For example, the river-assembly task separately assesses water coverage, connections between river segments, and downstream water-level progression. The vehicle-control task assesses wheel-collider configuration, acceleration performance, and straight-line stability. These checks capture different aspects of completing a development task.
We verified that each golden reference scored 100% and each unmodified starting project scored 0%. We also conducted repeated experiments using four model-harness configurations spanning different capability levels , with at least five repeated runs per configuration. If a task showed substantial variation, such as frequently receiving both 0% and 100% under the same configuration, we investigated the underlying causes and considered refinements to the relevant milestones, followed by further validation. Comparisons across configurations spanning different capability levels additionally assessed whether the milestones could distinguish different levels of agent capability.
Response to reviewer QAAd - W1-1
We sincerely thank reviewer QAAd for the thoughtful feedback and valuable suggestions. Here, we respond to each of the reviewer's concerns.
Response to W1.
We appreciate GameDevBench's contribution and compare it with other benchmarks in Table 1. UnityBench-Mini3D is an independently designed benchmark that complements GameDevBench through distinct task construction and requirement-level scoring. While GameDevBench derives tasks from web and video tutorials through agent-assisted construction and human review, our 77 tasks and 1178 milestone tests were manually authored by Unity developers around six core skill categories plus a miscellaneous category, yielding seven categories in total. These cover scene hierarchies, prefab/component configuration, and interactions among animation, physics, and rendering. Our milestone scores capture partial completion and identify unmet requirements, providing additional diagnostic detail beyond task-level binary success.
The experiments also provide practical insights: harness choice substantially affects performance, dedicated engine tools help, and visual and GUI access provide uneven benefits. Compilation fails in only 2.27% of runs, yet the average scene-construction score is 44.3%, highlighting the gap between compilable code and correct project state.
Easy tasks intentionally measure basic operations, while Hard tasks assess their combination; the average Hard-task milestone score across configurations is only 35.21%. The reported 72.49% measures average milestone completion. Our benchmark also supports project-level binary success, similar to GameDevBench, and we additionally report this metric below (in the next response): a task succeeds only if all milestone tests pass. The configuration achieving 72.49% fully completes only 23/77 tasks 298˙7, including 3/23 Hard tasks 130˙4. The highest complete-task success rate across all 32 configurations is 33.77%. Completing all task requirements therefore remains challenging.
We post the table in the next comment.