acceptodds
Under review as a conference paper at ICLR 2027

UnityBench-Mini3D: Evaluating LLM Agents for 3D Game Development in Unity

Abstract

Game development stresses coding agents in ways general software engineering does not: correctness depends on how engine APIs, serialized scene state, and runtime behavior interact, not on source code alone. Existing benchmarks mainly target platforms such as Godot and heavily rely on LLM judges or coarse task-level outcomes. We present UnityBench-Mini3D, a benchmark of 77 project-level Unity tasks organized around seven expert-defined skill categories rather than tutorials or genres, with three difficulty tiers. Each task pairs a complete Unity project with natural-language requirements and is evaluated using 1,178 held-out, human-authored milestone tests covering scene structure, component configuration, and runtime behavior. This in-engine evaluation provides deterministic partial credit, localizes failures, and distinguishes partial implementations from complete solutions without relying on subjective judges. Across 21 models and 9 coding harnesses, the average pass rate reaches 80.9% on easy tasks but falls to 35.2% on hard tasks, with just 57.7% overall. UnityBench-Mini3D provides a reproducible, diagnostic testbed distinguishing systems across difficulty levels.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Benchmarking, Game Development, Computer Graphics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QAAdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • This work introduces UnityBench-Mini3D with 77 project tasks with 1178 tests.
  • This work tackles a limitation of prior game development by extending to Unity and evaluates partial correctness of solutions.
  • The authors evaluated 21 models across nine coding harnesses, alongside GUI-agent baselines and tool ablation
Critical strengths
  • Useful expansion of game development benchmarks to Unity.
  • Comprehensive evaluation across models and harnesses.
Critical weaknesses
  • The paper offers limited novelty beyond extending evaluation to Unity. Prior benchmarks already test project-level game development using skill categories and executable tests, and the experiments largely repeat GameDevBench’s evaluation setup (e.g., across model, harness, difficulty, modality, etc). As such, the takeaways and additional insights provided by this work remain unclear. Further, the best model already has over 72% performance, indicating the problems are not necessarily that difficult.
  • The paper does not establish whether Unity introduces harder or distinct challenges compared with other engines.
  • The paper does not report agreement between authored milestones and expert assessments.
Questions for authors

Please address weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to reviewer QAAd - W1-1

We sincerely thank reviewer QAAd for the thoughtful feedback and valuable suggestions. Here, we respond to each of the reviewer's concerns.

W1. The paper offers limited novelty beyond extending evaluation to Unity. Prior benchmarks already test project-level game development using skill categories and executable tests, and the experiments largely repeat GameDevBench's evaluation setup eg˙,˙acrossmodel,harness,difficulty,modality,etce\.g\., across model, harness, difficulty, modality, etc. As such, the takeaways and additional insights provided by this work remain unclear. Further, the best model already has over 72% performance, indicating the problems are not necessarily that difficult.

Response to W1.

We appreciate GameDevBench's contribution and compare it with other benchmarks in Table 1. UnityBench-Mini3D is an independently designed benchmark that complements GameDevBench through distinct task construction and requirement-level scoring. While GameDevBench derives tasks from web and video tutorials through agent-assisted construction and human review, our 77 tasks and 1178 milestone tests were manually authored by Unity developers around six core skill categories plus a miscellaneous category, yielding seven categories in total. These cover scene hierarchies, prefab/component configuration, and interactions among animation, physics, and rendering. Our milestone scores capture partial completion and identify unmet requirements, providing additional diagnostic detail beyond task-level binary success.

The experiments also provide practical insights: harness choice substantially affects performance, dedicated engine tools help, and visual and GUI access provide uneven benefits. Compilation fails in only 2.27% of runs, yet the average scene-construction score is 44.3%, highlighting the gap between compilable code and correct project state.

Easy tasks intentionally measure basic operations, while Hard tasks assess their combination; the average Hard-task milestone score across configurations is only 35.21%. The reported 72.49% measures average milestone completion. Our benchmark also supports project-level binary success, similar to GameDevBench, and we additionally report this metric below (in the next response): a task succeeds only if all milestone tests pass. The configuration achieving 72.49% fully completes only 23/77 tasks 298˙729\.87%, including 3/23 Hard tasks 130˙413\.04%. The highest complete-task success rate across all 32 configurations is 33.77%. Completing all task requirements therefore remains challenging.

We post the table in the next comment.

AuthorsOpenReview ↗

Response to reviewer QAAd - W1-2

Response to W1 (continued)

HarnessModelUIAnm.Edtr.Scn.P.&S.Phys.Misc.OverallEasyMed.Hard
CodelyClaude Opus 522.2216.6745.4515.7933.3327.2733.3325.975613.798.7
CodelyGPT 5.6 Sol11.1116.6718.1815.7933.3327.27019.48406.913.04
CodelyKimi K311.112545.4526.3233.3318.1833.3327.275613.7913.04
CodelyDeepSeek V4 Pro016.6736.3615.792518.18018.18486.90
CodelyDeepSeek V4 Flash11.1133.3327.2715.792518.18020.78446.913.04
CodelyMiniMax M322.2216.679.0908.339.0909.092404.35
CodelyGLM 5.211.11259.0915.79259.09015.583210.344.35
CodelyQwen 3.8 Max33.3316.6754.5521.0516.6718.18024.685213.798.7
CodelyGrok 4.633.332536.3621.0533.3318.1833.3327.276413.794.35
CodelyHunyuan 3016.67015.7909.0907.791210.340
CodelySeed 2.1 Pro08.339.09016.679.0906.49123.454.35
OpenCodeKimi K322.222527.2721.052527.2733.3324.684820.694.35
OpenCodeGPT 5.6 Sol22.2233.3345.4521.0516.6727.27025.974813.7917.39
OpenCodeGLM 5.211.1116.6718.1815.7909.09011.69286.90
OpenCodeDeepSeek V4 Flash08.339.095.2609.0905.191204.35
CodexGPT 5.6 Sol22.222536.3615.795036.3633.3329.876017.2413.04
CodexGPT 5.6 Terra22.222545.4515.7933.339.09023.384813.798.7
CodexGPT 5.6 Luna22.2233.3327.2710.5316.6718.18019.484017.240
CodexGPT 5.511.112536.3610.5316.6727.27019.484410.344.35
Claude CodeClaude Opus 511.112536.3621.0533.3327.2733.3325.974420.6913.04
Claude CodeClaude Sonnet 5016.6718.1815.792518.18015.58406.90
Grok BuildGrok 4.633.3333.3327.2721.0533.339.0933.3325.976413.790
Grok BuildGrok 4.511.1141.6745.4521.0541.6727.2733.3331.176420.698.7
AntigravityGemini 3.1 Pro016.679.095.268.339.0907.79166.90
AntigravityGemini 3.7 Flash11.11259.0910.53259.0933.3315.58406.90
Qwen CodeQwen 3.8 Max11.1133.3345.4526.322518.18025.975220.694.35
Qwen CodeQwen 3.7 Max08.33010.538.3318.1807.79163.454.35
Qwen CodeQwen 3.7 Plus016.679.0910.5316.6718.18011.69283.454.35
Kimi CodeKimi K322.2241.6736.3615.7916.6718.18023.385210.348.7
Kimi CodeKimi K2.7 Code08.339.0910.53018.1807.79163.454.35
DS HarnessDeepSeek V4 Flash02536.3615.792518.18019.484010.348.7
DS HarnessDeepSeek V4 Pro22.2233.3336.3636.845027.27033.775631.0313.04
AuthorsOpenReview ↗

Response to reviewer QAAd - W2

W2. The paper does not establish whether Unity introduces harder or distinct challenges compared with other engines.

Response to W2.

Unity provides a distinct development setting through its GameObject-component model, Prefab relationships, and asset-reference system. Its C# scripts and text-serializable scenes and Prefabs support direct inspection and modification, while Unreal's principal asset formats are binary and Godot uses a node-and-resource model with text-based scene representations. Godot scenes commonly express hierarchy through readable parent paths, whereas Unity uses local fileIDs and asset GUIDs for serialized references and records Prefab instance changes as overrides. Agents editing Unity files must preserve these relationships, while Unreal's binary assets generally require editor tools or APIs. When relevant, Unity agents must also identify the UI and rendering systems used by each project, such as UI Toolkit or UGUI and Built-in or URP, and coordinate changes with asset imports and script reloads.

UnityBench-Mini3D grounds its milestone tests in these concrete development mechanisms. Prefab-editing tasks check whether changes preserve instance connections, inherited properties, and existing transforms, while vehicle-control tasks assess WheelCollider and Rigidbody configuration alongside acceleration and straight-line stability. In the Task 0004 case study (Appendix E.3\ in the paper), Grok 4.6 passed 9/9 checks in Codely using Unity APIs, but 0/9 in Grok Build after editing scene text, despite its own parser reporting no mismatches. This case illustrates why our scene tests inspect Unity's loaded object model to check task-specific requirements. Separately, removing Codely's dedicated engine tools reduced overall scores by 2.01-7.44 percentage points across three models (Section 4.4\ in the paper). Evaluation must account for persistence, script compilation, asset import, and differences between editor and runtime state. It must also control execution conditions for physics and rendering checks and distinguish task failures from editor, tool-connection, or environment errors. By examining both project configuration and resulting runtime behavior, our milestones reveal which Unity-specific requirements an agent satisfies and where its implementation remains incomplete, providing a complementary perspective to benchmarks in other engines.

AuthorsOpenReview ↗

Response to reviewer QAAd - W3

W3. The paper does not report agreement between authored milestones and expert assessments.

Response to W3.

Our validation combines expert consensus on milestone criteria with repeated experiments assessing evaluation stability and discriminatory ability. The construction and expert-review process is outlined in Section 3.1, with further details below.

Expert review, agreement, and revision. Six Unity developers constructed the task specifications, starting projects, golden references, and milestone tests. Five additional developers with LLM / agent experience joined the review, focusing on whether task descriptions and starting projects provided agents with sufficient, clear information to complete the tasks. Together, reviewers examined whether the acceptance criteria matched the stated requirements and intended skills, whether the starting resources were sufficient, and whether the validation approach contained omissions or could be bypassed.

Each week, designers prepared candidate projects and acceptance criteria for discussion at the following week's group meeting. Designers presented each task's rationale and validation logic. When disagreements arose, they explained their design intent, and reviewers discussed ambiguities and potential biases. If multiple reviewers continued to consider a design unsuitable, the task was redesigned. Candidates were revised and resubmitted until all eleven reviewers approved them, establishing consensus on the task requirements and their acceptance criteria. Experts who had not authored the tasks attempted them manually using only the starting projects and specifications. Other experts assessed reasonable score ranges based on task completion. We checked whether these ranges overlapped with the evaluator-derived ranges as a qualitative check of specification clarity and scoring alignment.

Milestone design and empirical validation. We decompose each task's objective into concrete requirements with observable acceptance criteria. Experts cross-review these criteria for consistency with the task objectives and potential ambiguity. For example, the river-assembly task separately assesses water coverage, connections between river segments, and downstream water-level progression. The vehicle-control task assesses wheel-collider configuration, acceleration performance, and straight-line stability. These checks capture different aspects of completing a development task.

We verified that each golden reference scored 100% and each unmodified starting project scored 0%. We also conducted repeated experiments using four model-harness configurations spanning different capability levels Codely+Qwen36˙27B,Codely+GLM52˙,Codely+DeepSeekV4Flash,andCodex+GPT56˙SolCodely + Qwen 3\.6 27B, Codely + GLM 5\.2, Codely + DeepSeek V4 Flash, and Codex + GPT 5\.6 Sol, with at least five repeated runs per configuration. If a task showed substantial variation, such as frequently receiving both 0% and 100% under the same configuration, we investigated the underlying causes and considered refinements to the relevant milestones, followed by further validation. Comparisons across configurations spanning different capability levels additionally assessed whether the milestones could distinguish different levels of agent capability.

Reviewer_W2dpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UnityBench-Mini3D evaluates LLM agents on modifying existing Unity projects. It contains 77 manually designed tasks spanning seven development/skill categories and three difficulty levels. Each task provides an initial project and a natural-language specification. Evaluation uses 1,178 human authored milestone tests covering project structure, component configuration, and runtime behavior, providing partial credit without LLM judges.

The authors evaluate 21 models across nine coding harnesses, alongside GUI-based agents, and analyze performance, cost, tool use, and failures. Ablations examine visual input, dedicated Unity tools, and required GUI interaction. The results highlight substantial harness effects, persistent difficulty with scene construction, and limited overall gains from visual input or GUI interaction in the tested configurations.

Critical strengths
  1. The paper addresses a clear evaluation gap. I agree with the motivation that game-development correctness depends on interactions between engine APIs, serialized scene state, and runtime behavior. The focus on project-level Unity development adds useful coverage to a literature that has largely focused on other engines, particularly Godot. Organizing tasks around development skills also makes the results easier to interpret.
  2. The evaluation is broad and informative. The study includes a diverse set of contemporary proprietary and open-weight models. The comparisons across models, harnesses, and GUI-based agents are particularly valuable and less commonly explored in this area. Findings about harness choice, scene construction, and GUI interaction provide useful insight into the capabilities and limitations of current agents.
Critical weaknesses
  1. The benchmark construction and validation process is not explained clearly enough. The paper describes expert review, but the roles of the task designers and the five additional developers with LLM/agent experience are unclear. It also needs more detail on how reviewers checked specification–test alignment, resolved ambiguity, and assigned difficulty levels. These details matter for deciding whether the benchmark measures the intended skills fairly.
  2. The evaluation needs clearer reporting and stronger support for some interpretations. The paper does not clearly describe isolation between milestone tests within a task or provide exact model settings for each configuration. The small performance differences in the visual-input ablation also need cautious interpretation. Figures 16 and 17 suggest that the usefulness of visual input may depend on how informative the captured images are, rather than simply on whether vision is available.
Questions for authors
  1. How were task specifications authored and validated? What did the five developers with LLM/agent experience contribute, and how did their responsibilities differ from those of the six Unity experts? How did reviewers ensure that every tested requirement was clearly stated? Were tasks independently attempted using only the supplied project and specification to identify ambiguity?
  2. What criteria distinguish easy, medium, and hard tasks? Please explain how the labels account for implementation effort, the number of interacting requirements, and reasoning complexity.
  3. How are milestone tests isolated within a task? Separate benchmark tasks run in isolated environments, but are scenes and runtime state reset between milestone tests? Have results been checked for sensitivity to test order? How are dependencies handled when one missing prerequisite causes several tests to fail (could that happen?)?
  4. Can the exact experimental settings be reported? Please provide reasoning-effort settings, token limits, sampling parameters, and model identifiers for each model–harness configuration, including relevant defaults. “Highest available reasoning effort” and harness defaults can change over time, making the experiments difficult to reproduce.
  5. How was visual evidence acquired in the multimodal experiments? Figure 17 raises the possibility that a static screenshot misses the temporal information needed to classify particle effects, or that the selected frame is not representative. Please clarify capture timing, image selection, and whether multiple frames were available. Do the small performance differences persist across repeated runs?
7 comments
AuthorsOpenReview ↗

Response to reviewer W2dp - W1

We sincerely thank reviewer W2dp for the thoughtful feedback and valuable suggestions. Here, we respond to each of the reviewer's concerns.

W1. The benchmark construction and validation process is not explained clearly enough. The paper describes expert review, but the roles of the task designers and the five additional developers with LLM/agent experience are unclear. It also needs more detail on how reviewers checked specification-test alignment, resolved ambiguity, and assigned difficulty levels. These details matter for deciding whether the benchmark measures the intended skills fairly.

Response to W1.

We outline the benchmark construction and expert-review process in Section 3.1 in the paper and provide further details below.

Roles and task design. Six Unity developers constructed the task specifications, starting projects, golden references, and milestone tests. Five additional developers with LLM and AI-agent experience joined the review process, focusing on whether the task descriptions and starting projects provided agents with sufficient, clear information to complete the tasks. Tasks covered six core skill categories plus a miscellaneous category, yielding seven categories in total, and drew inspiration from routine development workflows, tutorials, and commercial-game interactions. Design criteria included clear objectives, sufficient resources and context, feasibility from the initial project, observable and verifiable outcomes, appropriate difficulty, and diverse development skills.

Cross-review and agreement. Tasks underwent iterative construction, review, and experimentation. Each week, designers prepared candidate projects and acceptance criteria for discussion at the following week's group meeting. Reviewers checked whether the acceptance criteria matched the stated requirements, whether the starting project provided sufficient resources, and whether the validation approach contained omissions or could be bypassed. When disagreements arose, designers explained their intent, and reviewers discussed ambiguities and potential biases. Candidates were revised and resubmitted until all eleven reviewers approved them.

Experimental validation. We verified that each golden reference scored 100% and each unmodified starting project scored 0%. Repeated experiments assessed evaluation stability and whether milestones distinguished different levels of agent capability. Substantial variation within the same configuration, such as frequent scores of both 0% and 100%, prompted investigation and consideration of milestone refinements, followed by further validation. We provide details of the experimental configurations and milestone design in our responses to Q2 and Q3, respectively.

Difficulty assignment. Experts initially assigned difficulty based on the complexity a human developer would face, then empirically calibrated the labels using four model-harness configurations spanning different capability levels, each evaluated over at least five repeated runs. We provide further details in our response to Q2.

Skill coverage. Table 2 in the paper reports the skills exercised, task counts, and milestone-test counts for all seven categories. This makes the benchmark's coverage explicit, while cross-review of specifications and acceptance criteria helps ensure that tasks assess the intended skills consistently.

AuthorsOpenReview ↗

Response to reviewer W2dp - W2

W2. The evaluation needs clearer reporting and stronger support for some interpretations. The paper does not clearly describe isolation between milestone tests within a task or provide exact model settings for each configuration. The small performance differences in the visual-input ablation also need cautious interpretation. Figures 16 and 17 suggest that the usefulness of visual input may depend on how informative the captured images are, rather than simply on whether vision is available.

Response to W2.

Run isolation. Each task run starts from a clean VM snapshot with the Unity Editor and Python preinstalled. Before launching the agent, we remove Git history, repository documentation, and unrelated files from the starter project, retaining only the files needed for the task. As described in Section 3.3 in the paper, the agent receives only this starter project and the task specification. Network access is limited to required services such as model inference. The original task and evaluation repositories are outside the allowlist.

When the agent finishes or reaches the time limit, we terminate its session and processes before fetching and injecting the held-out milestone tests. The evaluator automatically scores the submitted project using the Unity Test Framework. After scoring, we archive the workspace and execution trajectory. We then manually review every archived trajectory to check for errors attributable to the evaluation environment. Finally, we restore the VM snapshot before the next task. We provide completed reference projects (golden references) separately to help benchmark users analyze their results. The agent has no access to the held-out tests or golden references during the run.

Milestone tests. Development tasks are isolated from one another. Each run starts from a clean VM snapshot, so project and runtime state cannot carry over between tasks. Within a task, the evaluator records a pass or fail for each check and computes the score as the fraction of checks passed. Of the 1,178 checks, 146 123˙912\.39% belong to tiered test groups that award partial credit at progressively stricter thresholds. For example, the rolling-ball task reloads the scene for each test and scores whether the ball reaches at least one, three, or all five checkpoints. The remaining 1,032 checks 876˙187\.61% assess individual criteria without tiers. Checks within a task may share prerequisites. In all experiments reported in the paper, we ran the milestone tests in the benchmark's predefined execution order.

Model settings. Table 3aa in the paper lists the model-harness combinations. To reflect how developers typically use agents in game development, we retained each harness's default settings for temperature, top_p, and maximum output tokens. Where reasoning effort was configurable, we selected the highest level supported by the model through that harness to elicit its strongest performance. We reported detailed model identifier, reasoning efforts, harness version (also reported in Appendix H) and other harness default parameters of these harness versions (if applicable) to make sure the whole experiment is fully reproducible. We provide the detailed experimental settings in our response to Q4.

Visual ablation. Disabling visual input changed the overall scores by less than 2pp in the OpenCode and Codely comparisons. As shown in Figure 2, agents rarely invoked visual tools even when available, which may partly explain the small aggregate differences. We discuss these results in detail in Section 4.4 in the paper and provide further explanations of the multimodal setup, screenshot acquisition, and repeated runs in our response to Q5.

AuthorsOpenReview ↗

Response to reviewer W2dp - Q1

Q1. How were task specifications authored and validated? What did the five developers with LLM/agent experience contribute, and how did their responsibilities differ from those of the six Unity experts? How did reviewers ensure that every tested requirement was clearly stated? Were tasks independently attempted using only the supplied project and specification to identify ambiguity?

Response to Q1.

The benchmark was constructed around six core skill categories plus a miscellaneous category, yielding seven categories in total. Experts drew inspiration from existing tutorials, real commercial games, and frequently encountered development scenarios, then created a starting project, task description, and acceptance criteria for each task. Task design followed criteria including practical development value, clear objectives, sufficient initial conditions, verifiable outcomes, and evaluation stability. For example, the automatic-door task specifies the axis of rotation and requires a repeated opening and closing cycle; its acceptance tests check both scene structure and runtime behavior. The vehicle task evaluates physics control through actual driving behavior. After the first version of each task was completed, experts cross-reviewed its rationale and validation approach at a group meeting in the following workweek. When disagreements arose, the designer explained the design intent and the team discussed potential biases; if multiple experts still considered the task unsuitable, it was redesigned. Reviewed tasks then underwent repeated experiments, with identified issues fixed and re-evaluated, while components producing highly variable results were narrowed following expert discussion. Experts who had not authored a given task also attempted to complete it manually using only the supplied starting project and task specification. Other experts assessed the resulting implementations against the stated requirements and identified score ranges they considered reasonable given the degree of task completion. We compared these judgments with the scores produced by the milestone evaluator, checking for overlap between the evaluator-derived and expert-assessed score ranges. This process provided a qualitative check of specification clarity and the alignment between automated scoring and expert judgment. Experts initially assessed task difficulty from the perspective of a human developer and subsequently adjusted the difficulty categories using at least five sets of experimental results. Tasks were organized using a consistent structure of project templates and corresponding evaluators. For each run, a working copy was created from the template and evaluated after the agent completed the task\

Reviewer_xu8nOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UnityBench-Mini3D, a benchmark for evaluating LLM agents on Unity projects. It contains 77 tasks across seven development categories, evaluated through pre-defined milestone test cases. Experiments cover diverse models including GPT-5.6-sol and Opus-5, and nine coding-agent harnesses. The results show that dedicated Unity tools improve performance within Codely, but most models evaluated in both settings perform better in their provider-native harnesses.

Critical strengths
  • The paper addresses an important gap in evaluating agents for Unity game development. Tasks require modifying projects, including their code, scenes, and assets. The benchmark provides deterministic evaluation without LLM judges, and milestone-level partial credit captures progress rather than only full-task success.

  • The paper evaluates a broad range of models and harnesses. The 32 (model, harness) configurations provide useful comparisons, including how the same model performs under different harnesses.

Critical weaknesses
  • The variability across repeated runs is unclear, as each configuration is evaluated only once per task. The bootstrap analysis in Appendix D.3 captures uncertainty from task composition, but not run-to-run variability. Repeating a few representative (model, harness) configurations at least four times and reporting mean scores and standard deviations would help assess the stability of the results.

  • The analysis of Unity-specific tools is limited to Codely. Although these tools improve performance within Codely, most models still perform better in their provider-native harnesses. It would be useful to test whether the tools offer limited gains across harnesses or whether Codely limits their effectiveness. Could the authors add the same Unity MCP tools to Codex or Claude Code and compare performance with and without them, using the same model and time budget?

Questions for authors
  • Can the Unity MCP tools used by Codely be exposed to Codex or Claude Code? If so, how do these harnesses perform with the same tools?
  • How much do the 77 starting projects share assets, templates, or scene structures? Clarifying this would help readers assess task diversity.
5 comments
AuthorsOpenReview ↗

Response to reviewer xu8n - W1

We sincerely thank reviewer xu8n for the thoughtful feedback and valuable suggestions. Here, we respond to each of the reviewer's concerns.

W1. The variability across repeated runs is unclear, as each configuration is evaluated only once per task. The bootstrap analysis in Appendix D.3 captures uncertainty from task composition, but not run-to-run variability. Repeating a few representative (model, harness) configurations at least four times and reporting mean scores and standard deviations would help assess the stability of the results.

Response to W1.

Due to time constraints, we selected 27 representative challenges covering all task categories and difficulty levels and evaluated four model-harness configurations, with five independent runs per challenge and configuration. The first table reports pass@1, pass@3, and pass@5. Easy-task pass@5 scores approach 100%, ranging from 96.17% to 99.26%, whereas Hard-task scores range from 53.02% to 76.41%. Hard tasks therefore retain a substantial performance spread of 23.39 percentage points, compared with 3.09 points on Easy tasks. These results support the difficulty tiers under repeated attempts: Easy tasks assess foundational capabilities, while Hard tasks remain challenging and distinguish frontier-model configurations.

HarnessModelMetricUIAnm.Edtr.Scn.P.&S.Phys.Misc.OverallEasyMed.Hard
CodelyGPT 5.6 SolPass@140.0674.3150.3150.1390.0079.0354.8262.9686.5671.4538.24
CodelyGPT 5.6 SolPass@358.6285.4298.3358.6692.5079.8458.1576.5996.7979.2358.38
CodelyGPT 5.6 SolPass@558.6287.9298.3360.3697.5079.8458.1577.9596.7982.0959.91
CodelyGLM 5.2Pass@146.4956.8075.6648.7386.7371.9429.1760.4883.3663.4039.90
CodelyGLM 5.2Pass@350.5860.5583.7470.6691.7371.9458.4370.0896.1768.5849.68
CodelyGLM 5.2Pass@551.9063.3387.1971.2395.5871.9458.4371.8596.1770.1753.02
CodelyKimi K3Pass@149.7070.0068.1670.0992.5064.2755.3867.5987.0474.3647.38
CodelyKimi K3Pass@357.6088.0693.1681.4597.5068.8162.2279.0098.1582.7460.95
CodelyKimi K3Pass@568.4288.0694.0285.0197.5071.0962.2281.5998.1586.3665.02
CodexGPT 5.6 SolPass@166.9689.3694.0265.1268.0865.1258.4372.9683.6475.1062.86
CodexGPT 5.6 SolPass@371.0595.8397.4775.9995.5879.8458.4382.9098.0283.6770.04
CodexGPT 5.6 SolPass@571.0597.2297.4775.9995.5895.9762.1485.9199.2683.6776.41

We also report individual-run scores and their means and standard deviations. Three of the four configurations have standard deviations between 2.42 and 3.88pp, indicating relatively consistent aggregate performance across five runs. Codely + GPT 5.6 Sol shows greater variability, with a standard deviation of 6.76pp. These results support the stability of aggregate scores for most tested configurations while making configuration-dependent variability explicit.

HarnessModelR1R2R3R4R5Mean ± SD
CodelyGPT 5.6 Sol62.9666.4648.7858.5756.2458.60 ± 6.76
CodelyGLM 5.260.4864.1453.7856.8558.5958.77 ± 3.88
CodelyKimi K367.5973.1472.8473.0373.0271.93 ± 2.42
CodexGPT 5.6 Sol72.9678.2177.5978.4079.9577.42 ± 2.64
AuthorsOpenReview ↗

Response to reviewer xu8n - W2 and Q1

W2. The analysis of Unity-specific tools is limited to Codely. Although these tools improve performance within Codely, most models still perform better in their provider-native harnesses. It would be useful to test whether the tools offer limited gains across harnesses or whether Codely limits their effectiveness. Could the authors add the same Unity MCP tools to Codex or Claude Code and compare performance with and without them, using the same model and time budget?

Response to W2.

Codely is closed-source, so we could not directly port its built-in engine-specific tools to other harnesses. Instead, we used UnityMCP, an open-source alternative, and compared performance with and without it in Codex, OpenCode, and Kimi Code, keeping the model and time budget fixed within each comparison. We also include the baseline results without UnityMCP from Table 3 of the paper to make it easier for the reviewer to compare the statistics.

HarnessModelSettingUIAnm.Edtr.Scn.P.&S.Phys.Misc.OverallEasyMed.Hard
CodexGPT 5.6 Solw/ UnityMCP66.4981.9090.7364.5088.8572.6362.1476.0693.9770.9063.09
CodexGPT 5.6 Solw/o UnityMCP65.8074.6572.5761.6990.0478.7458.8072.4990.2566.9760.13
OpenCodeKimi K3w/ UnityMCP66.3285.5480.9550.1086.8277.9165.5672.2293.6571.7249.56
OpenCodeKimi K3w/o UnityMCP48.6962.8359.6856.3579.3665.7158.8061.9684.6367.6630.13
OpenCodeGLM 5.2w/ UnityMCP56.9958.4868.3945.3084.1776.8155.1062.9684.6461.5941.11
OpenCodeGLM 5.2w/o UnityMCP50.8649.1265.7636.5261.0158.0655.1051.9582.4245.4827.00
Kimi CodeKimi K3w/ UnityMCP62.0170.9285.9356.6788.3580.7955.5872.0494.0271.9848.21
Kimi CodeKimi K3w/o UnityMCP65.5778.3463.8650.5277.5873.6358.4366.3589.7865.5341.90

UnityMCP improved overall scores by 3.57pp for Codex + GPT 5.6 Sol, 10.26pp for OpenCode + Kimi K3, 11.01pp for OpenCode + GLM 5.2, and 5.69pp for Kimi Code + Kimi K3. These results align with Table 5, where disabling Codely's built-in engine-specific tools reduced performance by 2.01-7.44 pp across the tested configurations. Both Codely's proprietary tools and the open-source UnityMCP therefore support the same finding: engine-specific tools improve agent performance in Unity across the tested harnesses.

Q1. Can the Unity MCP tools used by Codely be exposed to Codex or Claude Code? If so, how do these harnesses perform with the same tools?

Response to Q1.

As discussed in W2, Codely's proprietary engine-specific tools could not be directly ported. We therefore used the open-source UnityMCP alternative in Codex, OpenCode, and Kimi Code, keeping the model and time budget fixed within each comparison. Overall scores improved by 3.57-11.01pp across all four tested configurations, including 72.49% to 76.06% for Codex + GPT 5.6 Sol, consistent with the benefits observed for Codely's built-in tools.

HarnessModelSettingUIAnm.Edtr.Scn.P.&S.Phys.Misc.OverallEasyMed.Hard
CodexGPT 5.6 Solw/ UnityMCP66.4981.9090.7364.5088.8572.6362.1476.0693.9770.9063.09
CodexGPT 5.6 Solw/o UnityMCP65.8074.6572.5761.6990.0478.7458.8072.4990.2566.9760.13
OpenCodeKimi K3w/ UnityMCP66.3285.5480.9550.1086.8277.9165.5672.2293.6571.7249.56
OpenCodeKimi K3w/o UnityMCP48.6962.8359.6856.3579.3665.7158.8061.9684.6367.6630.13
OpenCodeGLM 5.2w/ UnityMCP56.9958.4868.3945.3084.1776.8155.1062.9684.6461.5941.11
OpenCodeGLM 5.2w/o UnityMCP50.8649.1265.7636.5261.0158.0655.1051.9582.4245.4827.00
Kimi CodeKimi K3w/ UnityMCP62.0170.9285.9356.6788.3580.7955.5872.0494.0271.9848.21
Kimi CodeKimi K3w/o UnityMCP65.5778.3463.8650.5277.5873.6358.4366.3589.7865.5341.90
AuthorsOpenReview ↗

Response to reviewer xu8n - Q2

Q2. How much do the 77 starting projects share assets, templates, or scene structures? Clarifying this would help readers assess task diversity.

Response to Q2.

We designed the 77 challenges to span a broad range of Unity workflows, including scene construction, animation, physics, UI, rendering, asset management, and editor automation. To quantify reuse, we compared file contents in each starting project's Assets/ directory, excluding empty files, Unity .meta files, and task prompts. Each challenge contained an average of 562 asset files, with a median of 22.

Exact asset overlap was limited for most challenge pairs. Of the 2,926 pairs from these 77 challenges, 82.0% shared no identical file contents. Only 73 pairs 24˙92\.49% shared at least 10 distinct file contents, and 27 pairs 09˙20\.92% shared at least 100.

Based on file paths, 97.1% of distinct shared file contents could be attributed to identifiable resource collections, libraries, or settings. As a share of all 11,796 distinct shared contents, a set of 3D art asset collections accounted for 9,009 764˙76\.4%, a weapons audio collection for 1,314 111˙11\.1%, a networking library for 776 66˙6\.6%, a residential environment asset collection for 329 28˙2\.8%, and render-pipeline settings or profiles for 24 02˙0\.2%. These resources include reusable art and audio assets, a networking library, and render-pipeline settings that Unity projects may import without modification. They provide generic building material rather than task-specific logic, so their reuse has limited bearing on what each challenge asks the agent to do or how it is evaluated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.