acceptodds
Withdrawn from ICLR 2027

Agent's GameDev Exam

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Evaluating AI agents for game development requires testing whether they can translate requirements into working gameplay within complex engine environments. We introduce Agent's GameDev Exam, a production-oriented benchmark spanning Unreal Engine 5, Unity, Godot, and RPG Maker, with an estimated inventory of 448 candidate tasks from 55 source projects and packages. Two complementary tracks assess component-level feature development within existing projects and End2end-level game creation from supplied assets, functional specifications, and reference screenshots. Our construction pipeline combines expert-defined requirements, LLM-assisted task preparation, and structural and in-engine validation; component-level tasks remove complete feature implementations while preserving the surrounding project. Evaluation combines deterministic checks with ground-truth-referenced video assessment to test requested functionality and, where applicable, preservation of existing behaviors. Across nine model–harness configurations, the current evaluation yields a best component-level full-pass rate of 14.23%, while no configuration achieves a full pass on the End2end-level track. Analysis of UE5 component-level submissions identifies incomplete execution paths and disconnected state-driven effects as the dominant issue categories, accounting for 48.3% and 18.3% of categorized issues, respectively. These findings expose a gap between locally plausible implementations and complete feature delivery, positioning Agent's GameDev Exam as a testbed for improving system-level reasoning, integration, and runtime verification in game-development agents.

Primary area:
datasets and benchmarks
Keywords:
game development, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_x2ncOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Agent's GameDev Exam, which consists of 448 tasks from 55 source projects and packages across Unreal Engine 5, Unity, Godot, and RPG Maker. The component-level track asks agents to restore a removed feature while preserving specified existing behaviors. The End2end-level track asks them to build games. Evaluation combines deterministic checks with video assessment against reference executions. Nine LLM configurations are evaluated, and none of them achieves a full pass on the End2end-level track.

Critical strengths
  • It is well-motivated to evaluate coding agents on game development across major engines. Working with real projects requires agents to integrate their implementations into an existing gameplay environment.
  • The feature-removal procedure is carefully designed to remove the target implementation while preserving shared dependencies. Verifying the modified project in its engine helps prevent faulty starting conditions from being counted against the agent.
  • The evaluation covers nine frontier LLMs, showing how difficult complete feature delivery remains for current agents.
Critical weaknesses
  • A valid End2end solution could fail if its level layout causes the evaluator's fixed test path to miss the object being tested. This makes it difficult to assess agents that build more complex game environments, since a failed test may simply mean the evaluator could not reach the relevant part of the scene.
  • The paper emphasizes multi-engine coverage across Unreal Engine, Unity, Godot, and RPG Maker. However, the experimental results are only reported in aggregate. Table 2 provides no engine-wise breakdown.
  • The paper describes how experts prepare the task specifications, but provides little evidence that the instructions given to agents cover the behavior required by the tests. An omitted interaction condition could cause a reasonable implementation to fail.
  • The evaluation is relatively narrow, focusing primarily on requirement satisfaction. It lacks an independent assessment of the overall quality of the completed games.
Questions for authors
  • How many tasks are included for each game engine, and how do different models perform across these engines?
  • How does the unchanged starting project perform under the same evaluation protocol? It seems there is a lack of a baseline evaluation.
  • How is the video evaluator calibrated and validated in practice?
  • Why do runtime and token usage vary substantially across models and between the component-level and End2end tracks, particularly for cases such as DeepSeek, whose token usage is relatively low on component-level tasks but extremely high on End2end tasks?
Reviewer_m4wROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Agent's GameDev Exam, a benchmark for evaluating AI agents on real-world game development tasks. It comprises 448 tasks from 55 game projects across Unreal Engine 5, Unity, Godot, and RPG Maker. The benchmark includes two tracks: Component-level, which evaluates feature implementation in existing projects while preserving functionality, and End2end-level, which assesses complete game development from assets, specifications, and visual references.

Critical strengths
  1. The paper presents a substantial benchmarking effort, constructing 448 tasks from 55 game projects and packages.

  2. The benchmark is based on real-world game projects and covers multiple game engines. It also introduces two complementary settings, Component-level and End2end-level, to evaluate different aspects of game development capabilities.

  3. The paper evaluates a diverse set of recent and advanced models with their respective agent harnesses.

Critical weaknesses
  1. The writing quality is not good. The manuscript contains frequent single-word line breaks and defensive phrasing, which suggests AI generation.

  2. The experimental analysis is insufficient. Despite covering multiple game engines and game genres, the paper only reports aggregate results without finer-grained comparisons. In particular, all models achieve a 0% full-pass rate on End2end-level tasks, yet the paper provides little insight into the underlying failures or differences across models. While the failure analysis in Section 5.3 is limited to UE5 Component-level tasks. Additionally, the paper reports execution times but does not explore their relationship with model performance.

  3. The novelty of the benchmark is somewhat limited compared to existing game-development benchmarks. Its main distinction appears to be combining component-level feature development within existing projects and end-to-end game creation from raw assets into a single benchmark.

  4. Although the paper mentions human calibration and blind multi-LLM assessments, it does not report quantitative agreement rates with human experts or consistency across different judges. Given the role of video-based evaluation in determining task success, these results are important for establishing the reliability of the benchmark scores.

Questions for authors
  1. Which multimodal LLM was used as the video judge in the actual evaluation? What reasoning effort was used for each model?

  2. How many of the 448 tasks belong to the Component-level and End2end-level tracks, respectively? Can you share the model performance across different game engines and task settings, rather than only reporting aggregate results?

  3. What hardware configurations were used to run the benchmark, and how many tasks were executed in parallel? Given the large number of tasks and repeated evaluations, running all 448 tasks sequentially would be extremely time-consuming. How did you tackle this?

  4. Why does GPT-6 Astra terminate so early on End2end-level tasks (1.56 hours on average), while DeepSeek-V4.1-Flash runs for much longer (43.80 hours)? Can you explain the reasons behind these substantial differences in execution time?

Reviewer_Q8tvOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

Agent's GameDev Exam presents a production oriented benchmark for game development. It presents various types of tasks from functional to end to end level design tasks.

Critical strengths
  1. The paper is generally well written and somewhat easy to follow.
  2. The topic is interesting and the production level dataset is useful.
  3. There are a variety of different tasks and evaluation methodologies, all of which are reasonable.
  4. Tasks have different context lengths
Critical weaknesses
  1. I don't think this really pushes the boundaries in anyway. It's more data and evaluation, but no real new methodology has been provided.
  2. While the paper is overall clear, at some points it can become difficult to follow. In particular, sections 3 and 4 are somewhat difficult to follow. It could be better organized and the figures could be clearer (right now it's just way too much going on in two identical looking figures; I found the figures more confusing than useful)
Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.