- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Agent's GameDev Exam, which consists of 448 tasks from 55 source projects and packages across Unreal Engine 5, Unity, Godot, and RPG Maker. The component-level track asks agents to restore a removed feature while preserving specified existing behaviors. The End2end-level track asks them to build games. Evaluation combines deterministic checks with video assessment against reference executions. Nine LLM configurations are evaluated, and none of them achieves a full pass on the End2end-level track.
Critical strengths
- It is well-motivated to evaluate coding agents on game development across major engines. Working with real projects requires agents to integrate their implementations into an existing gameplay environment.
- The feature-removal procedure is carefully designed to remove the target implementation while preserving shared dependencies. Verifying the modified project in its engine helps prevent faulty starting conditions from being counted against the agent.
- The evaluation covers nine frontier LLMs, showing how difficult complete feature delivery remains for current agents.
Critical weaknesses
- A valid End2end solution could fail if its level layout causes the evaluator's fixed test path to miss the object being tested. This makes it difficult to assess agents that build more complex game environments, since a failed test may simply mean the evaluator could not reach the relevant part of the scene.
- The paper emphasizes multi-engine coverage across Unreal Engine, Unity, Godot, and RPG Maker. However, the experimental results are only reported in aggregate. Table 2 provides no engine-wise breakdown.
- The paper describes how experts prepare the task specifications, but provides little evidence that the instructions given to agents cover the behavior required by the tests. An omitted interaction condition could cause a reasonable implementation to fail.
- The evaluation is relatively narrow, focusing primarily on requirement satisfaction. It lacks an independent assessment of the overall quality of the completed games.
Questions for authors
- How many tasks are included for each game engine, and how do different models perform across these engines?
- How does the unchanged starting project perform under the same evaluation protocol? It seems there is a lack of a baseline evaluation.
- How is the video evaluator calibrated and validated in practice?
- Why do runtime and token usage vary substantially across models and between the component-level and End2end tracks, particularly for cases such as DeepSeek, whose token usage is relatively low on component-level tasks but extremely high on End2end tasks?