- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Game, a new game-dev benchmark of 247 tasks grounded in 41 executable Godot games for coding agents. While existing benchmarks have focused on evaluating generated games' quality with either VLM judges or simple unit tests, SWE-Game diversifies the evaluation metrics to measure various aspects of mechanic correctness, as well as to use VLM judges. Across six models, Opus 5 achieves the best performance, but the overall score remains below 60 out of 100 across the tasks. The paper also compared its evaluation methods to human annotations, which showed high agreement above 90% balanced accuracy (especially for executable checks).
Critical strengths
First, this benchmark is well-built and can evaluate the coding agent's capabilities in several ways, including generating from a brief description, generating from a GDD, fixing bugs, and even porting from Godot to Unity. In addition, the evaluation is thorough because the authors introduced not only functional evaluation (with a new game instrumentation interface) but also visual evaluation. Across 5 different tasks with 2 different evaluation methods, I believe that this benchmark is well balanced to assess diverse capabilities of coding agents. In addition, their analysis regarding automatic eval vs. human eval ensures that this evaluation method is valid and precise.
Also, the experiments and analysis on 6 coding agents/models show that existing models are far from generating or rebuilding the game in the way that the user wants to build. These results imply the necessity of the benchmark, which is a critical strength, I believe.
Critical weaknesses
Importantly, this paper does not have any link to the code implementation, which blocks transparent analysis/reproducibility.
Furthermore, there's no further analysis of the proposed benchmark (which seems necessary), such as:
- whether the games included are prone to contamination issues
- performance between 2d vs. 3d games, or genre-specific scores, or even the relationship between game complexity vs. model scores.
- Especially for "certified routes", I'm not sure if all the details are included in the paper (e.g., # of registered routes per game, length of routes, # of actions per route, ratio of route that uses semantic start, etc).
While the paper proposes a lots of notations and concepts, it's hard to follow up on all the details, especially in terms of readability. It would be better to make a bold/italic for new terms or notations.
Questions for authors
Is "keyboard & mouse action" correct for Figure 2? As far as I understand, this game is controlled by predefined actions (e.g., gb_left, gb_jump, gb_action)?
It would be helpful if the authors provide some failure patterns regarding mismatch btw executable checks vs. human evaluation for 41/452 incorrect assertions.
One thing I'm not sure: whether model failing at "certified routes" especially during brief-to-game task means (1) there's a problem in the generated game (which is intended), or (2) inconsistency btw generated game vs. reference game (i.e., generated game is valid w/o problem, still input sequence is not working here due to systematic difference). For option 2, does "semantic start" can all solve this issue?