acceptodds
Withdrawn from ICLR 2027

SWE-Game: Can Coding Agents Build the Games We Want?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

20 views

Abstract

We introduce SWE-Game, a benchmark of 247 tasks grounded in 41 executable reference Godot games spanning 13 gameplay categories in 2D and 3D. Five task types cover development from a brief, implementation from a game design document, skeleton completion, repair of 83 injected-fault cases, and Godot-to-Unity porting. Reference materials specify the intended gameplay, while a shared instrumentation interface lets evaluator-owned drivers and probes execute actions and observe independently implemented games. Evaluation combines engine-state checks, certified reference-input replay, and agent-authored feature demonstrations to assess mechanic correctness, demonstrated playability, and behavioral restoration and preservation after repairs. Game-specific vision-language rubrics separately assess presentation. Across six models, Opus5 achieves the highest overall score in all five task types. Best overall scores remain below 60 out of 100 across the three construction tasks, with Brief-to-Game reaching 50.38. Analysis of reviewed submissions identifies requirement omissions and gameplay logic errors as predominant implementation problems. On human-labeled behaviors from 100 agent-built games, executable checks achieve 92.59% balanced accuracy, compared with 78.41% for a video-based VLM judge. Rubric-based visual scores reach a Spearman correlation of 0.829 with human ratings of 200 gameplay clips. Together, these results characterize current agent capabilities across game-development activities and support combining runtime evidence with visual assessment.

Primary area:
datasets and benchmarks
Keywords:
Coding agents, Game development, Software engineering benchmarks, Execution-based evaluation, Multimodal evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bL51OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SWE-Game, a new game-dev benchmark of 247 tasks grounded in 41 executable Godot games for coding agents. While existing benchmarks have focused on evaluating generated games' quality with either VLM judges or simple unit tests, SWE-Game diversifies the evaluation metrics to measure various aspects of mechanic correctness, as well as to use VLM judges. Across six models, Opus 5 achieves the best performance, but the overall score remains below 60 out of 100 across the tasks. The paper also compared its evaluation methods to human annotations, which showed high agreement above 90% balanced accuracy (especially for executable checks).

Critical strengths

First, this benchmark is well-built and can evaluate the coding agent's capabilities in several ways, including generating from a brief description, generating from a GDD, fixing bugs, and even porting from Godot to Unity. In addition, the evaluation is thorough because the authors introduced not only functional evaluation (with a new game instrumentation interface) but also visual evaluation. Across 5 different tasks with 2 different evaluation methods, I believe that this benchmark is well balanced to assess diverse capabilities of coding agents. In addition, their analysis regarding automatic eval vs. human eval ensures that this evaluation method is valid and precise.

Also, the experiments and analysis on 6 coding agents/models show that existing models are far from generating or rebuilding the game in the way that the user wants to build. These results imply the necessity of the benchmark, which is a critical strength, I believe.

Critical weaknesses

Importantly, this paper does not have any link to the code implementation, which blocks transparent analysis/reproducibility.

Furthermore, there's no further analysis of the proposed benchmark (which seems necessary), such as:

  • whether the games included are prone to contamination issues
  • performance between 2d vs. 3d games, or genre-specific scores, or even the relationship between game complexity vs. model scores.
  • Especially for "certified routes", I'm not sure if all the details are included in the paper (e.g., # of registered routes per game, length of routes, # of actions per route, ratio of route that uses semantic start, etc).

While the paper proposes a lots of notations and concepts, it's hard to follow up on all the details, especially in terms of readability. It would be better to make a bold/italic for new terms or notations.

Questions for authors

Is "keyboard & mouse action" correct for Figure 2? As far as I understand, this game is controlled by predefined actions (e.g., gb_left, gb_jump, gb_action)?

It would be helpful if the authors provide some failure patterns regarding mismatch btw executable checks vs. human evaluation for 41/452 incorrect assertions.

One thing I'm not sure: whether model failing at "certified routes" especially during brief-to-game task means (1) there's a problem in the generated game (which is intended), or (2) inconsistency btw generated game vs. reference game (i.e., generated game is valid w/o problem, still input sequence is not working here due to systematic difference). For option 2, does "semantic start" can all solve this issue?

Reviewer_DJECOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SWE-Game presents a benchmark for end to end game development. It presents 247 tasks across 41 godot games and evaluates them for functionality and visual fidelity. Inputs include a full game design doc (GDD), a brief design description, and/or video. Evaluation is done via execution of the game itself.

Critical strengths

SWE-Game presents a few novel contributions:

  1. it is one of the first true end-to-end agentic game development benchmarks.
  2. it uses actual execution in its evaluation rather than relying on static code LLM judgements.
  3. it provides a variety of interesting ways to evaluate games, including visuals, functional correctness, and bug repair
  4. it includes some human evaluation comparions which is crucial for papers which use llm judge techniques.

These seem few, but is actually a big step towards what I believe is the right formulation for end to end game development benchmarking.

Critical weaknesses

The following are the weaknesses:

  1. Simply not enough information is provided. At its core, a benchmark paper needs to demonstrate clearly where its dataset comes from, why it's accurate / correct, and explain how that data provides insights It is incredibly unclear where most of the data. For example, I read some of the GDDs and they don't seem like a real GDD written by a game developer?
  2. It uses llm judges and execution throughout its evaluation, but doesn't provide enough support to indicate why this is better than llm judge on code directly. There is no comparison here despite it being the obvious, simpler choice.
  3. After reading the paper, I'm not left with any new and interesting insights. Why does this matter? What does Opus 5.5 scoring 8x% on your benchmark mean and why is that important? What should I learn from the failures? There's one small paragraph on failure modes but I find that insufficient.
  4. Lastly, the paper clearly uses LLMs heavily for its writing. It is oftentimes very difficult to read as certain wording is unclear. For example, a "Retained" route is never defined in the paper despite being repeatedly used. There's some brief sentence explaining how a route is retained and table deep in the appendix, but never defined despite its usage as a defined term.
  5. smaller issue: the raw task count seems to be small for each type since there's only 247 tasks total; this makes sub-task understanding difficult / likely not significant.

For (2) I would suggest citing the following two works that recently came out to at least have some support as to why execution is necessary and better than static LLM judge on code.

[1] Interaction-as-a-Judge: https://arxiv.org/abs/2610.17895 [2] The Verification Horizon: https://arxiv.org/abs/2606.26300

Ultimately, the weaknesses means the novel idea is not executed quite well enough to be an ICLR paper in its current form.

Questions for authors
  1. Game licensing? Are these open source games?
  2. Where are the briefs coming from? The game design documents? Are these designed by the game designer itself? How can we ensure that you can even achieve 100% accuracy with the briefs or game design documents? Same question for all of the rest of the data as well.
Reviewer_gD2zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces SWE-Game, a benchmark for testing coding agents on game-development tasks. It contains 247 tasks based on 41 Godot games. The five task types are building a game from a short brief, implementing a game design document, completing unfinished code, fixing injected bugs, and converting a Godot game to Unity.

The benchmark uses automated tests to check game behavior and a VLM to judge visual quality. Six model agent setups are evaluated. Opus 5 achieves the highest scores across all five task types, but its scores for building games remain below 60/100. The authors also compare their evaluation with human judgments, reporting 92.59% balanced accuracy for functional checks and a correlation of 0.829 between visual scores and human ratings.

Critical strengths
  1. Coverage of several meaningful game-development activities. The benchmark extends evaluation across construction, completion, repair, and engine migration. This range of tasks is useful because game development involves maintaining and adapting interactive systems as well as generating initial implementations. The executable reference projects provide a concrete foundation for evaluating these activities.
  2. Combining runtime evidence with visual assessment. Checking engine state and input-dependent behavior addresses an important limitation of judging games solely from their appearance. The comparison with human judgments provides useful evidence for this approach, particularly the functional validation on 1,200 assertions from 100 agent-built games and visual validation on 200 clips.
Critical weaknesses
  1. Several important details are missing, making the benchmark difficult to reproduce and use. It is unclear who created the reference game design documents and how the injected bugs were originally designed. The VLM judge is not named, and the paper does not clearly describe frame sampling or the settings used for each coding model. Although a formula is provided for repair scores, it is unclear how its individual values are calculated and how partial repairs receive credit. The human evaluation procedure also needs more detail. The many task rules, files, and scoring choices make the benchmark difficult to follow.

  2. The results are difficult to interpret beyond the tested setup. The benchmark uses 41 relatively small projects, so it is unclear how well it represents work on larger games with more code and assets. The paper should explain how much the authors added to the original projects, particularly if some began as small demos or templates. In addition, the scoring weights are not clearly justified, the models use different agent frameworks, and some task comparisons use different scoring rules. These choices make it difficult to draw general conclusions about which model is better at game development.

Questions for authors
  1. Where did the reference GDDs come from? Were they already available with the original games, written by the authors, generated by AI, or created using a combination of these methods? How were they checked? Please explain what the authors added to the original games and clearly distinguish these reference GDDs from the GDDs written by agents in Brief-to-Game.
  2. How were the bugs created? Were the injected bugs designed by humans, generated by an agent, or added using fixed rules? If an agent was used, which model and prompts were used? How did the authors decide whether these bugs were similar to bugs found in real game development?
  3. How is repair quality measured? Section 3.4 explains how a faulty case is checked before inclusion, and Appendix D.1 gives the final scoring formula. However, how are the values in that formula calculated? For example, is restoration measured by the percentage of broken checks that pass after repair? Do all checks have the same weight? How much credit is given when an agent fixes only part of a bug? Please provide an example showing the checks before and after repair and the resulting score.
  4. Which VLM was used as the judge (section 3.4), and why? Please provide the exact model and version used for visual scoring and the video-based VLM comparison. Why were these models selected?
  5. How were the reference videos recorded and shown to the models? Were the games played by humans or scripted agents? How were the gameplay routes selected, and how long was each recording? Figure 3(b) only shows duration ranges. Did the videos show button presses or include player-input records? Were these available to the coding agents, the VLM judge, or both? Please also explain the frame-sampling rate, image resolution, and number of frames used for VLM evaluation.
  6. Why were the scoring weights chosen? What is the reason for assigning 85% weight to functional/design performance and 15% to visual quality in Equation 1? How were the other task-specific weights chosen? Were they checked against human judgments, and would changing them change the model rankings?
  7. Please discussion how these model versions and settings selected. Why was GPT‑5.6 Luna used rather than a stronger version such as Sol? Were models selected based on capability, cost, or available resources? Please report the exact model identifiers, thinking settings, effort levels, token and time limits, agent framework versions, and tools.
  8. The paper states that every model scores higher on GDD-to-Game than Brief-to-Game. Since these tasks have different inputs and scoring rules, what conclusion should readers draw from this?
  9. How was the human evaluation carried out in Section 4.5? The paper reports human labels for 1,200 assertions from 100 games and human ratings for 200 gameplay clips, but the judging process is unclear. How many people participated, and what experience did they have with game development? How were the games and clips selected? What instructions, scoring rules, and rating scales did the judges receive? Did multiple people judge each example, and how were disagreements resolved? Were judges shown the model names, automated test results, or VLM scores? Please also report how often the human judges agreed with each other.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.