acceptodds
Under review as a conference paper at ICLR 2027

Spec2Game Benchmark: Can LLMs Generate Complete Interactive Games from Specifications?

Abstract

Generating an executable program does not necessarily mean that it correctly implements the behavioral requirements specified in natural language. To evaluate large language models' ability to realize detailed specifications as complete interactive programs, we introduce Spec2Game, a benchmark that requires models to generate complete Pygame projects from detailed natural-language game specifications. Spec2Game comprises 15 game families and 150 task instances, with one canonical task and nine controlled rule variants per family, spanning three levels of implementation complexity. Using source-code, runtime, and visual evidence, we evaluate generated projects along four dimensions—Executability, Specification Realization, Code Quality, and User-Facing Quality. Across 14 LLMs and 3,330 generated projects, we find that high executability does not imply faithful specification realization. Component-level analysis further shows that models perform substantially better on Game Element Modeling than on Rule and Mechanism Modeling or Goal and Termination Modeling, indicating that faithfully implementing game rules and termination logic remains a major challenge.

Primary area:
datasets and benchmarks
Keywords:
large language models, code generation, benchmark, specification realization, interactive program generation, game generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7NiVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed Spec2Game benchmark that asks LLMs to generate complete Pygame projects in a fixed predefined order (i.e. not targeting agentic use cases). The target games are classical simple 2D games with controlled rule variants for 15 game families and 150 tasks. The input is a cmoplete specification of the game. The evaluation is across four dimensions: Executability, Specification Realization, Code Quality, and User-Facing Quality.

Critical strengths

None

Critical weaknesses

This is a benchmark paper with a lot of heuristic-based design decisions left unjustified. For example, why considering the 4 dimensions? In Section 4.2, completely arbitrary weights are used to assemble the scores which themselves have inexplicable ranges (some ranges 0-2, other range 0-5). No interesting hypothesis was formulated. The choice of Pygame is unmotivated. The quality, scale, and rigor of the presented benchmark is far inferior compared to prior works such as GameCraftBench or GameDevBench.

The fact that the benchmark assumes a fixed LLM calling order (state.py → game.py → rendering.py → main.py) is highly unrealistic of the game development loop. Prior works largely focused on agentic pipeline where an agent can freely decide to iterate on various components of the game. No execution or visual feedback is ever piped back to the LLM.

Questions for authors

None

Reviewer_AxWvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Spec2Game evaluates specification-to-game generation using 150 tasks in 15 families, each containing a canonical game and nine controlled rule variants. A fixed four-stage pipeline generates state, game logic, rendering, and integration code without execution-based repair. Evaluation separates executability, specification realization, code quality, and visual/dynamic quality. The study includes 14 models on Core-15, six on Full-150, and three generated candidates per model-task pair, totaling 3,330 projects. Reported scores indicate that passing executability checks does not ensure high specification realization, particularly for rules and termination conditions. Prompt ablations, an alternative text judge, and human-assessment analyses examine aspects of evaluation robustness.

Critical strengths
  1. The canonical/variant organization and explicit separation of evaluation dimensions make the benchmark more diagnostic than a single runnable/not-runnable score (Sections 3-4). Public behavioral requirements are distinguished from evaluator-only metadata, and the fixed pipeline limits variation in the generation protocol. The 3,330-project evaluation provides useful breadth within this deliberately constrained setting.
  2. The results show a useful distinction between evaluation dimensions: 87 of the 387 Core-15 projects passing all executability checks still score below 60 on specification realization (Section 5.2). Detailed scoring rules, prompt ablations, a text-judge sensitivity analysis, and human comparisons make these findings easier to inspect and interpret than an aggregate leaderboard alone.
Critical weaknesses
  1. The specification score can conflate implementation failure with failure to elicit evidence. A checkpoint can receive zero when the sampling process does not expose the target behavior or evidence is unavailable, not only when observed behavior violates the requirement. Consequently, low scores for rules or termination may partly reflect probe coverage, especially for behavior requiring particular action sequences or late-game states. The manuscript does not establish component-level false-negative rates using independently verified correct implementations. This limits the strength of the central interpretation about which programming capabilities models lack.
  2. Human validation leaves important coverage gaps. Of 5,550 primary-assignment item records, 2,045 are unrateable; overall D2 and D4 alignment is evaluated on 243 and 295 paired projects, respectively. Claude Sonnet 5, the highest-scoring Core-15 model, is absent. The questionnaire samples D2 checkpoints without independently scoring deviation penalties, and visual-quality agreement is low on the small shared set. The authors disclose these limitations and correctly distinguish candidate-level correlations from model-level rank validation. Nevertheless, without analyzing missingness and auditing the omitted scoring components, the reliability of model comparisons remains incompletely established.
Questions for authors
  1. How are sampling failures distinguished from implementation failures when assigning zero D2 scores? A manually verified set of correct games and controlled faulty variants could quantify evaluator errors, particularly for rules and termination checkpoints.
  2. What explains the unrateable human responses, and how are they distributed across models and dimensions? Extending the audit to the omitted leading model, deviation penalties, and action-question answer keys would help establish whether these gaps affect model rankings.
Reviewer_wnGvOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Spec2Game evaluates whether language models can generate complete 2D games from scratch using Pygame. The benchmark covers 15 classic game families (such as Snake and Pong) with 9 rule variants each, totaling 150 tasks. Models generate the project across four sequential files (state.py, game.py, rendering.py, and main.py) with reasoning turned off and without execution feedback. The entry point also implements an adapter interface so an automated evaluator can step the game and inspect its state.

Games are scored on launch stability (D1), rule realization using test traces compiled by Qwen (D2), code quality (D3), and visual quality assessed by Qwen-VL (D4), combined through an executability gate. Across 14 models on the canonical 15 games and 6 models on all 150 tasks, the paper finds that detailed specifications boost performance by 18 points over brief prompts, while runnable games frequently struggle with gameplay rules and win or loss conditions.

Critical strengths

Game specs concrete enough to grade automatically, paired with controlled rule variants. Benchmarking game implementations is a genuinely difficult task and models fall short in a variety of ways, some very subtle. The seven-part specs here pin down update order, collision geometry, rng seed behavior, and UI transitions. Building rule variants on top of each canonical game is also a good design choice. It makes it much harder for models to rely on memorized Pygame code from pretraining. These specs are in my estimation the most significant contribution of the paper.

Critical weaknesses

The overall harness and benchmark setup strike me as a highly questionable environment in which to grade modern LLMs / coding agents. It is four one-shot API calls made in sequence: Prompt → model outputs state.py Prompt + state.py → model outputs game.py Prompt + state.py + AST summary → model outputs rendering.py Prompt + AST summary → model outputs main.py

Additionally, reasoning is explicitly turned off. Even the LLM Qwen judges Qwen3-235B and Qwen3-VL run at temperature 0 with reasoning disabled, which doesn't make sense to me. I would think you would want max accuracy for the evaluation. And finally, there is zero execution feedback.

Now there is a sense in which every setting is an interesting eval, and one could say well we want to see what happens under these conditions. Fair enough, but then the entire nature of the claims made in the paper would need to reflect that, and should be benchmarked against how the models do in a more standard harness coding agent setting with reasoning. Instead, the paper's claims strike me as more broad. Moreover, these settings are highly incongruent with how recent generations of models are trained. They are trained specifically to code within a harness (it's a fair and interesting line of inquiry as to how sensitive models are to a particular harness), and reasoning is a crucial to performance, hence why many benchmarks plot performance against reasoning level.

Questions for authors
  1. Why should people care about this particular evaluation setting when it differs so significantly from how coding agents are trained and deployed?

  2. In addition to the generic critique, I had coding agents review the codebase and they found issues that seem legitimate to me. These are my questions based on reviewing the model's findings and the code itself:

The paper's central finding is that models often produce games that launch cleanly but fail to follow gameplay rules. Looking at the benchmark codebase in src/code2play/, three implementation issues directly contribute to these failures:

The prompt hides required dictionary keys from the model: In Stage 2, game.py is instructed to expect specific dictionary keys when handling player actions (keys_down, held_keys, mouse_position, dt). But in Stage 4, main.py is told to convert Pygame events into "the generic action command" without ever being given those key names. Because the model has to guess the schema, many games fail when translating user inputs, contributing to the drop from 92% launch success to 61% runtime stability in Figure 3a. This raises a more general concern that when a strict artificial harness is applied, small issues and mismatches can entirely undermine/invalidate the results.

Strict adapter checks zero out working games: The launchability test only runs main.py directly and does not test the adapter interface. Before testing any gameplay rules, an adapter audit checks the interface. If init() returns None, if snapshot() fails basic JSON formatting, or if any field transitions from None to a value during an idle step, the harness assigns a flat zero to both rules (D2) and visuals (D4). In Figure 10b, the paper cites GPT-5.6 Sol to show that runnable games miss rules and that there is a wide range in performance within and across tasks. However, 33 of Sol's 45 runs score an average of 81, while the remaining 12 runs sit on the exact zero floor (around 31.7 overall), matching an adapter interface failure, which would say little about the game implementation itself.

Source-code inspection was disabled across all checkpoints: Section 4.2 states that the evaluator inspects candidate source code for 1,200 checkpoints. In protocol_evaluator.py, the code looks for untranslated Chinese keywords in the detection method. Because the benchmark checkpoints were translated into English, this condition evaluates to false for every single checkpoint, meaning source code was never provided to the evaluator.

Will the authors address each of these points so we can have greater confidence in the correctness of the harness/eval setup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.