- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed Spec2Game benchmark that asks LLMs to generate complete Pygame projects in a fixed predefined order (i.e. not targeting agentic use cases). The target games are classical simple 2D games with controlled rule variants for 15 game families and 150 tasks. The input is a cmoplete specification of the game. The evaluation is across four dimensions: Executability, Specification Realization, Code Quality, and User-Facing Quality.
Critical strengths
None
Critical weaknesses
This is a benchmark paper with a lot of heuristic-based design decisions left unjustified. For example, why considering the 4 dimensions? In Section 4.2, completely arbitrary weights are used to assemble the scores which themselves have inexplicable ranges (some ranges 0-2, other range 0-5). No interesting hypothesis was formulated. The choice of Pygame is unmotivated. The quality, scale, and rigor of the presented benchmark is far inferior compared to prior works such as GameCraftBench or GameDevBench.
The fact that the benchmark assumes a fixed LLM calling order (state.py → game.py → rendering.py → main.py) is highly unrealistic of the game development loop. Prior works largely focused on agentic pipeline where an agent can freely decide to iterate on various components of the game. No execution or visual feedback is ever piped back to the LLM.
Questions for authors
None