- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
GameGo is a data-synthesis pipeline for training coding agents to generate browser games from short, under-specified user queries. The pipeline first collects about 70,000 game records from sources such as Steam and browser-game portals, then deduplicates them at the gameplay level using LLM-extracted gameplay information. Each seed is expanded into a detailed Product Requirements Document (PRD) through staged task definition, playable blueprinting, and asset grounding. The PRD is then compressed into a concise, task-adaptive query that keeps essential gameplay dependencies and leaves routine design choices to the agent. The authors also introduce GameGoBench, a 124-task held-out benchmark scored by an interactive multimodal agent that plays each game, and fine-tune Qwen-based 27B models (GameGoCoder) on the data. These models show substantial gains for the weaker backbone (Qwen 3.5) and smaller gains for the stronger one (Qwen 3.8), with improvements on public benchmarks as well. Human evaluators preferred the compressed queries over both full PRDs and direct queries, though the overall results remain below several strongest frontier models.
Critical strengths
-
The task is novel and well-motivated.
-
The overall pipeline design is concrete and may be useful for other domains.
-
This paper proposes datasets that can be useful for future research.
Critical weaknesses
- This paper is very difficult to read from many aspects:
(1) There are many concepts that the authors fail to give a concise definition. For example, though how "game seed" is collected is introduced in Section 3.1, but what exactly does this game seed look is not mentioned. Does it refer to a cluster of records, information extracted from an individual record, or anything else? This is a common issue across Section 3, other concepts such as "mechanics" in line 186, "behavioral specification" in line 196, etc. are not very explained. The authors should either explain what those are clearly, or give concrete examples.
(2) From Section 3.3, the authors do not explain how queries is constructed.
(3) The defined in Section 3.3 is not clear. I can briefly guess what the authors want to say, but the details are missing. What is considered as a "clause", why do we care about clauses, what is considered a "fully captured clause", and how is used in constructing the queries?
(4) The difficulty levels in Section 4.2 decided? The authors only mention that they are based on implementation complexity, but do not explain how to calculate implementation complexity.
(5) What is the teacher coding agent in Figure 2 and how it is used?
-
The evaluation metrics of GameGoBench lack human verification. The Requirements and Quality metrics here are very objective, and LLMs are not very good scorer if you ask them to score between 0-100 given a rubric. Though the authors conduct human evaluation on whether the game generated by the post-trained models are more preferred, the validity of the metrics themselves are unknown. Humans scores should be used to verify the validty of those LLM judges, though I doubt that even human experts cannot agree with each other on those metrics.
-
How is the Overall metric in the last column of Table 1 computed?
Questions for authors
There are many key information that are missing in this paper (Please refer to the weakness part), making this paper difficult to understand. Please clarify the methodology of the paper, at least answering what I pointed out, though I was not able to list everything that are ambiguous.
5 comments
Comment 2/3: W2 (human validation of the metrics)
We agree that user preference for our models does not inherently validate the underlying evaluation metrics. Accordingly, we conducted the rigorous validation study suggested by the reviewer.
How the scores are built. The verifier never outputs a direct 0–100 aggregate score. Instead, one agent executes the game within a browser environment while a secondary agent independently reviews the recorded execution trace. Requirements scores derive from four binary pass-or-fail checks evaluated hierarchically per level: the presence of the requested game archetype, functional input responsiveness, a demonstrable game loop, and the successful fulfillment of the query's most specific constraint. Quality scores rely on integer ratings ranging from 0 to 5 assigned to each visual factor across chronologically sampled frames rather than hand-picked snapshots.
Revised in the PDF: Sec. 4.2 "Artifact Evaluation" now explicitly details these scoring mechanisms.
Human validation study (new). Three domain experts evaluated 80 games spanning 20 tasks across all formats and difficulty levels, generated by 4 distinct models. These experts observed the identical video recordings as the verifier and applied identical criteria and visual factors without access to verifier scores or model identities. We evaluate whether the verifier aligns with human expert judgments comparably to the internal consensus among the experts themselves, with brackets denoting 95% confidence intervals derived from bootstrap resampling over tasks:
| Evaluation Metric | Human–Verifier Agreement | Human–Human Agreement |
|---|---|---|
| Req. checks, Cohen's (4 checks pooled) | 0.58 [0.50, 0.66] | 0.63 [0.56, 0.69] |
| Req. score, Spearman correlation | 0.75 [0.63, 0.85] | 0.77 [0.67, 0.85] |
| Quality score, Spearman correlation | 0.68 [0.55, 0.78] | 0.74 [0.63, 0.83] |
The automated verifier aligns with human experts nearly as well as the experts agree with one another, with their confidence intervals overlapping across every metric. The most challenging verification criterion for all evaluators is the query's most specific requirement, yielding a Cohen's of 0.51 for human-verifier and 0.54 for human-human evaluations.
Human evaluations confirm identical model rankings. All four models maintain the exact relative ranking order across Requirements, Quality, and Overall metrics. While the automated verifier assigns absolute scores approximately 3.5 points higher than human experts, this offset remains uniform across all evaluated models. Based strictly on human expert scores, GameGoCoder 3.5 surpasses its base model by 15.1 points in Overall performance, closely mirroring the 15.9-point gain registered under verifier scoring.
Comment 3/3: W3 (Overall in Table 1)
Overall is defined as the arithmetic mean of Requirements and Quality, computed per task and subsequently averaged across all tasks, while Execution Rate is reported independently. For instance, GameGoCoder 3.8 achieves an Overall score of .
Revised in the PDF: Sec. 4.2 "Artifact Evaluation" now explicitly provides this formula.
We would be more than happy to continue our discussion if you have any further questions, and we have uploaded a revised PDF incorporating these changes, which you are welcome to consult if needed.
Rebuttal to Reviewer C8L6
Comment 1/3: W1 & Q (definitions)
We thank the reviewer for the careful reading. We have uploaded a revised PDF that incorporates each requested clarification directly into the paper, marked in blue. The revisions span Sec. 3 (definitions), Sec. 4.2 (difficulty assignment rules, scoring protocols, and the Overall formula), and the newly added Appendix G.4 (an end-to-end walkthrough tracking a task from seed to query). Each response below concludes with the specific location of its revision.
We acknowledge that Sec. 3 was difficult to follow. To resolve this, each key term is now explicitly defined where it first appears.
(1) Seed, mechanics, behavioral specification.
Revised in the PDF: Sec. 3.1 (seed), Sec. 3.2 "Task definition" (mechanics) and "Playable blueprinting" (behavioral specification).
(2) Query construction. An LLM autonomously synthesizes the execution query from the comprehensive PRD. It parses the rendering format (2D, 2.5D, or 3D) and estimates task complexity, allocating a proportionally longer query budget to more intricate games. A requirement is preserved strictly if its omission would alter the core game identity or render the game unplayable. All remaining implementation details are deliberately deferred to the downstream coding agent. Independent LLM validators confirm that no essential constraints are omitted and that sufficient design latitude remains open, triggering iterative query rewriting upon failure. The coding agent receives exclusively the query and never accesses the PRD.
Revised in the PDF: Sec. 3.3, first two paragraphs. Full policy detailed in Appendix C.4.
(3) Clause and G_s.
Revised in the PDF: Sec. 3.3, final paragraph.
(4) Difficulty. An LLM assigns each task an implementation complexity score ranging from 1 to 10 prior to model execution, after which the authors manually verify every assigned score. This rating accounts for rendering complexity, mechanical coupling, state dependencies, and content breadth. Tasks are subsequently stratified into Easy, Medium, and Hard tiers within each rendering format.
Revised in the PDF: Sec. 4.2 "Task Source". The specific evaluation factors and per-format cut-offs appear in Appendix C.7 (Table 5).
(5) Teacher. Claude Opus 5, deployed as the primary coding agent.