acceptodds
Under review as a conference paper at ICLR 2027

GAMEGO: Training Game-Dev Agents with Synthetic Trajectories Anchored in Real-World Assets

Abstract

Recent advances in Large Language Models (LLMs) have demonstrated remarkable capabilities in web front-end execution, with browser-based game generation emerging as a particularly prominent frontier. While previous efforts frequently rely on complex multi-turn workflows or focus on static game evaluation benchmarks, this work targets direct end-to-end real-world game synthesis driven by coding agents. However, generating complex games directly from sparse user queries often forces coding agents to make underspecified assumptions, yielding incomplete mechanics, disconnected gameplay flows, and limited visual aesthetics. To resolve this issue, this paper presents GameGo, a scalable framework that systematically transforms brief game seeds into comprehensive Product Requirements Documents grounded in industry game-development practices. To retain core gameplay constraints without restricting design exploration, GameGo uses task-specific dynamic compression to maximize information density while preserving instruction following. Based on this pipeline, GameGoData is constructed with 55,060 development trajectories across 2D, 2.5D, and 3D games, alongside GameGoBench, a benchmark comprising 124 diverse game queries. Training GameGoCoder on GameGoData yields a model that outperforms matched baselines and is comparable to frontier models across gamedev benchmarks. All code, datasets, and models will be made publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Coding Agent, Code Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_C8L6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

GameGo is a data-synthesis pipeline for training coding agents to generate browser games from short, under-specified user queries. The pipeline first collects about 70,000 game records from sources such as Steam and browser-game portals, then deduplicates them at the gameplay level using LLM-extracted gameplay information. Each seed is expanded into a detailed Product Requirements Document (PRD) through staged task definition, playable blueprinting, and asset grounding. The PRD is then compressed into a concise, task-adaptive query that keeps essential gameplay dependencies and leaves routine design choices to the agent. The authors also introduce GameGoBench, a 124-task held-out benchmark scored by an interactive multimodal agent that plays each game, and fine-tune Qwen-based 27B models (GameGoCoder) on the data. These models show substantial gains for the weaker backbone (Qwen 3.5) and smaller gains for the stronger one (Qwen 3.8), with improvements on public benchmarks as well. Human evaluators preferred the compressed queries over both full PRDs and direct queries, though the overall results remain below several strongest frontier models.

Critical strengths
  1. The task is novel and well-motivated.

  2. The overall pipeline design is concrete and may be useful for other domains.

  3. This paper proposes datasets that can be useful for future research.

Critical weaknesses
  1. This paper is very difficult to read from many aspects:

(1) There are many concepts that the authors fail to give a concise definition. For example, though how "game seed" is collected is introduced in Section 3.1, but what exactly does this game seed look is not mentioned. Does it refer to a cluster of records, information extracted from an individual record, or anything else? This is a common issue across Section 3, other concepts such as "mechanics" in line 186, "behavioral specification" in line 196, etc. are not very explained. The authors should either explain what those are clearly, or give concrete examples.

(2) From Section 3.3, the authors do not explain how queries is constructed.

(3) The GsG_s defined in Section 3.3 is not clear. I can briefly guess what the authors want to say, but the details are missing. What is considered as a "clause", why do we care about clauses, what is considered a "fully captured clause", and how GsG_s is used in constructing the queries?

(4) The difficulty levels in Section 4.2 decided? The authors only mention that they are based on implementation complexity, but do not explain how to calculate implementation complexity.

(5) What is the teacher coding agent in Figure 2 and how it is used?

  1. The evaluation metrics of GameGoBench lack human verification. The Requirements and Quality metrics here are very objective, and LLMs are not very good scorer if you ask them to score between 0-100 given a rubric. Though the authors conduct human evaluation on whether the game generated by the post-trained models are more preferred, the validity of the metrics themselves are unknown. Humans scores should be used to verify the validty of those LLM judges, though I doubt that even human experts cannot agree with each other on those metrics.

  2. How is the Overall metric in the last column of Table 1 computed?

Questions for authors

There are many key information that are missing in this paper (Please refer to the weakness part), making this paper difficult to understand. Please clarify the methodology of the paper, at least answering what I pointed out, though I was not able to list everything that are ambiguous.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer C8L6

Comment 1/3: W1 & Q (definitions)

We thank the reviewer for the careful reading. We have uploaded a revised PDF that incorporates each requested clarification directly into the paper, marked in blue. The revisions span Sec. 3 (definitions), Sec. 4.2 (difficulty assignment rules, scoring protocols, and the Overall formula), and the newly added Appendix G.4 (an end-to-end walkthrough tracking a task from seed to query). Each response below concludes with the specific location of its revision.

We acknowledge that Sec. 3 was difficult to follow. To resolve this, each key term is now explicitly defined where it first appears.

(1) Seed, mechanics, behavioral specification.

  • Seed: a single cleaned source record comprising its title, description, metadata, and screenshots.
  • Mechanics: the atomic actions available to the player and the corresponding state-transition rules triggered by each action. In the catch game illustrated in Appendix G.4, dragging translates the hoop, and a ball scores points exclusively upon intersecting the hoop boundary.
  • Behavioral specification: the sequential play phases spanning from initial start to restart, where each phase governs specific player actions, system events, and termination conditions. In the same game, a single missed catch terminates the run, while restarting resets the accumulated score.

Revised in the PDF: Sec. 3.1 (seed), Sec. 3.2 "Task definition" (mechanics) and "Playable blueprinting" (behavioral specification).

(2) Query construction. An LLM autonomously synthesizes the execution query from the comprehensive PRD. It parses the rendering format (2D, 2.5D, or 3D) and estimates task complexity, allocating a proportionally longer query budget to more intricate games. A requirement is preserved strictly if its omission would alter the core game identity or render the game unplayable. All remaining implementation details are deliberately deferred to the downstream coding agent. Independent LLM validators confirm that no essential constraints are omitted and that sufficient design latitude remains open, triggering iterative query rewriting upon failure. The coding agent receives exclusively the query and never accesses the PRD.

Revised in the PDF: Sec. 3.3, first two paragraphs. Full policy detailed in Appendix C.4.

(3) Clause and G_s.

  • Clause: an individual rule within the PRD formulated as a condition-action pair alongside its parameters, such as "when the timer reaches zero, display the results screen." We track clause counts because raw length metrics fail to capture which specific rules are discarded by a compressed query.
  • Fully captured: a condition where the query states the rule along with all its associated parameters and boundary conditions, meaning that merely mentioning the rule while dropping a numeric threshold or condition does not qualify.
  • G_s: the ratio of fully captured clauses divided by the proportion of retained PRD tokens. A value of Gs>1G_s > 1 demonstrates that the query preserves rules at a higher information density than the original PRD.
  • Role: GsG_s is computed strictly post-construction solely for quantitative analysis and is never utilized during the query generation process itself.

Revised in the PDF: Sec. 3.3, final paragraph.

(4) Difficulty. An LLM assigns each task an implementation complexity score ranging from 1 to 10 prior to model execution, after which the authors manually verify every assigned score. This rating accounts for rendering complexity, mechanical coupling, state dependencies, and content breadth. Tasks are subsequently stratified into Easy, Medium, and Hard tiers within each rendering format.

Revised in the PDF: Sec. 4.2 "Task Source". The specific evaluation factors and per-format cut-offs appear in Appendix C.7 (Table 5).

(5) Teacher. Claude Opus 5, deployed as the primary coding agent.

Comment 2/3: W2 (human validation of the metrics)

We agree that user preference for our models does not inherently validate the underlying evaluation metrics. Accordingly, we conducted the rigorous validation study suggested by the reviewer.

How the scores are built. The verifier never outputs a direct 0–100 aggregate score. Instead, one agent executes the game within a browser environment while a secondary agent independently reviews the recorded execution trace. Requirements scores derive from four binary pass-or-fail checks evaluated hierarchically per level: the presence of the requested game archetype, functional input responsiveness, a demonstrable game loop, and the successful fulfillment of the query's most specific constraint. Quality scores rely on integer ratings ranging from 0 to 5 assigned to each visual factor across chronologically sampled frames rather than hand-picked snapshots.

Revised in the PDF: Sec. 4.2 "Artifact Evaluation" now explicitly details these scoring mechanisms.

Human validation study (new). Three domain experts evaluated 80 games spanning 20 tasks across all formats and difficulty levels, generated by 4 distinct models. These experts observed the identical video recordings as the verifier and applied identical criteria and visual factors without access to verifier scores or model identities. We evaluate whether the verifier aligns with human expert judgments comparably to the internal consensus among the experts themselves, with brackets denoting 95% confidence intervals derived from bootstrap resampling over tasks:

Evaluation MetricHuman–Verifier AgreementHuman–Human Agreement
Req. checks, Cohen's κ\kappa (4 checks pooled)0.58 [0.50, 0.66]0.63 [0.56, 0.69]
Req. score, Spearman correlation0.75 [0.63, 0.85]0.77 [0.67, 0.85]
Quality score, Spearman correlation0.68 [0.55, 0.78]0.74 [0.63, 0.83]

The automated verifier aligns with human experts nearly as well as the experts agree with one another, with their confidence intervals overlapping across every metric. The most challenging verification criterion for all evaluators is the query's most specific requirement, yielding a Cohen's κ\kappa of 0.51 for human-verifier and 0.54 for human-human evaluations.

Human evaluations confirm identical model rankings. All four models maintain the exact relative ranking order across Requirements, Quality, and Overall metrics. While the automated verifier assigns absolute scores approximately 3.5 points higher than human experts, this offset remains uniform across all evaluated models. Based strictly on human expert scores, GameGoCoder 3.5 surpasses its base model by 15.1 points in Overall performance, closely mirroring the 15.9-point gain registered under verifier scoring.


Comment 3/3: W3 (Overall in Table 1)

Overall is defined as the arithmetic mean of Requirements and Quality, computed per task and subsequently averaged across all tasks, while Execution Rate is reported independently. For instance, GameGoCoder 3.8 achieves an Overall score of (66.94+43.64)/2=55.29(66.94 + 43.64) / 2 = 55.29.

Revised in the PDF: Sec. 4.2 "Artifact Evaluation" now explicitly provides this formula.

We would be more than happy to continue our discussion if you have any further questions, and we have uploaded a revised PDF incorporating these changes, which you are welcome to consult if needed.

Reviewer_PcCYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

GameGo constructs game-development training tasks from real game descriptions. It expands each seed into a product requirements document covering gameplay, state transitions, and assets, then compresses that document into a task-adaptive query for a coding agent. Teacher executions produce GameGoData, with 55060 development trajectories. Two 27B models are fine-tuned on this corpus are evaluated on the new 124 tasks GameGoBench and two external benchmarks. The trained models improve over their matched base checkpoints, while a fixed-teacher comparison finds that compact queries produce preferred games more often than direct queries or full PRDs. The paper also analyzes query compression, human preferences, and verifier sensitivity.

Critical strengths

The query-format comparison is fairly controlled. All three formats use the same teacher, scaffold, and execution budget; the compact queries are also similar in length to direct queries. Human evaluators prefer compact queries over full PRDs by 487 wins to 81 losses, with 176 ties. This supports the practical value of separating planning from execution instructions.

The corpus has demonstrated training utility, not just better screenshots. Both backbones improve on GameGoBench and the two public benchmarks, with additional human comparisons and three-verifier checks. The gain is much larger for the 3.5 backbone, but the broader evidence is still useful for a systems/data contribution.

Critical weaknesses

The experiments do not isolate why this construction pipeline produces better training data. Students are trained only on the task-adaptive trajectories; the direct-query and full-PRD ablation compares teacher artifacts, not students trained with those alternatives (Appendix D). A small matched-size SFT experiment, or a length- and compute-matched ordinary PRD-summary baseline, would help separate the effect of the proposed construction from domain-specific teacher distillation and extra planning compute.

The release boundary is unclear for a paper whose main contribution includes a training corpus. The abstract promises public data, but Appendix H says the accompanying implementation release excludes GameGoData trajectories and artifacts. The teacher is also confidential. Without inspectable supervision and enough collection detail, independently auditing or reproducing the training gains will be difficult. But overall the contribution is sound and the method description is clear.

Questions for authors
  1. Is there a smaller matched-data SFT comparison that isolates task-adaptive construction from collecting the same number of teacher trajectories using simpler queries?

  2. Will the actual training trajectories and artifacts be released, along with production prompts and sufficient teacher information to reproduce collection?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer PcCY

Comment 1/2: W1 & Q1 (matched-data SFT comparison)

We thank the reviewer for the supportive assessment and the concrete suggestion.

Setup. As suggested, we ran a smaller matched-data comparison. From the same seed pool and with the same teacher and scaffold, we collect 5,000 trajectories for each representation and fine-tune Qwen3.5-27B on each set for 2 epochs, evaluated as in Table 1.

Training dataExec.Req.QualityOverall
Qwen3.5-27B (no SFT)91.8036.8931.1234.00
+ Direct queries, 5k94.3343.0039.1841.10
+ Full PRDs, 5k95.7845.7440.2242.99
+ Task-specific queries, 5k97.2452.8242.5347.68
  • Teacher distillation alone (direct queries) gives +7.1 over the base model. Task-specific construction adds a further +6.6 at the same data size.
  • Prompt length: direct queries are length-matched to task-specific ones (median 227.5 vs. 235 tokens, Appendix D), so the gain is not a length effect.
  • Extra planning compute: full PRDs carry the full output of the same planning stages but score 4.7 lower, so the gain comes from selecting what to pass on, not from planning compute itself.

The student ranking (direct < full PRD < task-specific) matches the artifact preferences in Table 2.


Comment 2/2: W2 & Q2 (release and teacher)

Release. Upon acceptance, we will release the GameGoCoder models, GameGoData (training queries, trajectories and artifacts), the full data-construction pipeline with production prompts, and GameGoBench with its evaluation setup.

Revised in the PDF: we removed the outdated sentence in Appendix H that excluded trajectories, Appendix H now matches the abstract.

The teacher is Claude Opus 5.

We would be more than happy to continue our discussion if you have any further questions, and we have uploaded a revised PDF incorporating these changes, which you are welcome to consult if needed.

The discussion period between the reviewers and the authors will conclude within two days. We hope that our responses have addressed your questions. We highly welcome your further feedback and are willing to clarify any unresolved issues for you.

Most issues/concerns are addressed. I maintain my score and thanks the authors for their detailed responses.

Reviewer_GT14OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper develops a synthetic-data pipeline for browser-game coding agents: real-world seeds and assets are expanded into product requirements documents (PRDs), compressed into task-specific instructions, and executed to collect training trajectories. It releases a proposed 55,060-trajectory corpus and a 124-task benchmark, and fine-tunes two 27B backbones. The strongest evidence is improvement over matched base models on three benchmarks, complemented by blinded human preferences.

Critical strengths
  1. A substantial end-to-end resource spanning executable trajectories, assets, and interactive evaluation, rather than screenshots alone.
  2. Evaluation includes two external game subsets and matched base models, reducing dependence on the newly introduced benchmark.
  3. Six blinded expert evaluators complement automated judgments; the paper also documents evaluator sensitivity and agreement rather than relying exclusively on one judge.
  4. The compression analysis reveals which constraints survive and connects task complexity to instruction length.
Critical weaknesses

Limited evidence for complex, real-world game development. Although the paper motivates GameGo through real-world game synthesis and industry-inspired development practices, the evaluation primarily establishes its effectiveness on browser-game generation benchmarks. Execution success, requirement satisfaction, visual quality, and human preference are useful measures, but they do not sufficiently characterize the challenges of developing complex games, such as interactions among multiple gameplay systems, sustained progression, robustness over extended play, and iterative modification of an existing codebase. The concern is therefore less about the number of benchmarks than about their coverage of the capabilities implied by the paper’s framing. More demanding case studies with explicit complexity criteria and detailed failure analysis would help establish the scope of the claimed capability.

Limited isolation of generalizable insights beyond the integrated pipeline. The paper presents a substantial data-generation pipeline, but it remains unclear which design choices are essential and what transferable principles emerge from the results. The finding that task-specific queries outperform both direct queries and full PRDs is promising; however, the comparisons do not disentangle the effects of planning, information selection, prompt length, asset grounding, and implementation freedom. Moreover, these representation comparisons evaluate generated artifacts rather than models trained on matched datasets constructed through the alternative routes. Consequently, the evidence supports the effectiveness of the overall pipeline more strongly than it explains why it works or when its design principles should generalize. Targeted, budget-matched ablations would make the scientific takeaway substantially clearer.

Questions for authors

Is there any evidence to prove the assets or game application could be comparable with real world assets?

6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer GT14

Comment 1/3: W1 (evidence for complex, real-world game development)

We thank the reviewer for pinning down the concern. Our claim is scoped to single-turn, from-scratch browser game generation. "Real-world" refers to the task sources (Steam, 14 browser-game websites, WeChat mini-games) and the industry-style synthesis process (Sec. 3), not to developing commercial-scale games.

Revised in the PDF: Sec. 1 now states this scope more clearly.

Explicit complexity criteria. Each benchmark task is scored 1–10 on rendering complexity, coupling between mechanics, state dependencies and content breadth, then split into Easy, Medium and Hard (Sec. 4.2, Appendix C.7). The most complex Hard tasks specify 12–14 ordered phases and 20–24 mechanics.

Does the gain hold on complex tasks? Δ = GameGoCoder − base, paired bootstrap 95% CI:

TiernQwen3.5 ΔQwen3.8 Δ
Easy22+2.9 [−14.3, +20.9]+0.7 [−13.6, +16.5]
Medium70+11.8 [−0.9, +24.2]+6.7 [−2.4, +16.6]
Hard32+17.3 [+3.9, +31.0]−3.4 [−17.5, +10.3]

For Qwen3.5, the gain grows with complexity and is significant on Hard tasks. For Qwen3.8 the gain is on Medium tasks, with none on Hard.

Case studies: interacting systems and sustained progression.

  • Success (steam_20553: 3D shooter, 14 phases, 24 mechanics). The base model stopped after 1 step. GameGoCoder 3.8 wrote 3,299 lines across five modules, the countdown, three workstation panels, tool heat and drone respawns all work together, and all four requirement checks pass.
  • Failure (steam_2693: Hard 3D squad shooter, 12 phases, 20 mechanics). The game deadlocks at the first phase transition: a loading overlay waits for HUD state that is published only in the next phase, and it blocks the click needed to enter that phase.
  • Failure (steam_41885: 2D exploration with turn-based combat). The world module passes an entity ID where combat expects an enemy type, so the first enemy contact crashes the game, the combat, boss and ending phases are never reached.

Failure analysis. In both failures every module works on its own, but the state hand-off between modules is wrong. This is the dominant failure on the hardest tasks: training teaches the model to build each required system, not to keep their interfaces consistent.

Robustness over extended play (new). We replay all 248 Qwen3.8 base and GameGoCoder 3.8 builds under 5 minutes of seeded input. At least 96% of running builds per model never crash, and no build hangs or crashes the page. GameGoCoder has slightly more mid-play crashes (4/106 vs. 1/108, Fisher p = 0.21), in line with the hand-off failures above. Extended play also catches late failures that the verifier's short session misses, such as steam_41885.

Iterative modification of an existing codebase. Not evaluated: every task starts from a fresh project template.

Revised in the PDF: the new Limitations section (Sec. 7) states this explicitly.

Comment 2/3: W2 (isolating generalizable insights)

We ran new experiments for them.

Students trained on matched data (new). For each task representation, we collect 5,000 teacher trajectories from the same seed pool and fine-tune Qwen3.5-27B for 2 epochs, evaluated as in Table 1.

Training dataExec.Req.QualityOverall
Qwen3.5-27B (no SFT)91.8036.8931.1234.00
+ Direct queries, 5k94.3343.0039.1841.10
+ Full PRDs, 5k95.7845.7440.2242.99
+ Task-specific queries, 5k97.2452.8242.5347.68

What each comparison isolates.

  • Prompt length: direct and task-specific queries have similar lengths (median 227.5 vs. 235 tokens, Appendix D), yet differ by +6.6 Overall, so the gain is not a length effect.
  • Planning: full PRDs add +1.9 over direct queries.
  • Information selection and implementation freedom: task-specific queries, built from the same plan, add +4.7 over full PRDs.
  • Asset grounding (artifact-level): the full PRD is preferred over a PRD without Stage 3 asset grounding (207 / 93 / 72 W / T / L, preference 0.742).
  • Adaptive budget (artifact-level): adaptive compression is preferred over a fixed, complexity-independent budget (251 / 65 / 56, preference 0.818).

The last two follow the Table 2 protocol on all 124 benchmark tasks, with 3 raters (372 judgments each), preference = W / (W + L), ties excluded.

Planning helps, but selecting what the coding agent cannot infer helps more. The gain does not come from longer instructions, instruction budgets should scale with task complexity, and asset contracts should be fixed before coding.

Comment 3/3: Q (comparison with real-world games)

Term. "Real-world assets" refers to the real game records used as seeds (descriptions, metadata, screenshots). No art is imported from the source games, visuals are made procedurally in code or by an image-generation model, following each PRD's asset contract (Sec. 3.2, Appendix H.3).

Revised in the PDF: Sec. 1 now defines "real-world assets" more clearly.

Blinded comparison with the seed originals (new). We built 20 pairs, each a GameGoCoder 3.8 game vs. the original its seed came from, randomized and unlabeled. All 9 web originals are playable. For 11 randomly sampled Steam games only store screenshots and trailers exist, so only visuals are compared. Three raters score each side on 1–5 (means below) and pick the better one.

GroupVisuals: generated vs. originalPlayability: generated vs. originalOverall preference (pairs)
Web games, 9 pairs3.88 vs. 3.964.08 vs. 4.008 comparable, 1 original
Steam games, 11 pairs4.00 vs. 4.27n/a10 comparable, 1 original

On gameplay-driven web games, generated games are on par with the originals, on art-heavy commercial games, the originals look better. This is a preliminary study (3 raters, Cronbach's α = 0.72 over the 20 pairs).

Revised in the PDF: the Limitations section (Sec. 7) names external asset libraries as future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the Area Chairs and reviewers for their careful evaluation and constructive feedback. The reviews recognized the novelty and motivation of the task, the value of GameGoData and GameGoBench as resources, the controlled query-format comparison, and the evaluation on external benchmarks with blinded human judgments.

We are encouraged that Reviewer PcCY confirmed that our responses addressed their concerns. We also addressed the concerns of Reviewers C8L6 and GT14 through detailed responses, new experiments, and manuscript revisions, though we regret that we have not yet received further feedback despite our follow-up.

  • Shared concern (PcCY, GT14): isolating why the pipeline produces superior training data. We trained student models on 5,000 matched trajectories per task representation. Task-specific queries outperform length-matched direct queries by 6.6 Overall points and full PRDs by 4.7 points, demonstrating that the gains stem from selecting what to pass on rather than from prompt length or planning compute.
  • Reviewer C8L6: we defined all key concepts in Sec. 3 (seed, mechanics, behavioral specification, clause, GsG_s, query construction, and difficulty), added the Overall formula and an end-to-end walkthrough, and conducted a human validation study across 3 experts and 80 games. The verifier agrees with human experts nearly as well as experts agree with each other and preserves the same model ranking.
  • Reviewer GT14: we clarified the scope of our claims, added a Limitations section, and provided a per-difficulty analysis with case studies and failure analysis, a 5-minute extended-play robustness test, and a blinded comparison against the original games.
  • Reviewer PcCY: we provided the matched-data SFT comparison, clarified the release scope (models, GameGoData trajectories and artifacts, the full pipeline with production prompts, and GameGoBench), updated Appendix H accordingly, and disclosed the teacher model.

The revised manuscript is accessible via the submission's PDF link, with all revisions highlighted in blue. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.