acceptodds
Withdrawn from ICLR 2027

GameCode-Bench: Benchmarking Repository-Scale Game Development with Coding Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As large language models rapidly evolve, the capabilities of autonomous coding agents are advancing to new limits. Recent software engineering benchmarks, however, primarily evaluate conventional software projects, while game benchmarks focus on generating miniature, tutorial, or demo-level games. Agents' capacity to handle mature, highly interactive game systems, therefore, remains largely underexplored. To address this gap, we introduce GameCode-Bench, a benchmark for repository-scale gameplay engineering comprising 86 tasks across eight independent, open-source browser game repositories. Our benchmark captures complementary developer- and community-facing development workflows, and agents operate directly within mature codebases containing a median of 162,055 non-test source lines, with the largest exceeding one million. Alongside frontend, backend, and API integration maintenance tasks, GameCode-Bench centers on core game logic and challenges models to localize relevant implementation paths, infer intended behavior, and modify existing systems while preserving surrounding functionality. Across eight evaluated models, we find that failures concentrate in state integration, conditional logic, and incomplete behavioral branches; trajectory analysis further shows that additional exploration or testing is beneficial only when accurately targeted. Evaluated against hidden, deterministic, human-reviewed scoreable variants, GameCode-Bench provides a rigorous testbed for measuring and diagnosing coding-agent capabilities in complex game development.

Primary area:
datasets and benchmarks
Keywords:
benchmark, game development, coding agents

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iTVfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces GameCode-Bench for evaluating coding agents on repository-scale game engineering on 86 tasks from eight open-source browser game repositories. Tasks are reconstructed from historical development artifacts, with solutions assessed through hidden behavioral tests. Eight models are evaluated using mini-swe-agent, with Claude Opus 5 achieving the highest resolution rate. The authors find that failures concentrate in state integration, conditional logic, and incomplete behavioral branches.

Critical strengths
  • It is well-motivated to evluate coding agents on maintenance tasks in established game repositories, which requires understanding gameplay rules distributed across different parts of the codebase.
  • The two construction pipelines provide practical ways to recover tasks from incomplete development records. T-Q reconstructs requests from developer-authored tests. Q-T generates behavioral tests for historical issues with known fixes.
  • The experiments cover multiple frontier models using mini-swe-agent and show that modifying existing gameplay rules remains challenging. Additional trajectory analysis identifies common failures in state integration and handling of rule exceptions.
Critical weaknesses
  • The reported setup is close to conventional software repair on game repositories. It does not clearly include debugging through the game interface, and success is determined entirely by predefined behavioral tests. The authors should consider giving agents access to game visuals and controls, or adding human or MLLM evaluation of the updated games to assess the effects of the changes on the player experience.
  • The paper does not sufficiently establish what distinguishes the challenges in GameCode-Bench from conventional software maintenance. Its evaluation protocol closely follows SWE-bench, and the major failure modes are also common in general software engineering. The authors should further investigate the complex interactions inherent in game development, potentially through experiments in dedicated game engines such as Unity or Unreal Engine.
  • Q-T contributes only nine tasks, limiting the reliability of conclusions about community maintenance scenarios. The paper also provides no separate results for Q-T and T-Q. It is unclear whether the main findings hold for both workflows.
  • The difficulty analysis risks circular reasoning. Categories with lower success rates are described as involving more complex gameplay, but this complexity is not independently measured. The authors should define task complexity independently and check whether more complex tasks actually have lower success rates.
Questions for authors
  • Were agents explicitly given access to game screenshots or a live game interface for debugging?
  • How were the eight repositories selected? How many candidate tasks were considered and discarded during curation?
  • Did human reviewers run the modified games themselves, or were their judgments based on the generated code?
Reviewer_7xSfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a coding benchmark for game development, including complicated open-source game development repositories and comprehensive tasks.

Critical strengths
  • The paper covers the real complexity of game development, augmenting the current benchmark that is mostly considered as creating from scratch.
  • The tasks are comprehensive, including the verifications, reviews and with a focus on aligning the behavior with demands.
Critical weaknesses
  1. The methodological contribution is unclear. The core task formulation, measurement approach, and scoring protocol do not appear substantially different from those of existing general SWE benchmarks. The T-Q/Q-T pipelines primarily recover missing requirements or tests, adapting game-development tasks to an established evaluation paradigm. The paper does not clearly demonstrate what substantive methodological changes the game dev setting requires. Consequently, the work appears to instantiate an existing evaluation paradigm in a new domain, without clearly demonstrating the substantive intellectual contribution required for that instantiation.

  2. The insights enabled by the new data are unclear. If the primary contribution is a game-domain dataset, the authors should explain how it advances our understanding of agent capabilities. For example, by establishing that mechanisms observed in general SWE tasks also hold in game development, or by revealing challenges that existing benchmarks inadequately expose. However, the paper lacks a systematic comparison with empirical observations from general SWE benchmarks. It remains unclear whether state-integration errors, incorrect logic, and missing branches reproduce established findings or reflect problems with distinct underlying mechanisms. Readers therefore cannot readily identify what insights the new data provide beyond another set of model scores and failure statistics.

Questions for authors
  • Did you measure any agent capabilities that are not adequately covered in the general coding benchmark in prior work?
  • Beyond adapting task construction to incomplete repository histories, does game development require substantive changes to the evaluation methodology of general SWE benchmarks? If so, what are these changes, and how does GameCode-Bench implement them?
  • Beyond game-specific examples, do your experiments identify any failure mechanisms that are distinctive to game development? The reported categories, such as incorrect logic, missing branches, and state-integration errors, appear applicable to general SWE tasks. What, if anything, distinguishes the underlying failure mechanisms in your setting?
Reviewer_PEjBOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GameCode-Bench, a benchmark of 86 tasks drawn from 8 mature open-source browser-game repositories. The median repository has about 162k non-test source lines, and the largest exceeds 1M. The agent sees only the repository at the base commit and a natural-language request. It is scored by hidden, deterministic behavioral tests: fail-to-pass (F2P) tests for the new behavior and pass-to-pass (P2P) tests for existing behavior.

Tasks come from three routes:

  • Test-to-Query (T-Q): 74 tasks with historical tests and an LLM-synthesized request.
  • Query-to-Test (Q-T): 9 tasks with a historical issue and synthesized tests.
  • Complete chain: 3 tasks where issue, code change and tests are all historical.

All tasks then go through standardized refinement and manual audit. About 69% of tasks target game logic, split into Action Eligibility, Targets, Effects, Timing and Decisions.

Eight frontier models are evaluated with mini-swe-agent (300 steps, low and max reasoning effort):

  • The best model, Claude Opus 5, reaches 62.8% PASS@1. The others fall between 37% and 47%.
  • The union of all models solves 77.9% of tasks; 19 tasks remain unsolved.
  • Trajectory and failure analyses find that most failures involve state/ordering integration, wrong conditions or missing branches. Testing helps only when the tests are well targeted.
Critical strengths
  1. A realistic and missing evaluation setting. Existing game benchmarks are mostly greenfield or tutorial-scale. This one evaluates edits inside large, historically evolved game codebases, with no list of pre-scoped files to edit, deterministic behavioral tests and P2P regression checks. That fills a real gap between game-generation benchmarks and SWE-bench-style evaluation.
  2. Unsaturated and diagnostic. The top score is 62.8%, 19 tasks are unsolved, and the per-category gaps are large (for example, Action Eligibility vs. Targets). The trajectory analysis yields concrete findings: the compact repair style of GPT-5.6 Sol vs. heavy exploration by others, a localization bottleneck for GLM-5.2, and test quality mattering more than test quantity. These make the benchmark useful beyond a single leaderboard number.
Critical weaknesses
  1. Task validity and specification leakage are not established. 86% of tasks (T-Q) use requests synthesized by an LLM from the hidden tests themselves. That LLM, GPT-5.6 Sol, is also one of the evaluated models. The requests also spell out behaviors, edge cases and invariants in detail. This raises three issues:

    • The "realistic developer request" may partly encode the hidden tests.
    • The task may reduce to implementing a spec that mirrors the tests.
    • Generator–model bias is possible.

    Other gaps compound this:

    • The repositories are popular public projects, so pre-training contamination of the gold patches is plausible, and no analysis by task date vs. model cutoff is given.
    • There is no quantitative evidence that the tests do not reject valid alternative implementations (false-negative rate).
    • The "102 hidden evaluator variants" are never defined.
  2. Statistical and empirical support is thin for a benchmark paper.

    • Single runs on a small benchmark. Each model gets one run on 86 tasks, so the 95% interval is roughly ±10 points. Yet the paper ranks models separated by 1–5 tasks (32–37 resolved; Opus 33 vs. DS-Pro 32 at low effort).
    • Per-category claims rest on very few tasks. Each category has 10–13 tasks, so one task moves a rate by 8–10 points.
    • Annotation reliability is unreported. Failure-mode and validation-quality labels are pre-labeled by GPT-5.6 Sol, and no inter-annotator agreement is given. The protocol also says "four annotators" in one place and "the two annotators" in another.
    • No artifacts for review. The benchmark is promised only "once accepted", so its quality cannot be checked.
Questions for authors
  1. Which request variant is used at evaluation time for T-Q tasks: the detailed test-focused one or the general feature-intent one? How do scores change between the two? What exactly are the 102 evaluator variants?
  2. Can you report results split by task date relative to each model's knowledge cutoff, or another contamination check?
  3. Please report repeated runs or confidence intervals. Are the rankings among the models resolving 32–37 tasks stable?
  4. How many failed patches were manually checked for being correct but rejected by the tests? What is the estimated false-negative rate?
  5. How many annotators were there, what was their inter-annotator agreement, and how much did they revise GPT-5.6 Sol's preliminary labels?
  6. Do the reported output tokens include reasoning tokens? Opus at max effort uses 0.029M output over about 88 steps, which seems low. If reasoning tokens are excluded, the efficiency comparisons may be misleading.
  7. For DeepSeek v4 Flash, the text says max-effort input tokens fall to 6.88M, but Table 3 reports 8.68M, which is an increase. Which is correct?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.