- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces GameCode-Bench for evaluating coding agents on repository-scale game engineering on 86 tasks from eight open-source browser game repositories. Tasks are reconstructed from historical development artifacts, with solutions assessed through hidden behavioral tests. Eight models are evaluated using mini-swe-agent, with Claude Opus 5 achieving the highest resolution rate. The authors find that failures concentrate in state integration, conditional logic, and incomplete behavioral branches.
Critical strengths
- It is well-motivated to evluate coding agents on maintenance tasks in established game repositories, which requires understanding gameplay rules distributed across different parts of the codebase.
- The two construction pipelines provide practical ways to recover tasks from incomplete development records. T-Q reconstructs requests from developer-authored tests. Q-T generates behavioral tests for historical issues with known fixes.
- The experiments cover multiple frontier models using mini-swe-agent and show that modifying existing gameplay rules remains challenging. Additional trajectory analysis identifies common failures in state integration and handling of rule exceptions.
Critical weaknesses
- The reported setup is close to conventional software repair on game repositories. It does not clearly include debugging through the game interface, and success is determined entirely by predefined behavioral tests. The authors should consider giving agents access to game visuals and controls, or adding human or MLLM evaluation of the updated games to assess the effects of the changes on the player experience.
- The paper does not sufficiently establish what distinguishes the challenges in GameCode-Bench from conventional software maintenance. Its evaluation protocol closely follows SWE-bench, and the major failure modes are also common in general software engineering. The authors should further investigate the complex interactions inherent in game development, potentially through experiments in dedicated game engines such as Unity or Unreal Engine.
- Q-T contributes only nine tasks, limiting the reliability of conclusions about community maintenance scenarios. The paper also provides no separate results for Q-T and T-Q. It is unclear whether the main findings hold for both workflows.
- The difficulty analysis risks circular reasoning. Categories with lower success rates are described as involving more complex gameplay, but this complexity is not independently measured. The authors should define task complexity independently and check whether more complex tasks actually have lower success rates.
Questions for authors
- Were agents explicitly given access to game screenshots or a live game interface for debugging?
- How were the eight repositories selected? How many candidate tasks were considered and discarded during curation?
- Did human reviewers run the modified games themselves, or were their judgments based on the generated code?