- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper builds a benchmark containing 72 gameplay-logic tasks in Godot projects for game development agents. All tasks come with tick-level assertions that can be deterministically verified without needing a judge model. The tasks span isolated mechanics, multi-system interactions, and repository-scale features. To ensure the soundness of the evaluator, careful techniques such as mutants and multiple scenarios are used. Studies were done on trending open-source and frontier models.
Critical strengths
The core evaluation design of using tick-level assertions is quite novel to me, and it aligns closely with runtime game logic than alternatives such as final state checks or VLM judging. This is a design that is highly specific to game dev (because of the usual main loop).
The construction of the benchmark is careful with mutant-based judge validation and the use of multiple scenarios. The use of different task scales is also commendable. The authors are also very clear about leakage if network access is open, and thus decided to run benchmark in sealed environment.
Critical weaknesses
While it is extremely neat to have deterministically verifiable signals in a game dev benchmark like tick-level assertions, the authors did not quantify how universal such signals can be designed. It would be great to have a discussion of the coverage: what kind of game mechanics can and cannot be covered in such tick-level assertions. The 7 capability classes in table 7 all seem quite low-level and correspond to very small, rigid mechanics. As such, it wouldn't serve as a benchmark for building moderate to large-scale mechanics which inevitably would involve evaluation of funness, aesthetics, pacing, or other fuzzy metrics.
Another weakness is that the size of the benchmark is still relatively small at 72 tasks, possibly due to the high authoring cost of involving human in many parts of the pipeline.
Questions for authors
Can authors discuss how to extend the tick-level assertions to bigger scale mechanics beyond examples like state transitions and local/runtime correctness? If not possible, what other ways can we evaluate bigger mechanics?
We thank the reviewer for the positive feedback on our tick-level assertions and mutant-based judge validation.
Coverage (W1). A mechanic is covered when runtime state and event history determine its correct behavior at every tick, which takes three forms:
Mechanics judged by look or feel, such as visual appeal, control feel, and pacing, are screened out (§2.2).
Larger mechanics (W1, C1). Larger subsystems are evaluated by combining tick-level invariants from several capability classes. Each class in Table 7 categorizes what a single assertion checks, while Repo tasks implement complete subsystems of up to 945 lines inside projects of up to 62,895 lines. Our character-controller task, for instance, checks the state transitions, timing windows, and spatial constraints of a movement component shared by the animation, camera, and input systems.
Fun, aesthetics, and pacing (W1, C1). State assertions do not measure fun, aesthetics, or pacing, so evaluating larger mechanics requires pairing deterministic functional checks with experiential raters scoring control responsiveness, pacing, visual readability, and engagement. Among automated raters, scripted screenshot or video capture with a VLM rubric judge is inexpensive, yet misses states outside fixed recordings and differs from an interactive player's perspective. Pure GUI playtesting agents match human play more closely, yet remain costly and unreliable on complex games even with frontier models. Combining CLI control with GUI playtesting in a hybrid agent judge offers a practical path forward that mirrors how human QA testers actually work: using engine-side commands to reach target or edge-case states directly, and then playing through the GUI to assess visual presentation and feel.
We plan to expand the benchmark in two ways: (1) collecting mechanics from more genres and from other engines that support headless fixed-step inspection, and (2) accepting community pull requests on GitHub under the same calibration, agent review, and human review pipeline.