- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The manuscript introduces GameASG-Bench, a benchmark that makes behavioral testability part of the generation task for game development.
Critical strengths
The paper proposes a practical testing framework that is (1) straightforward to implement and (2) supports behavioral evaluation of generated games more efficiently.
Critical weaknesses
-
The paper provides limited evidence that passing its checks reflects actual requirement compliance. An independent human audit of the generated games is missing.
-
The paper does not assess how testability requirements affect game development. A comparison with and without these requirements is needed to evaluate their impact on game quality and development cost.
-
No targeted adversarial or negative-control tests are reported. It remains unclear whether the checks can detect test-specific logic disconnected from actual gameplay.
Questions for authors
-
Would you provide a comparison with human and your proposed checking?
-
Would you provide a test on whether your introduced testability development affect game major development? Are there any shortcuts, bugs or safety issues introduced with the testability development?
-
Is it possible that agent develop a submodule that could only be triggered after the testing logic is on?
-
Would you provide more visual demonstration of your concepts to make the draft more friendly to readers?
5 comments
Weakness 3: “No targeted adversarial or negative-control tests are reported. It remains unclear whether the checks can detect test-specific logic disconnected from actual gameplay.”
Answer 3: We inspected all game implementations and found no independent gameplay logic that was separate from actual gameplay and ran only during testing. Our requirement is to provide a unified access interface to the game’s own functionality. Games already need to support initialization, process player actions, and maintain and read game state. We standardize how these functions are called so that the evaluator can actively prepare game states, execute actions, and read results. The interface specification requires reusing the state and logic used in normal gameplay, rather than asking the agent to implement a separate set of gameplay logic solely for testing.
Question 1: “Would you provide a comparison with human and your proposed checking?”
Answer 1: We added an independent human evaluation to directly compare automated check results with human judgments under normal player interactions. Two evaluators independently assessed the same set of 46 behavioral instances spanning eight games through keyboard and mouse interactions and visible feedback. They did not inspect source code or invoke test interfaces and were blinded to model identities and automated check results.
Automated checks agreed with human judgments after review on 23/24 instances (95.8%) in the RQ1 sample stratum and all 22 instances (100.0%) in the historical failure supplement. The sole disagreement was an automated PASS and a human FAIL. The evaluation setup, results by stratum, and disagreement counts are provided in our response to Weakness 1 above.
Question 2: “Would you provide a test on whether your introduced testability development affect game major development? Are there any shortcuts, bugs or safety issues introduced with the testability development?”
Answer 2: We compared generation with and without test interface requirements across eight games, generating each game three times under each condition. Human experts assessed the resulting artifacts against the same player-observable gameplay requirements. Both groups completed 24/24 deliveries. Requirement satisfaction rates were 99.6% with interface requirements and 95.3% without them, while mean generation time and output tokens increased by approximately 2.9% and 3.8%, respectively, in the group with interface requirements. The complete experimental setup and results are provided in our response to Weakness 2 above.
Regarding test-specific logic, we inspected all game implementations and found no independent gameplay logic that was separate from normal gameplay and ran only during testing. The interface specification standardizes how the game’s own initialization, input handling, and state access functions are called, and requires reusing the state and logic used in normal gameplay. Further details are provided in our response to Weakness 3 above.
Question 3: “Is it possible that agent develop a submodule that could only be triggered after the testing logic is on?”
Answer 3: We inspected all game implementations and found no independent gameplay submodule that was separate from normal gameplay and triggered only during testing.
The test interface standardizes how the game’s own functions are called, including initialization, processing player actions, and reading game state. The interface specification requires these calls to reuse the state and logic of normal gameplay, enabling the evaluator to actively prepare states, execute actions, and observe outcomes. Further details are provided in our response to Weakness 3 above.
Question 4: “Would you provide more visual demonstration of your concepts to make the draft more friendly to readers?”
Answer 4: Thank you for your suggestion. We have added the following figures to the revised manuscript PDF:
- Figure 1 (framework overview): Shows how task documents guide game generation and how L1 source checks and L2 behavioral checks evaluate generated artifacts, alongside images of benchmark games.
- Figure 2 (concrete evaluation example): Uses the food-delivery action in Diner Dasher to illustrate the “prepare–act–observe” process: first prepare a customer order and a burger on the tray, then deliver the food through real mouse drag-and-drop interaction, and finally observe the delivery count increase from 0 to 1 and the number of items on the tray decrease from 1 to 0.
- Appendix G (Figures 4–11): Provides a screenshot of a running generated implementation for each of the 47 games, accompanied by a brief gameplay description showing its main controls or objectives, helping readers understand the specific tasks included in the benchmark.
Thank you again for your feedback on the value and presentation of our work. We would be happy to discuss this further with you.
Thanks for the rebuttal.
Here are several points that I think might increase the quality of the work: 1) increase the number of testcase: current dataset only has 46 cases, 2) increase the complexity of the testcase: current dataset only covers trival games (I do not recognize it in my initial review for lack of illustration in the manuscript).
I hereby maintain the original score.
PS. Please mark your changes with another color next time for revision.
Response to Reviewer 3psW: Additional Experiments and Clarifications
Thank you for your careful review and for recognizing our framework’s practicality and efficiency in evaluating the behavior of generated games. Our response was delayed because the additional experiments required evaluation by human experts. Thank you for your patience.
Weakness 1: “The paper provides limited evidence that passing its checks reflects actual requirement compliance. An independent human audit of the generated games is missing.”
Answer 1: Thank you for your suggestion. We conducted an independent human validation experiment, sampling 24 instances from the pool of game artifacts that passed evaluation and 22 instances from the pool that failed evaluation, yielding 46 artifact–behavior instances spanning eight games. Two evaluators independently assessed the same set of instances using normal keyboard and mouse interactions and visible game feedback. They did not inspect source code or invoke test interfaces and were blinded to model identities and automated check results.
For comparison, we aligned the required initial conditions, player actions, and observable outcomes. All 46 instances were comparable and assessable. The comparison between automated check results and human judgments after review is shown below:
The results show a high level of agreement between automated checks and human judgments. The sole disagreement occurred in an instance that passed automated evaluation but was judged to fail by the human evaluators. These findings provide independent evidence of alignment between automated checks and player-observable requirements. We will include the human evaluation setup, results, and disagreement case in the final paper.
Weakness 2: “The paper does not assess how testability requirements affect game development. A comparison with and without these requirements is needed to evaluate their impact on game quality and development cost.”
Answer 2: Thank you for your suggestion. We added an experiment examining the impact of testability requirements on game development. We compared two generation conditions across eight games: with and without test interface requirements. Each game was independently generated three times under each condition, yielding 48 generation runs. Because DeepSeek recently retired DeepSeek-V4-Flash, both groups used DeepSeek-V4.1-Flash / Claude Code, with identical gameplay requirements, tools, budgets, and delivery policies. The only difference was the interface requirements. To ensure a common evaluation standard, human experts assessed both groups against the same player-observable gameplay requirements from the benchmark.
The human evaluation results and generation costs are shown below:
Both groups completed all 24 deliveries. The group with interface requirements achieved a player-observable requirement satisfaction rate of 99.6%, exceeding the group without them by 4.3 percentage points, while mean generation time and output tokens increased by approximately 2.9% and 3.8%, respectively. In this experiment, the group with interface requirements achieved higher requirement satisfaction with modest additional generation overhead, supporting the inclusion of testability requirements in game development specifications. Together with the online evaluation results in RQ1, this comparison further illustrates the impact of predeclared interfaces on generation costs and evaluation efficiency.
A test interface can be understood as a structured development contract. It specifies initialization, legal state transitions, action semantics, and observable state before implementation. These responsibilities already exist in games that need to consistently support startup, progression, termination, and restart. Defining a unified external interface for these responsibilities provides a reusable way to execute game behaviors across implementations. The ablation results support using this contract in practice: within the scope of this evaluation, it preserved player-perceived gameplay quality.