- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper starts by mining traces from agents tasked with creating games from prompts to produce 4 skills for game authoring. These are incorporated into a harness for game creation that uses playtesting of the games for feedback to improve the games.
Validation compares the harness to calling models via API and using an existing harness (Cursor). On the custom set of tasks the proposed harness improves scores according to tests on types of interactions, playtest with automated controller, and visual assessments. A small user study indicates preference for games from the harness compared to Cursor.
Critical strengths
None.
Critical weaknesses
The paper does not compare to other models using their native harnesses. New models and harnesses - GPT 6.0 astra, GPT 6.1 sol, Claude Fable 5.1, Claude Opus 5.5 - have improved substantially on game generation. It is not clear whether the proposed method still improves over them. This matters as many previously helpful workflows have shown diminishing returns or to be outright harmful to the stronger models, so it is no longer obvious this additional workflow structure would help.
Evaluation does not assess if users consider the games to be complete / "done". Even if they prefer a result it does not mean it is considered a complete game. Ideally the model would have been evaluated on a game generation benchmark like GameCraft-Bench or GameDevBench.
Questions for authors
What effort level was used for models? This makes a substantial difference to performance.
What variability was there over the repeated runs? Results only report means.