- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a structured workflow for agents implementing games from provided text prompts: first identify and diagnose problems, then propose and verify fixes. The workflow couples this with a persistent memory - a checklist - and uses a backup of best performing games when iterating. Validation uses GameCraft-Bench and compares to baseline models with a fixed harness (Play2Code) or no harness. This shows the proposed harness impproves in terms of the benchmark metrics, while reducing cost compared to the fixed harness. Finetuning on traces from the iteration process improves over tuning on good initialization traces alone.
Critical strengths
Shows this harness can improve over baseline harnesses for generating games. Comparisons used existing models in a relatively fair benchmark, offering hope these results hold.
Critical weaknesses
- New models and harnesses - GPT 6.0 astra, GPT 6.1 sol, Claude Fable 5.1, Claude Opus 5.5 - have improved substantially on this task. It is not clear whether the proposed method still improves over them. This matters as many previously helpful workflows have shown diminishing returns or to be outright harmful to the stronger models, so it is no longer obvious this additional workflow structure would help.
- The GameCraft-Bench website reports performance from several models (not as new as above) that are competitive with or surpass RSIGame: https://tongxuluo.github.io/gamecraft-bench-website/
- Opus 5 xhigh (68.44), Fable 5 high (65.72)
- Evaluations only use a single run for generation. We do not know how much variability there is in outputs across generation runs, only evaluations of the same artifacts. This makes it hard to have high certainty in the results.
Questions for authors
The GameCraft-Bench public leaderboard results conflict those in the paper for GPT 5.5 on high using codex. Why is there this discrepancy?
Does this mean we should expect variance in performance across runs of similar magnitude? The paper reports only a single run of each method (though variability in evaluations is measured), hence my inquiry to understand how much variation in generation to expect and whether differences observed are a strong signal of underlying performance differences.
5 comments
Response to Reviewer NFnX (2/3)
Weakness3 & Question2: Variability Across Independent Generation Runs
Evaluations only use a single run for generation. We do not know how much variability there is in outputs across generation runs, only evaluations of the same artifacts. This makes it hard to have high certainty in the results. Does this mean we should expect variance in performance across runs of similar magnitude? The paper reports only a single run of each method (though variability in evaluations is measured), hence my inquiry to understand how much variation in generation to expect and whether differences observed are a strong signal of underlying performance differences.
Following the reviewer's suggestion, we conduct additional experiments to evaluate variability across game generation runs. Table-R2 reports the performance across repeated runs. We find that game generation shows small variability across independent runs: across six generators with three repeated-generation runs, the standard deviations of Overall scores range from 0.44 to 3.37 points. In particular, RSIGame's improvement on GPT-5.5 achieves 14.50, the improvement on GPT-6.1 Sol achieves 15.40, and the improvement on Claude Opus 5.5 achieves 6.85.
Conclusion. Generation quality across repeated runs are stable. We have added results and analysis of generation variability in paper's Appendix D5 Line 4607 Page 86 and Table15 in Page 87. We open-source all game generation and scores results across different models over 3 runs in this anonymous website for a more detailed understanding.
Table-R2. RSIGame Stability across different models over 3 runs.
| Method | Mechanics | Depth | Visuals | Art | Overall ↑ |
|---|---|---|---|---|---|
| Generator: Codex+Qwen3.8-27B | |||||
| Base | 46.0 ± 2.6 | 37.5 ± 2.9 | 39.1 ± 2.9 | 38.1 ± 2.6 | 39.26 ± 1.75 |
| + Play2Code | 53.8 ± 2.3 | 39.6 ± 2.2 | 44.4 ± 1.8 | 44.5 ± 1.4 | 44.23 ± 1.78 |
| + RSIGame | 60.9 ± 3.5 | 44.1 ± 1.3 | 52.0 ± 1.6 | 58.3 ± 5.2 | 52.82 ± 2.87 |
| Δ | +14.9 | +6.6 | +12.9 | +20.2 | +13.56 |
| Generator: Codex+GLM-5.3-Flash | |||||
| Base | 38.2 ± 1.5 | 32.5 ± 1.8 | 33.1 ± 2.8 | 32.4 ± 4.4 | 33.44 ± 2.54 |
| + Play2Code | 49.4 ± 1.4 | 37.7 ± 4.7 | 39.1 ± 1.3 | 35.6 ± 0.9 | 38.94 ± 1.68 |
| + RSIGame | 56.9 ± 2.6 | 42.6 ± 3.5 | 49.8 ± 0.8 | 55.5 ± 2.3 | 50.38 ± 1.83 |
| Δ | +18.7 | +10.1 | +16.7 | +23.1 | +16.94 |
| Generator: Codex+Kimi-K2.6 | |||||
| Base | 39.5 ± 3.9 | 30.4 ± 3.1 | 32.2 ± 1.9 | 22.9 ± 0.5 | 29.43 ± 2.04 |
| + Play2Code | 50.3 ± 1.2 | 38.3 ± 1.2 | 40.7 ± 0.8 | 30.1 ± 0.2 | 37.61 ± 0.44 |
| + RSIGame | 56.4 ± 4.3 | 41.7 ± 1.3 | 49.4 ± 3.1 | 49.5 ± 5.2 | 47.79 ± 3.37 |
| Δ | +16.9 | +11.3 | +17.2 | +26.6 | +18.36 |
| Generator: Codex+GPT-5.5 (high) | |||||
| Base | 58.9 ± 1.5 | 52.2 ± 2.3 | 52.3 ± 0.4 | 49.2 ± 1.5 | 52.25 ± 0.77 |
| + Play2Code | 56.8 ± 4.2 | 48.0 ± 2.8 | 50.2 ± 2.6 | 45.3 ± 2.5 | 48.73 ± 2.87 |
| + RSIGame | 76.6 ± 2.7 | 61.2 ± 3.3 | 67.5 ± 0.4 | 67.7 ± 2.4 | 66.75 ± 2.02 |
| Δ | +17.7 | +9.0 | +15.2 | +18.5 | +14.50 |
| Generator: Codex+GPT-6.1 sol | |||||
| Base | 67.7 ± 2.1 | 61.0 ± 2.7 | 62.3 ± 1.0 | 60.6 ± 0.9 | 62.10 ± 1.47 |
| + Play2Code | 68.7 ± 3.1 | 60.6 ± 3.1 | 61.7 ± 2.8 | 59.5 ± 3.4 | 61.62 ± 3.02 |
| + RSIGame | 87.1 ± 2.0 | 73.6 ± 1.3 | 77.5 ± 3.1 | 77.2 ± 2.2 | 77.50 ± 1.90 |
| Δ | +19.4 | +12.6 | +15.2 | +16.6 | +15.40 |
| Generator: Claude Code+Claude Opus 5.5 (high) | |||||
| Base | 80.8 ± 1.8 | 75.4 ± 1.0 | 76.2 ± 0.8 | 78.1 ± 0.9 | 77.28 ± 0.49 |
| + Play2Code | 82.3 ± 2.0 | 73.5 ± 2.7 | 73.3 ± 2.2 | 75.5 ± 1.8 | 75.58 ± 2.19 |
| + RSIGame | 91.9 ± 1.5 | 83.0 ± 0.4 | 81.5 ± 1.5 | 82.9 ± 3.1 | 84.13 ± 1.54 |
| Δ | +11.1 | +7.6 | +5.3 | +4.8 | +6.85 |
Response to Reviewer NFnX (3/3)
Question1: The discrepancy of GPT5.5 with GameCraft-Bench public leaderboard
The GameCraft-Bench public leaderboard results conflict those in the paper for GPT 5.5 on high using codex. Why is there this discrepancy?
Thanks for the reviewer's careful comparison! We respectfully specify that our evaluation builds upon the official GameCraft-Bench code and method described in paper Section3.2, and further improve gameplay replay reliability and address issues we identified during evaluation. Our paper reports 50.26 for Codex + GPT-5.5 (high), compared with 39.49 on the official GameCraft-Bench leaderboard. However, we also have concerns in the GPT-5.5 performance provided by the public leaderboard, because Codex GPT-5.5 high (39.49) is lower than Deepseek-V4-Flash (40.61) and Kimi-K3 (56.96). We carefully analyze possible factors in the following:
1. Differences in harness: Our experiments use Codex CLI 0.150.1, whereas the official implementation does not specify its Codex version. Its Codex + GPT-5.5 result is dated 2026-04-23, and the newest Codex release on that date was 0.124.0.
2. Effective development budgets: GameCraft-Bench imposes a two-hour wall-clock limit without a fixed tool-call limit. Under this constraint, execution speed, model latency, and system load can affect how much development an agent completes. As shown in Table-R3, our runs consume 3.83M total tokens per task, approximately 3.5× the official 1.08M. The token usage difference indicates substantially different execution characteristics.
Table-R3. Codex + GPT-5.5 (high) on the GameCraft-Bench: public leaderboard vs. our paper.
| Source | Tasks | Overall Score | Tokens / task |
|---|---|---|---|
| Public leaderboard | 140 | 39.49 [1] | 1.08M [2] |
| Ours (paper) | 140 | 50.26 (Table 1) | 3.83M (Table 9) |
| Ratio (ours / leaderboard) | 3.5× |
3. System Load and Gameplay Replay Reliability: We observed that high CPU utilization can slow down game execution and cause stalls during gameplay replay, potentially affecting evaluation scores. To ensure stable gameplay recordings, we carefully controlled execution concurrency and machine load. We also made several adjustments to improve replay reliability, including waiting for the first frame to render before starting replay, extending short demonstrations, and re-recording failed demos caused by environment-related issues. We have explicitly document these modifications in the revised manuscript at Line 4454 Page83.
In conclusion, the discrepancy between the two GPT-5.5 scores may stem from differences in harness versions and execution environments, as discussed above. Importantly, we maintained consistent harness configurations and carefully controlled execution conditions across our experiments, supporting the reliability of our experimental comparisons and conclusions.
[1] GameCraft-Bench Leaderboard, Codex + GPT-5.5 high. https://tongxuluo.github.io/gamecraft-bench-website/
[2] GameCraft-Bench official repository, README, "Token usage" table, GPT-5.5 (Codex): 148.5M input and 2.8M output tokens over 140 tasks. Commit a433475, 2026-08-13. https://github.com/FreedomIntelligence/gamecraft-bench/blob/a43347534374df9a0c1a6c001aa9380862783f6d/README.md#L68-L77
Response to Reviewer NFnX (1/3)
We sincerely thank the reviewer for the constructive feedback! We address all concerns raised in the weaknesses and questions below.
Weakness1&2 RSIGame Improvement over New Models
Following the reviewer's suggestion, we conduct additional experiments to evaluate whether RSIGame remains effective on games generated by stronger new models. Table R1 shows that the improvement of RSIGame consistently persists. On GPT-6.1 Sol, our method RSIGame improves game quality by +15.40 points over the initial generation. Notably, on the stronger Claude Opus 5.5 initialization, the baseline Play2Code reduces game quality by 1.70 points, while our method RSIGame still achieves a +6.85 points improvement. These results demonstrate that RSIGame still remains effective on games generated by stronger new models.
Table-R1. RSIGame Improvement over new models over 3 runs.