- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a harness that iteratively refines game (more genreally, software) designs, using an inner loop of improving the game and an outer loop that changes the harness based on progress to further improve the game. The inner loop tests changes to the harness while the outer loop maintains reusable proposals for improvement. The harness evaluates games with a combination of fixed evaluation criteria and variable criteria that are proposed and revised by the outer loop. Evaluation compares to a baseline harness and alternative with a fixed game-specific harness on matched benchmarks. The comparisons show improvements using GameEvolve on four game benchmarks and two software benchmarks.
Critical strengths
None
Critical weaknesses
- The paper does not compare to other models using their native harnesses. New models and harnesses - GPT 6.0 astra, GPT 6.1 sol, Claude Fable 5.1, Claude Opus 5.5 - have improved substantially on game generation. It is not clear whether the proposed method still improves over them. This matters as many previously helpful workflows have shown diminishing returns or to be outright harmful to the stronger models, so it is no longer obvious this additional workflow structure would help.
- The GameCraft-Bench website reports performance from several models (not as new as above) that are competitive with or surpass Game-Evolver: https://tongxuluo.github.io/gamecraft-bench-website/
- Opus 5 xhigh (68.44), Fable 5 high (65.72), Codex GPT 5.6 Sol high (60.50)
- V-GameGym shows GPT 5 (83.25), GPT 5 mini (80.78), and Claude Sonnet 4 thinking (79.86) exceed the best Game-Evolver (48.45).
- Related: the paper does not compare to other harness improvement methods as an alternative baseline. As that is the methodological innovation it is the comparison I would expect.
Questions for authors
What model effort was used for the results reported?
What statistical variability was there among runs on the same tasks? How many repeats were run for a given configuration and how was variability measured? This is importance to assess magnitude of differences.
GameDevBench reports 333 tasks, while the paper reports 132 tasks. What accounts for the difference?
2 comments
Q4. Positioning against harness-improvement work
We agree the draft does not position against this line of work and we will fix it. One distinction concerns which object each method improves. RSIGame's local and global loops improve the game artifact across rounds, and it additionally internalizes development experience into the generator through supervised fine-tuning, reported as Qwen3.8-27B reaching 61.38 on Godot after that training and further multi-round development. Both objects are per-run or per-model: the artifact belongs to one task, and the weights belong to one backbone and must be retrained for another. Its controller, verification and repair modules are shared across tasks.
Our outer loop changes the harness H = <M, U, π_ctx, π_wf, π_val, Z>, with the backbone frozen and no weights touched. A harness edit is applied by configuration and transfers unchanged, which is what Table 4 and Fig. 4 exercise when the same harness moves to Terminal-Bench and τ²-Bench. The difference is in what the self-improvement applies to, and the two are composable: its artifact-level loop and weight-level training could sit inside our inner loop, with our outer loop continuing to edit the harness around them. We will say this in the related-work section and will also position against harness-optimization work in our own family (Meta-harness, observability-driven harness evolution, AgentX), running a controlled comparison at matched backbone and budget within the revision window.
Q5. Reproducibility and standard of evidence
For the rubric dimensions on GameCraft-Bench and GameDev-Bench, scoring comes from the benchmark's own verifier and rubric categories; where our own component enters the scoring path we disclose it as in Section 3. Our unmodified GPT-5.5 baseline reproduces the published row to two decimals on all four categories and within 0.02 Overall.
One correction. The abstract and the contribution list state that Game-Evolver delivers "several-fold gains on weaker models and up to 30% on the strongest". The first half holds: DeepSeek-V4-Flash improves 4.00× on GameDev-Bench (12.31 to 49.25), Qwen3.6-27B 2.83× on GameCraft-Bench (4.23 to 11.96), and Kimi-K2.7-Code 2.30× on GameDev-Bench (3.00 to 6.90). The second half is not a statement about the strongest model across benchmarks: on GPT-5.5 the relative gain is +6.8% on GameCraft-Bench (39.47 to 42.17), +4.3% on GameDev-Bench (55.56 to 57.96) and +4.1% on V-GameGym (43.87 to 45.67), and the +29.6% figure comes from VeriGame alone (50.50 to 65.45). We will rewrite both sentences to report the per-benchmark figures instead of a single peak. This changes no table value, and the summary as written is more favourable than the tables support.
Q6. Effort settings
Every backbone was run at the lowest reasoning tier its provider exposes, applied uniformly rather than tuned per model: GPT-5.5 with reasoning explicitly disabled, Claude-Sonnet-4.6, GLM5.2, Qwen3.6-27B and DeepSeek-V4-Flash at their non-reasoning default, and Kimi-K2.7-Code at low thinking. No number in Tables 1-3 was produced with extended reasoning enabled.
The setting is shared by both sides of every A/B pair. Our GPT-5.5 baseline at this tier also falls within 0.02 Overall of the published high row, with all four categories matching to two decimals. We have added the per-backbone setting to the Table 1-3 captions and an appendix recording the exact setting used for every reported number, and we will re-run our strongest backbones at the highest tier each provider accepts, stating the actual tier for each line.
Q7. Statistical variability
The tables are single-run per cell and we will label them as such. For two representative cells, GLM5.2 on GameCraft-Bench and GPT-5.5 on V-GameGym, we will run three independent seeds and report mean ± std with per-seed trajectories. We state this as a planned addition. The direction of the effect is consistent across six backbones and four benchmarks, and a variance estimate will quantify the noise.
Q8. GameDevBench 132 versus 333
This is a version difference rather than a subsetting choice. The first release (arXiv:2602.11103v1) contains 132 tasks, 115 base and 17 variants; the current release contains 333 tasks, 154 base and 179 variants. Our experiments use the complete 132-task v1 inventory, which is the release our paper cites and the one behind all our baseline numbers. The v2 additions postdate our runs and were never candidates under our protocol. The skill mix also shifted, with gameplay logic the largest category in v1 at 35.6% and 2D graphics the largest in v2 at 33.3%, so category-level scores are not directly comparable across inventories either. We will state the version in Sec. 5 and in the Table 2 caption, and re-run our strongest backbones on v2 so both sets can be read with the version labelled.
We are grateful for the time you spent on this review.
We thank you for the direct assessment. On one of the five substantive points the comparison as stated compares different objects, on another we were wrong in our reading of your figures and correct the record below, and on a third we have repositioned the claim. We will run the experiments that settle the remaining disagreement.
Q1. Calibration of our GPT-5.5 baseline
Our Table 1 baseline for GPT-5.5 is
39.47 / 54.36 / 38.61 / 41.84 / 32.94(Overall / Mechanics / Depth / Visuals / Art); the GameCraft-Bench leaderboard entry forCodex + GPT-5.5 highis39.49 / 54.36 / 38.61 / 41.84 / 32.94. All four rubric categories agree to two decimals and Overall differs by 0.02, so our unmodified baseline reproduces the official row on every sub-score and to within 0.02 on the aggregate. The evolved configuration scores 42.17 on the same pipeline, +2.70 over our own baseline and +2.68 over the official entry.Two qualifications. The gain is not uniform: Art falls from 32.94 to 23.40 while Overall rises, and we will report that trade-off in the ablation discussion. And the leaderboard row carries
Model Type = Vision; our configuration is likewise vision-enabled in two places, with screenshots of the running game supplied to the judge behind Visuals and Art and rendered frames available to the developing agent during the build. The agent scaffold, judge model and effort level differ; this is calibration of our scoring path, covering GameCraft-Bench only. On V-GameGym the channels come from our own judge, which we disclose in Section 3.Q2. The leaderboard comparison
We verified all three figures on the benchmark's website and take them at face value. The rows differ in what they measure. Every row is a model-harness pair, and two further columns are not held constant: the Agent (
Claude Codefor Opus 5 xhigh 68.44 and Fable 5 high 65.72,Codexfor GPT-5.6-sol high 60.50) andModel Type. All five rows above 50 are Vision rows; the six Text rows span 2.15 to 40.61 with a mean of 22.54 against 51.74 for the eight Vision rows. Our Table 1 holds the backbone fixed and every delta is a before/after within one backbone under one harness, so our 42.17 and the 68.44 row differ in scaffold and model.The internal dispersion also bears on how rows should be compared: two Text rows score 40.61 and 2.15, an 18.9-fold spread inside the same modality. We will state in the table caption that leaderboard entries are a different experimental condition. Under a fixed harness, evolution raises every backbone we tested.
Q3. The V-GameGym figures
The cited numbers are accurate. The comparison is not like-for-like because the live leaderboard and the released paper score the visual channels on different scales. Both use the same aggregation rule, the equal-weight mean of Code, Screenshot and Video, which we verified on all 72 leaderboard rows and on the released Table 2, and both report essentially the same Code score for GPT-5 (96.6 in the paper, 96.64 on the leaderboard). The visual channels differ: the released paper reports Total 45.0 with Screenshot 17.6 and Video 20.7, while the live leaderboard reports 77.89 and 75.21 for the same model. Across the fourteen models common to both releases the visual channels grow by a factor of 2.7 to 4.8, median 3.6, so this is a systematic difference between releases rather than something specific to these rows. That is the source of the 45.0-versus-83.25 gap.
Our numbers sit between the two scales. Our best configuration, GLM5.2, scores 48.45 Overall from 90.79 Code, 29.43 Screenshot and 25.11 Video, and across Table 3 the two visual channels span 5.79 to 29.43 against 11.9 to 22.4 for all rows of the released Table 2. Our visual channels come from a different judge, and our top visual score exceeds every row in that release. Our V-GameGym channels use our own text judge rather than the benchmark's official judge pair, which is the most likely reason our visual scores reach 29.43 and 28.43. To settle it we will re-run our two strongest backbones with the official judge and state both the scale and the judge in the Table 3 caption, which the draft currently omits.