acceptodds
Under review as a conference paper at ICLR 2027

Game-Evolver: Dual-Loop Evolution Elicits Better End-to-End Game Orchestration

Abstract

Recent coding agents have made significant strides in tackling complex tasks across various domains. However, even frontier agents can solve only a minority of game generation tasks, owing to the extreme long-horizon requirements from building games and interacting in it. Moreover, most agent frameworks rarely provide comprehensive, holistic workflows specifically designed for game making tasks, preventing their further applications. These factors jointly pose a unique challenge in the context of agent game development. We address this challenge with Game-Evolver, a dual-loop framework that couples two agents. A game orchestration agent (GOA) builds the game; a harness proposal agent (HPA) proposes bounded changes to the GOA's harness. In the inner loop, the GOA A/B-tests each proposed harness against the best candidate so far. A harness is admitted only when the new harness passes dual-rubric verification, where hard rubrics enforce the game legality and soft rubrics measure its quality. In the outer loop, the HPA distills the GOA's completed runs into recurring failure modes, and uses them to update its own harness library. Game-Evolver thus improves both the game-making and the agent that generates it. Extensive experiments on four game benchmarks and six backbone models show that Game-Evolver improves every baseline, delivering several-fold gains on weaker models and up to 30% on the strongest. Our approach also transfers to general agent tasks, while showing impressive applicability to real game projects.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agent, Game Generation, Vibe-Gaming

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4a54OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a harness that iteratively refines game (more genreally, software) designs, using an inner loop of improving the game and an outer loop that changes the harness based on progress to further improve the game. The inner loop tests changes to the harness while the outer loop maintains reusable proposals for improvement. The harness evaluates games with a combination of fixed evaluation criteria and variable criteria that are proposed and revised by the outer loop. Evaluation compares to a baseline harness and alternative with a fixed game-specific harness on matched benchmarks. The comparisons show improvements using GameEvolve on four game benchmarks and two software benchmarks.

Critical strengths

None

Critical weaknesses
  • The paper does not compare to other models using their native harnesses. New models and harnesses - GPT 6.0 astra, GPT 6.1 sol, Claude Fable 5.1, Claude Opus 5.5 - have improved substantially on game generation. It is not clear whether the proposed method still improves over them. This matters as many previously helpful workflows have shown diminishing returns or to be outright harmful to the stronger models, so it is no longer obvious this additional workflow structure would help.
  • The GameCraft-Bench website reports performance from several models (not as new as above) that are competitive with or surpass Game-Evolver: https://tongxuluo.github.io/gamecraft-bench-website/
    • Opus 5 xhigh (68.44), Fable 5 high (65.72), Codex GPT 5.6 Sol high (60.50)
  • V-GameGym shows GPT 5 (83.25), GPT 5 mini (80.78), and Claude Sonnet 4 thinking (79.86) exceed the best Game-Evolver (48.45).
  • Related: the paper does not compare to other harness improvement methods as an alternative baseline. As that is the methodological innovation it is the comparison I would expect.
Questions for authors

What model effort was used for the results reported?

What statistical variability was there among runs on the same tasks? How many repeats were run for a given configuration and how was variability measured? This is importance to assess magnitude of differences.

GameDevBench reports 333 tasks, while the paper reports 132 tasks. What accounts for the difference?

2 comments

We thank you for the direct assessment. On one of the five substantive points the comparison as stated compares different objects, on another we were wrong in our reading of your figures and correct the record below, and on a third we have repositioned the claim. We will run the experiments that settle the remaining disagreement.

Q1. Calibration of our GPT-5.5 baseline

Our Table 1 baseline for GPT-5.5 is 39.47 / 54.36 / 38.61 / 41.84 / 32.94 (Overall / Mechanics / Depth / Visuals / Art); the GameCraft-Bench leaderboard entry for Codex + GPT-5.5 high is 39.49 / 54.36 / 38.61 / 41.84 / 32.94. All four rubric categories agree to two decimals and Overall differs by 0.02, so our unmodified baseline reproduces the official row on every sub-score and to within 0.02 on the aggregate. The evolved configuration scores 42.17 on the same pipeline, +2.70 over our own baseline and +2.68 over the official entry.

Two qualifications. The gain is not uniform: Art falls from 32.94 to 23.40 while Overall rises, and we will report that trade-off in the ablation discussion. And the leaderboard row carries Model Type = Vision; our configuration is likewise vision-enabled in two places, with screenshots of the running game supplied to the judge behind Visuals and Art and rendered frames available to the developing agent during the build. The agent scaffold, judge model and effort level differ; this is calibration of our scoring path, covering GameCraft-Bench only. On V-GameGym the channels come from our own judge, which we disclose in Section 3.

Q2. The leaderboard comparison

We verified all three figures on the benchmark's website and take them at face value. The rows differ in what they measure. Every row is a model-harness pair, and two further columns are not held constant: the Agent (Claude Code for Opus 5 xhigh 68.44 and Fable 5 high 65.72, Codex for GPT-5.6-sol high 60.50) and Model Type. All five rows above 50 are Vision rows; the six Text rows span 2.15 to 40.61 with a mean of 22.54 against 51.74 for the eight Vision rows. Our Table 1 holds the backbone fixed and every delta is a before/after within one backbone under one harness, so our 42.17 and the 68.44 row differ in scaffold and model.

The internal dispersion also bears on how rows should be compared: two Text rows score 40.61 and 2.15, an 18.9-fold spread inside the same modality. We will state in the table caption that leaderboard entries are a different experimental condition. Under a fixed harness, evolution raises every backbone we tested.

Q3. The V-GameGym figures

The cited numbers are accurate. The comparison is not like-for-like because the live leaderboard and the released paper score the visual channels on different scales. Both use the same aggregation rule, the equal-weight mean of Code, Screenshot and Video, which we verified on all 72 leaderboard rows and on the released Table 2, and both report essentially the same Code score for GPT-5 (96.6 in the paper, 96.64 on the leaderboard). The visual channels differ: the released paper reports Total 45.0 with Screenshot 17.6 and Video 20.7, while the live leaderboard reports 77.89 and 75.21 for the same model. Across the fourteen models common to both releases the visual channels grow by a factor of 2.7 to 4.8, median 3.6, so this is a systematic difference between releases rather than something specific to these rows. That is the source of the 45.0-versus-83.25 gap.

Our numbers sit between the two scales. Our best configuration, GLM5.2, scores 48.45 Overall from 90.79 Code, 29.43 Screenshot and 25.11 Video, and across Table 3 the two visual channels span 5.79 to 29.43 against 11.9 to 22.4 for all rows of the released Table 2. Our visual channels come from a different judge, and our top visual score exceeds every row in that release. Our V-GameGym channels use our own text judge rather than the benchmark's official judge pair, which is the most likely reason our visual scores reach 29.43 and 28.43. To settle it we will re-run our two strongest backbones with the official judge and state both the scale and the judge in the Table 3 caption, which the draft currently omits.

Q4. Positioning against harness-improvement work

We agree the draft does not position against this line of work and we will fix it. One distinction concerns which object each method improves. RSIGame's local and global loops improve the game artifact across rounds, and it additionally internalizes development experience into the generator through supervised fine-tuning, reported as Qwen3.8-27B reaching 61.38 on Godot after that training and further multi-round development. Both objects are per-run or per-model: the artifact belongs to one task, and the weights belong to one backbone and must be retrained for another. Its controller, verification and repair modules are shared across tasks.

Our outer loop changes the harness H = <M, U, π_ctx, π_wf, π_val, Z>, with the backbone frozen and no weights touched. A harness edit is applied by configuration and transfers unchanged, which is what Table 4 and Fig. 4 exercise when the same harness moves to Terminal-Bench and τ²-Bench. The difference is in what the self-improvement applies to, and the two are composable: its artifact-level loop and weight-level training could sit inside our inner loop, with our outer loop continuing to edit the harness around them. We will say this in the related-work section and will also position against harness-optimization work in our own family (Meta-harness, observability-driven harness evolution, AgentX), running a controlled comparison at matched backbone and budget within the revision window.

Q5. Reproducibility and standard of evidence

For the rubric dimensions on GameCraft-Bench and GameDev-Bench, scoring comes from the benchmark's own verifier and rubric categories; where our own component enters the scoring path we disclose it as in Section 3. Our unmodified GPT-5.5 baseline reproduces the published row to two decimals on all four categories and within 0.02 Overall.

One correction. The abstract and the contribution list state that Game-Evolver delivers "several-fold gains on weaker models and up to 30% on the strongest". The first half holds: DeepSeek-V4-Flash improves 4.00× on GameDev-Bench (12.31 to 49.25), Qwen3.6-27B 2.83× on GameCraft-Bench (4.23 to 11.96), and Kimi-K2.7-Code 2.30× on GameDev-Bench (3.00 to 6.90). The second half is not a statement about the strongest model across benchmarks: on GPT-5.5 the relative gain is +6.8% on GameCraft-Bench (39.47 to 42.17), +4.3% on GameDev-Bench (55.56 to 57.96) and +4.1% on V-GameGym (43.87 to 45.67), and the +29.6% figure comes from VeriGame alone (50.50 to 65.45). We will rewrite both sentences to report the per-benchmark figures instead of a single peak. This changes no table value, and the summary as written is more favourable than the tables support.

Q6. Effort settings

Every backbone was run at the lowest reasoning tier its provider exposes, applied uniformly rather than tuned per model: GPT-5.5 with reasoning explicitly disabled, Claude-Sonnet-4.6, GLM5.2, Qwen3.6-27B and DeepSeek-V4-Flash at their non-reasoning default, and Kimi-K2.7-Code at low thinking. No number in Tables 1-3 was produced with extended reasoning enabled.

The setting is shared by both sides of every A/B pair. Our GPT-5.5 baseline at this tier also falls within 0.02 Overall of the published high row, with all four categories matching to two decimals. We have added the per-backbone setting to the Table 1-3 captions and an appendix recording the exact setting used for every reported number, and we will re-run our strongest backbones at the highest tier each provider accepts, stating the actual tier for each line.

Q7. Statistical variability

The tables are single-run per cell and we will label them as such. For two representative cells, GLM5.2 on GameCraft-Bench and GPT-5.5 on V-GameGym, we will run three independent seeds and report mean ± std with per-seed trajectories. We state this as a planned addition. The direction of the effect is consistent across six backbones and four benchmarks, and a variance estimate will quantify the noise.

Q8. GameDevBench 132 versus 333

This is a version difference rather than a subsetting choice. The first release (arXiv:2602.11103v1) contains 132 tasks, 115 base and 17 variants; the current release contains 333 tasks, 154 base and 179 variants. Our experiments use the complete 132-task v1 inventory, which is the release our paper cites and the one behind all our baseline numbers. The v2 additions postdate our runs and were never candidates under our protocol. The skill mix also shifted, with gameplay logic the largest category in v1 at 35.6% and 2D graphics the largest in v2 at 33.3%, so category-level scores are not directly comparable across inventories either. We will state the version in Sec. 5 and in the Table 2 caption, and re-run our strongest backbones on v2 so both sets can be read with the version labelled.

We are grateful for the time you spent on this review.

Reviewer_rgdwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Game-Evolver optimizes the framework used by a game-building agent. A game orchestration agent produces artifacts, while a harness proposal agent suggests bounded changes to instructions, tools, skills, and workflows. The inner loop compares parent and proposed frameworks through matched runs, admitting changes only when hard constraints and weighted soft scores do not regress on valid comparison cases. The outer loop uses accumulated outcomes to update the proposal agent's element library. Experiments report higher overall scores across four game benchmarks and six backbones, with additional general-agent results and case studies. Ablations examine progressively broader framework changes and the added outer loop.

Critical strengths
  1. The paper makes a useful distinction between improving an individual game and improving the development framework itself (Sections 3-5). Bounded modifications, matched parent-child runs, and rubrics frozen within each comparison provide a concrete mechanism for evaluating proposed changes. The combination of per-case hard and soft non-regression checks is more informative than accepting a proposal solely because its average score increased.
  2. The evaluation covers multiple tasks and backbones, with gains in overall scores across all four game benchmarks (Tables 1-3). For example, GLM5.2 increases from 18.00 to 35.33 on GameCraft-Bench and from 53.75 to 73.57 on GameDev-Bench. The dimension-level reporting also reveals tradeoffs rather than universal improvement, and the appendices provide concrete examples of the framework modifications produced by evolution.
Critical weaknesses
  1. An independent final evaluation is not clearly established. The fixed 20% held-out set is repeatedly used for inner-loop scoring and framework selection, and its outcomes inform the outer-loop updates. It therefore functions as an adaptive validation set. The manuscript does not clearly identify a further untouched test partition or specify whether the final reported tables exclude all tasks involved in selection. Using self-constructed rather than public benchmark rubrics does not establish task-level independence. This is a protocol ambiguity, not proof of leakage, but it prevents a confident interpretation of the gains as generalization to unseen tasks.
  2. The outer-loop ablation shows a quality gain but does not establish an advantage at a matched computational budget. The full method takes approximately 1.5-1.7 times as long as L3, which already includes the full inner loop. This supports the value of adding the outer loop at its reported cost, but leaves open whether the same resources spent on more inner-loop proposals or restarts would yield comparable gains. Repeated independent evolution runs and uncertainty estimates are also not reported. These comparisons matter for assessing the specific value and reliability of the proposed dual-loop design.
Questions for authors
  1. Which exact task partitions produced Tables 1-4, and was the final framework frozen before evaluation on tasks never used for proposal generation, admission, or library updates? For the general-agent experiments, was the framework transferred unchanged or re-optimized on each target benchmark?
  2. Does the full method outperform L3 with equal total time or token budgets, including proposal generation and verification?
  3. What minimum valid-case count K, judge configurations, and case-exclusion rules are used in Equation 6? Are build or runtime failures caused by a proposed framework scored as failures, rather than excluded as infrastructure errors?
3 comments

We thank you for the most precise reading of our protocol. Your three weaknesses identify real ambiguities in how we described the evaluation. We address each below.

Q1. Is the 20% held-out set an adaptive validation set?

Your reading is correct that our description is ambiguous.

The 20% held-out set serves exactly one role in the inner loop: scoring the parent-versus-child A/B test that governs admission. It is not used to fit model weights, since the backbone is frozen and API-served, and the numbers in Tables 1-3 are not produced by the self-constructed rubric that drives admission. The paper states that we use our own rubrics for validation rather than the dataset's public evaluation benchmarks, while the reported scores follow each benchmark's published protocol on its own task set. The draft should have separated these two scoring paths: the in-loop path uses our own rubrics, and the reported path follows the benchmark's own evaluation. For GameCraft-Bench and GameDev-Bench that is the benchmark's official verifier and rubric categories; for V-GameGym the reported Overall keeps the official Code, Screenshot and Video channels, while the judge producing those channel scores is our own text judge, which we disclose separately and will state in the caption. The rubric driving admission is therefore not the rubric behind the reported numbers.

There is, however, one respect in which this concern is well founded: a set used repeatedly for admission decisions is an adaptive validation set in the statistical sense, and our framework selection has therefore seen it. We will correct the wording, stating that the 20% set is a selection set and that Tables 1-3 report held-out official benchmark scores for a framework frozen at the end of evolution. Our rubrics establish that the scored quantity is not the benchmark's own rubric.

To make this checkable we will add the exact task partition per benchmark as an appendix table, a frozen-framework re-evaluation that takes the harness frozen at epoch T, runs it once on the reported task set and reports that number as the headline, and an explicit statement that the harness is transferred unchanged to Terminal-Bench and τ²-Bench with no re-optimization there.

Q2. Does the outer loop beat L3 at a matched compute budget?

This is the sharpest question in your review, and the current draft does not answer it.

The admission rule enforces budget matching within each A/B pair, comparing three aggregate cost quantities between parent and child: the number of model calls, evaluator queries and probes. Any mismatch marks the case invalid. Within a pair the two sides therefore run at matched cost.

The cumulative cost is not equal. Ours consumes more time than L3, and the 1.5-1.7× figure comes from the Fig. 7 ablation on GameCraft-Bench and covers the two backbones plotted there, Kimi and GLM5.2, not all six. The outer loop adds proposal generation and verification on top of the full inner loop, and the current figure plots score against time, which conflates the two.

We will add a spend-matched comparison, plotting Ours and L3 against cumulative token consumption and additionally running L3 with its epoch budget extended until its spend matches Ours. Two things in the draft require correction: the runs do not log per-epoch token consumption, so this needs usage accounting and re-running rather than re-plotting; and the Fig. 7 caption states that token consumption is shown as point size, which the instrumentation does not support, so we will correct the caption to report time cost. If L3 catches up at matched spend we will report that. We will also report the token breakdown into proposal generation, verification and inner-loop rollouts so the multiplier is decomposable.

Q3. Repeated independent evolution runs and uncertainty estimates

This is a genuine gap. Every number in Tables 1-3 is a single evolution run per cell, and we will label them as single-run in the captions.

For two representative cells, GLM5.2 on GameCraft-Bench and GPT-5.5 on V-GameGym, we will run three independent evolution seeds and report mean ± std with the per-seed trajectory, so the reported gains can be separated from run-to-run noise; all remaining cells will be labelled single-run. We state this as a planned addition.

On what the gate enforces: a child is admitted only if no hard rubric fails, no single valid case falls more than a fixed tolerance below its paired parent, and the median paired change is not negative. The gate guards against harmful changes. It does not test whether the median improvement is distinguishable from run-to-run noise, and it does not substitute for the variance estimate above.

Q4. Which partitions produced Tables 1-4? Was the framework frozen? Is transfer re-optimized?

Tables 1-3 use all non-held-out tasks for evolution, with the 20% selection split excluded, and the reported numbers follow each benchmark's published scoring protocol on its own task set: the official verifier and rubric categories for GameCraft-Bench and GameDev-Bench, and for V-GameGym the official Code, Screenshot and Video channels with the per-channel judge being our own text judge, which we disclose and will state in the caption. The framework is frozen after the final epoch and not updated during the reported evaluation. For Table 4 and Fig. 4, the harness evolved on game benchmarks is transferred unchanged to Terminal-Bench 2.0 and τ²-Bench v1.0.1 with no re-optimization, which we will state explicitly in Sec. 6.1.

On task counts, the two numbers come from two releases rather than from subsetting one inventory. The first release (arXiv:2602.11103v1) contains 132 tasks, 115 base and 17 variants, and is the release our paper cites and the one all our numbers use; the current release contains 333 tasks, 154 base and 179 variants. The v2 tasks postdate our runs and were never candidates under our protocol. We will state the version in the caption and in Sec. 5, and re-run our strongest backbones on v2 so our numbers can be read against the current leaderboard.

Q5. Equal total time or token budget versus L3?

As in C2, matched total budget is not yet established: within-pair matching is enforced, and across-variant matching is what we will add.

Q6. Minimum valid-case count K, judge configuration, case-exclusion rules, and whether failures count as failures

All four belong in the main text. The inner and outer loops share the same admission configuration, so these parameters apply once to the framework.

The minimum valid-case count is K = 3 in both loops on both GameCraft-Bench and GameDev-Bench, so if fewer than three paired cases survive the validity filter the comparison is void and no admission decision is taken. Rubric validation is mandatory in both loops, and if validation is missing or does not explicitly accept, the proposal is rejected. The sample size is 3 cases drawn from the same paired case set that passed the validity filter. A case is invalid under exactly three conditions: mismatched paired budgets on any of the three cost quantities, an infeasible replay outcome, or a missing final benchmark score.

On failures caused by the proposed framework, these count against it: a candidate that fails to build or run is an infeasible replay outcome that blocks admission. Infrastructure failures on the evaluator side are tracked under a separate status used to pause the run. We will add a short protocol box in Sec. 4 with these four items, stated once for both loops.

We are grateful for your careful reading of our protocol.

Thank you for the responses and clarifications. The rebuttal has largely addressed my concerns, although I remain cautious about the interpretation of the reported efficiency gains. I also consider the concerns raised by the other reviewers important, so I will maintain my current score. I would also understand if the final decision were not to accept the paper.

Reviewer_8Dw9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a kind of a meta-evolutionary method for Automatic Game Generation, where the inner loop is a "Game Orchestration Agent" (GOA) that builds the game when the outer loop, named "Harness Proposal Method" (HPA), proposes a new harness. When a new harness is proposed, GOA reruns the same task under the previous harness, and the new harness compares the results through an A/B test on author-curated rubrics. The outer loop, or HPA, turns the builder's past runs into a recurring failure types, and maintains a library of reusable harness elements based on how often each one is used and is accepted.

The authors use OpenGame as the agent framework. They test 6 different LLMs on four game benchmarks. They also test transfer learning on two general agent benchmarks. They use a vanilla baseline and a fixed-skill baseline. Overall, all scores improve across all benchmarks. Authors briefly perform ablations as well.

Critical strengths

The strengths are:

  • The meta-evolutionary setup allows Game-Evolver to give extra focus to each step of game generation, as, theoretically, one harness can focus on fixing one failure case while the other evolved harness can guide the creation of some other feature.
  • 4 benchmarks, 2 transfer learning benchmarks, and 6 models are quite sufficient for experimentation. For me, it is always important to have exhaustive experimentation.
  • Theory is well explained. Every aspect is mathematically formulated. It is good to see that.
Critical weaknesses

The weaknesses are:

  • Related work is a little bit off. It would have been better to have a section on how LLMs have been used in Automatic Game Generation and also how agent harness is being used in Automatic Game Generation. In fact, the term Automatic Game Generation is not being used at all, which should have been attributed.
  • LLM judges are not reliable, as LLMs can hallucinate. Rubrics being generated by an LLM should itself be evaluated.
  • There is no focus on Playtester. There is no mention of a search-based algorithm or something being used in the main text.
  • In Figure 6 the authors show how per-epoch scores vary, but there is no way to know if there is diversity within epochs.
  • Ablations are good not exactly explained how they are made. There are no LLM-as-a-judge ablations as well. It is definitely difficult to ablate but it should be.
  • Ablations have different timescale and token counts. Timescale is difficult to be precisely equal for each ablation but token count can be. Baselines should be turned to maximum thinking mode or something to make it equal to 200k "Ours".
Questions for authors
  • What is an Harness? Not all readers would know about this.
  • What is RSI?
  • Is your Figure 1 and 2 generated through an LLM?
  • What exactly is a playtester?
  • Can you make apples-to-apples comparison for ablation studes?
  • How can you test LLM-as-a-judge?
  • How do you test if Game-Evolver is not reward-hacking into a certain area of the game generation space to get higher scores?
  • How would you create a measure of diversity?

I overall like the idea but paper needs to answer above questions in order to be transparent and self-contained.

2 comments

We thank you for the constructive and detailed reading. We address every point below.

Q1. What is a harness?

A harness is everything around the frozen LLM that turns it into a working game-development agent. Formally (Sec. 4), H = <M, U, π_ctx, π_wf, π_val, Z>: instruction modules, engine/workspace tool interfaces, the context compiler, the workflow policy, the validation policy, and reusable assets. The backbone weights never change; only the harness is edited. We will move this definition to the first page of Sec. 3.

Q2. What is RSI?

RSI denotes recursive self-improvement: an agent improving the process that produces its outputs rather than only a single output. In our setting the inner loop improves one game artifact and the outer loop improves the harness that builds games. We will spell out the acronym at first use in Sec. 1.

Q3. Are Figures 1 and 2 LLM-generated?

Both figures are author-drawn schematics; a generative image model contributed only some decorative elements inside the panels. Concretely, we decided which components appear, how they connect, the panel arrangement and every label, and we assembled the diagrams ourselves. The icons and illustrations within some panels were generated with an image model and then placed and sized inside our own layout. We will add one sentence to each caption stating this.

Q4. What exactly is a playtester?

The playtester is the runtime-evidence collector, not a separate learned model. It takes the built artifact, collects the runtime evidence the dual-rubric judge scores, and checks the artifact against the demonstration input traces shipped with each task. A game that imports but never responds to input scores zero.

The two scoring paths differ. On the reported path the benchmark's own verifier launches the project in a real engine and replays each recorded trace, producing the replay logs the score is computed from. On the in-loop admission path the checks are lighter: every trace with actionable input events must have a corresponding completed replay log, no log may carry a parse, script or fatal marker, and the project must pass structural checks, so admission's runtime evidence is inherited from the verifier's replays. We will describe the two paths separately in Sec. 3.2 and state which path each reported number comes from.

Q5. Can you make the ablations comparable?

We agree the current presentation is under-specified. Within a comparison the budgets are equal: the admission rule rejects any case whose paired budgets differ on the three recorded cost quantities (model calls, evaluator queries, probes), marking that case invalid. What Fig. 7 does not show is the cumulative budget across L0-L3 and Ours.

We will replot Fig. 7 as score against cumulative spend with time cost as a secondary axis. Two precisions: the runs record per-cell scores and per-epoch checkpoints but not per-epoch token consumption, so the cumulative curve requires usage logging and re-running rather than re-plotting existing data; and the caption currently describes token consumption as encoded in point size, which the instrumentation does not support, so we will correct it to report time cost.

We will also state the construction in Sec. 7: L0 prompts only, L1 adds policies, L2 adds skills, L3 adds tools and workflow as the full inner loop, and Ours adds the outer loop. Each level differs from the next by exactly one harness component class.

Q6. How can you test LLM-as-a-judge?

This is the weakest point of the current draft and a real gap. Our current safeguards are three, and we will add a fourth. The hard rubrics are not LLM judgments: legality and runnability are decided by deterministic execution evidence. Rubrics are instantiated before scoring and frozen within each pair, and both sides are compared under budgets matched on the three recorded cost quantities.

What we will add is a judge-reliability study on a stratified sample of pairwise comparisons, scored independently by human annotators, reporting raw agreement, Cohen's κ and the judge's self-consistency across re-runs. If agreement is low we will report it and re-weight the soft-rubric share.

Q7. How do you test that Game-Evolver is not reward-hacking into a region of game space?

We have two mechanisms and one planned test. The two gates operate on different objects: the hard gate is boolean and per-case, while the score-level check compares each case's final score against its paired parent with a fixed tolerance. An aggregate soft sub-score can therefore decline while both gates pass, and it happens in our own table: on GameCraft-Bench the GPT-5.5 Art sub-score falls from 32.94 to 23.40 while Overall rises from 39.47 to 42.17. We will report that trade-off in the ablation discussion.

The hard rubrics are fixed, dataset-agnostic and boolean: launch without crash, respect for task constraints, a runnable artifact, and no hidden-test leakage. A child is rejected if any hard rubric fails on any valid case, and every demonstration trace containing actionable input events must have a corresponding completed replay log.

The planned test is to evaluate the final evolved harness on held-out game families it never saw during evolution, for example evolving on platformer, strategy and tycoon families and testing on horror and rhythm. This is a new experiment: the split used throughout the paper is a random 20% task-level held-out set, not a family-level hold-out, and our per-genre breakdown shows gains concentrated in a few game types.

Q8. How would you create a measure of diversity?

We propose to report, per epoch, the distribution rather than only the mean of dual-rubric scores across the evaluated cases: standard deviation, interquartile range, and the fraction of cases at the floor or ceiling, with a shaded band around the mean line in Fig. 6. Producing it requires extending the logging to write the complete per-case score vector each epoch and re-running the affected runs.

W1. Related work and the term "automatic game generation"

We will use the term explicitly and attribute it, and add a dedicated related-work paragraph on LLM-based automatic game generation (GameCraft-Bench, GameDev-Bench, OpenGame, V-GameGym, VeriGame, RSIGame) alongside harness-level self-improvement, so the two literatures are visibly separated.

The separating axis is the object a method improves, not whether it is recursive. RSIGame improves the per-run game artifact and internalizes development experience into the generator through supervised fine-tuning; our outer loop improves the harness, which is applied by configuration and transferred without retraining.

W2. Playtester and search-based algorithms in the main text

We will promote the playtester from the appendix to Sec. 3.2 and 4. On search: our method is a bounded hill-climb over harness configurations with a hard feasibility filter, not a search over game code, and we will say so explicitly.

W3. Figure 6 within-epoch diversity

As in Q8, the band requires per-epoch per-case score vectors, while the current artifacts retain per-epoch aggregates and only a truncated slice of per-pair deltas, so it means extending the logging and re-running.

W4. Ablation construction and LLM-judge ablation

Construction is as in Q5. On the judge ablation, we will add a variant with the soft-rubric share set to zero, hard rubrics only, and one with the judge replaced by the dataset's official evaluator, so the contribution of the LLM judge is isolated.

W5. Different timescales and token counts; baselines at maximum thinking

Within an A/B pair the admission rule matches the three cost quantities it records, model calls, evaluator queries and probes, but not token counts, since token spend is not instrumented. Across the variants cumulative token spend differs and we will make that comparison spend-matched, which as noted in Q5 needs instrumented re-runs.

On maximum thinking mode, all backbones were run at the lowest reasoning tier their provider exposes, applied uniformly: GPT-5.5 with reasoning explicitly disabled, Claude-Sonnet-4.6, GLM5.2, Qwen3.6-27B and DeepSeek-V4-Flash at their non-reasoning default, and Kimi-K2.7-Code at low thinking, the same setting on both sides of every pair. Our GPT-5.5 baseline at that tier falls within 0.02 Overall of the published Codex + GPT-5.5 high row, with all four categories matching to two decimals; that row is a Vision row and our configuration is vision-enabled in the same two places, screenshots supplied to the judge behind Visuals and Art and rendered frames available to the developing agent, while the scaffold, judge model and effort level differ. We have added the per-backbone setting to the Table 1-3 captions and an appendix recording the exact setting for every reported number, and we will re-run the strongest backbones at the highest tier each provider accepts, stating the actual tier for each line.

We are grateful for the care you brought to this review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.