acceptodds
Under review as a conference paper at ICLR 2027

GameLogicBench: Evaluating Coding Agents on Runtime Game Logic with Tick-Level State Assertions

Abstract

Coding agents can now modify and test code across large software projects. Game development is one such domain, where agents must implement gameplay rules. A game can end in a valid state even after violating its rules during the run. Current game-development benchmarks replay fixed examples, score videos, or ask another model to judge the result. However, no existing benchmark checks game rules throughout execution across varied evaluator-selected scenarios while ensuring exactly reproducible verdicts. We introduce GameLogicBench, a benchmark of 72 gameplay-logic tasks in Godot projects. An automated evaluator checks each game's rules at every simulation tick. Across 403 hand-designed scenarios, seeded parameter variations produce 1,451 test cases. To ensure that the evaluator measures behavior rather than implementation choice, it must accept different correct implementations for each task while rejecting mutants, implementations with one required capability removed. The tasks span isolated mechanics, multi-system interactions, and repository-scale features. Across 20 combinations of language models and scaffolds, the best observed run solves 52.78% of tasks. Under Claude Code, all twelve models solve fewer tasks as task scope expands from isolated mechanics, through interacting systems, to repository-scale features. Agents inspect code more often and make more tool calls on repository-scale tasks than on isolated-mechanic tasks. Most unsuccessful submissions are runnable, but implement some required game behavior incorrectly. We compared versions of our benchmark evaluator built with and without validation using mutants. Without this validation, some incorrect agent submissions passed. A separate analysis finds agents copying code from the original public repositories when network access is open. Reliable evaluation thus depends both on what the tests reject and on what external code agents can access.

Primary area:
datasets and benchmarks
Keywords:
coding agents, game development, runtime verification, behavioral evaluation, software engineering benchmarks, Godot

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bWHxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper builds a benchmark containing 72 gameplay-logic tasks in Godot projects for game development agents. All tasks come with tick-level assertions that can be deterministically verified without needing a judge model. The tasks span isolated mechanics, multi-system interactions, and repository-scale features. To ensure the soundness of the evaluator, careful techniques such as mutants and multiple scenarios are used. Studies were done on trending open-source and frontier models.

Critical strengths

The core evaluation design of using tick-level assertions is quite novel to me, and it aligns closely with runtime game logic than alternatives such as final state checks or VLM judging. This is a design that is highly specific to game dev (because of the usual main loop).

The construction of the benchmark is careful with mutant-based judge validation and the use of multiple scenarios. The use of different task scales is also commendable. The authors are also very clear about leakage if network access is open, and thus decided to run benchmark in sealed environment.

Critical weaknesses

While it is extremely neat to have deterministically verifiable signals in a game dev benchmark like tick-level assertions, the authors did not quantify how universal such signals can be designed. It would be great to have a discussion of the coverage: what kind of game mechanics can and cannot be covered in such tick-level assertions. The 7 capability classes in table 7 all seem quite low-level and correspond to very small, rigid mechanics. As such, it wouldn't serve as a benchmark for building moderate to large-scale mechanics which inevitably would involve evaluation of funness, aesthetics, pacing, or other fuzzy metrics.

Another weakness is that the size of the benchmark is still relatively small at 72 tasks, possibly due to the high authoring cost of involving human in many parts of the pipeline.

Questions for authors

Can authors discuss how to extend the tick-level assertions to bigger scale mechanics beyond examples like state transitions and local/runtime correctness? If not possible, what other ways can we evaluate bigger mechanics?

1 comment

We thank the reviewer for the positive feedback on our tick-level assertions and mutant-based judge validation.

W1 & C1. Coverage of tick-level assertions and scaling to larger mechanics.

  1. Coverage (W1). A mechanic is covered when runtime state and event history determine its correct behavior at every tick, which takes three forms:

    • A unique valid state, e.g., a guard may chase an intruder only when the line of sight is clear.
    • A required event order or timing, e.g., a weapon cannot fire again before its cooldown ends.
    • A conserved quantity, e.g., supplies in depots, in transit, and held by units always equal production minus consumption.

    Mechanics judged by look or feel, such as visual appeal, control feel, and pacing, are screened out (§2.2).

  2. Larger mechanics (W1, C1). Larger subsystems are evaluated by combining tick-level invariants from several capability classes. Each class in Table 7 categorizes what a single assertion checks, while Repo tasks implement complete subsystems of up to 945 lines inside projects of up to 62,895 lines. Our character-controller task, for instance, checks the state transitions, timing windows, and spatial constraints of a movement component shared by the animation, camera, and input systems.

  3. Fun, aesthetics, and pacing (W1, C1). State assertions do not measure fun, aesthetics, or pacing, so evaluating larger mechanics requires pairing deterministic functional checks with experiential raters scoring control responsiveness, pacing, visual readability, and engagement. Among automated raters, scripted screenshot or video capture with a VLM rubric judge is inexpensive, yet misses states outside fixed recordings and differs from an interactive player's perspective. Pure GUI playtesting agents match human play more closely, yet remain costly and unreliable on complex games even with frontier models. Combining CLI control with GUI playtesting in a hybrid agent judge offers a practical path forward that mirrors how human QA testers actually work: using engine-side commands to reach target or edge-case states directly, and then playing through the GUI to assess visual presentation and feel.


W2. Benchmark size.

We plan to expand the benchmark in two ways: (1) collecting mechanics from more genres and from other engines that support headless fixed-step inspection, and (2) accepting community pull requests on GitHub under the same calibration, agent review, and human review pipeline.

Reviewer_tWXpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GameLogicBench evaluates whether coding agents correctly implement runtime game logic. It contains 72 Godot tasks spanning isolated mechanics, interacting mechanics, and integration into existing repositories, with 403 scenarios and 1,451 test cases. Its central design combines tick-level state and event assertions across evaluator-selected scenarios, with evaluator calibration using correct solutions, behavior-preserving alternatives, and single-defect mutants. Across 20 model–scaffold configurations, the best observed single-run solve rate is 52.78%. According to the evaluator’s classifications, 74.3% of failed scenarios involve runnable but behaviorally incorrect solutions. Ablations support execution-time checking and scenario diversity, while a separate open-network audit examines the influence of upstream source retrieval.

Critical strengths
  1. On the same 36-task audit set, terminal-only scoring accepts 236/666 known-defective mutants, while preview-only scoring accepts 508/666, corresponding to 35.4% and 76.3% respectively (Table 5). Because these mutants contain identified defects, this provides stronger evidence than score disagreement alone. It directly supports the paper’s central contribution: evaluating game-logic correctness during execution across varied conditions.
  2. The three integration tiers and 20 model–scaffold configurations expose behavioral failures beyond compilation or launch success. Separating solving from judging, sealing network access in the main experiments, and separately auditing source retrieval also improve interpretability. Together, these choices make the benchmark a useful resource for studying agents’ ability to implement complex runtime behavior.
Critical weaknesses
  1. Correct references and mutants provide important calibration, but they also participate in constructing and repairing the evaluator. Appendix H.1.2 explicitly does not require an independently written correct implementation to pass, and the behavior-preserving control in H.1.3 may simply refactor the reference. Consequently, the validation strongly supports rejecting anticipated defects but provides less evidence about false rejection of other implementations satisfying the public contract. Independent verification and human review are valuable safeguards, but the paper does not report independently adjudicated correctness labels for actual agent submissions. Because scoring validity is central to a benchmark contribution, this limits the reliability claims.
  2. The central ablations use 36 of the 72 tasks without clearly explaining their selection or tier and capability distributions, leaving uncertainty about how well the measured effects generalize to the full benchmark. The main table also reports one attempt per configuration, with repeated evaluation covering only three configurations. Their single-attempt score standard deviations are 3.67–4.24 percentage points (Table 4). This does not undermine the benchmark’s overall difficulty or large performance differences, but smaller ranking and scaffold differences should be treated as preliminary observations.
Questions for authors
  1. For how many tasks were the behavior-preserving controls independently developed from agent-visible materials using substantially different designs? Were samples of passing and failing agent submissions independently adjudicated, and how often did those judgments disagree with the evaluator?
  2. How were the 36 audit tasks selected, and how do their tier and capability distributions compare with the full benchmark? Were defects or submissions excluded from evaluator repair used to test whether improvements transfer beyond the calibration set?
2 comments

We thank the reviewer for recognizing our mutant-based scoring ablation and sealed evaluation setup, and for the constructive suggestions.

W1, C1 & C2 (part 2). Judge calibration, control diversity, independent adjudication, and out-of-calibration transfer.

  1. Independent human implementations (W1). Because the same construction agent builds the judge and its calibration examples, we also checked false rejection in a pre-submission review of all 1,440 agent submissions, none of which were used to calibrate the judge. Writing an independent human implementation for every task would substantially increase authoring cost, since Repo tasks sit inside existing game projects averaging 17,599 lines of code. During construction, a discoverability check verifies that every rule and constant checked by the judge is stated in the brief or readable in the starter codebase, and our 72 controls check that the judge accepts alternative implementations (Appendix G.1). After release, we will also accept community misjudgment reports, fix confirmed judge bugs, and rescore affected submissions (Appendix J.2).

  2. Control designs (C1). Of the 72 behavior-preserving controls, 56 use a substantially different algorithm or design and 16 perturb intermediate state. The 16 state-perturbed controls keep the same method as the proper solution so the judge cannot rely on proper-solution internals. All 72 controls are written by the implementation agent, while implementations written solely from agent-visible materials are tested across the 1,440 agent submissions below.

  3. Independent adjudication (C1). Across all 1,440 passing and failing submissions, our independent review disagreed with the judge on 39 runs (2.71%), with a 3.56% false-rejection rate and a 2.32% false-positive rate. An isolated AI agent first labeled each submission from the task brief and code diff without seeing the judge. Whenever its label disagreed with the judge, it inspected the judge code to locate the discrepancy, and we verified every judge error by hand. The confusion matrix below compares the calibrated pre-fix judge against the confirmed ground truth across all 1,440 main-table submissions:

    Ground truth \ Pre-fix judgeJudge PASSJudge FAILTotalJudge error rate
    Actually correct433164493.56% (false-negative / false-rejection rate)
    Actually incorrect239689912.32% (false-positive rate)
    Total4569841,4402.71% overall disagreement (39 / 1,440)

    We fixed all 39 errors before submission, so Table 3 already uses the corrected judge, and we now report this review in §2.3 and Appendix G.2.

  4. Transfer beyond the calibration set (C2, part 2). Mutant-driven judge fixes overturned 3 false passes in agent submissions, and the pre-fix judge matched independent review on 97.29% of all 1,440 runs. Neither set of agent submissions was used to build or repair the judge. In our 36-task validation comparison, mutant checks prompted 24 judge fixes that overturned those 3 false passes across 2 tasks and 3 models while all proper solutions and behavior-preserving controls still passed.

W2 & C2 (part 1). 36-task subset selection, 72-task ablations, and single-attempt variance.

  1. 36-task selection and 72-task ablations (W2, C2). The 36 tasks are a stratified sample of roughly half of each tier (Atom 10/21, Combo 13/28, Repo 13/23), covering all seven capability classes, and both scoring ablations now cover all 72 tasks with closely matching results. The table below compares Terminal-only and Preview-only scoring from the 36-task subset to all 72 tasks across all 893 mutants and 991 failed main-table submissions:

    MetricTerminal-only (36 tasks → all 72 tasks)Preview-only (36 tasks → all 72 tasks)
    Escaped mutants236 / 666 (35.4%) → 300 / 893 (33.6%)508 / 666 (76.3%) → 688 / 893 (77.0%)
    Tasks with escapes34 / 36 → 55 / 7236 / 36 → 72 / 72
    FAIL → PASS submissions64 / 488 (13.1%) → 157 / 991 (15.8%)418 / 488 (85.7%) → 833 / 991 (84.1%)
    Mean Δ solve rate+8.9 points → +10.9 points+58.1 points → +57.8 points

    Comparing judges before and after mutant validation stays on these 36 tasks because we archived pre-validation judges only for them, and we state this scope explicitly in §1, §4.6, and §6.

  2. Single-attempt variance (W2). Across three attempts for each of the top six configurations, the mean rank order shows no reversals relative to Table 3. The table below reports each configuration's three single-attempt solve rates, mean, standard deviation across the three attempts (SD, as in Table 4), and 95% CI bootstrapped over the 72 tasks (Attempt 1 is the Table 3 run):

    ModelSolve rate, attempts 1 / 2 / 3 (%)MeanSD95% CI of the mean
    Claude-Opus-552.78 / 50.00 / 52.7851.851.60[42.59, 61.57]
    Qwen-3.8-Max44.44 / 50.00 / 41.6745.374.24[36.11, 55.09]
    GPT-5.6-Sol41.67 / 36.11 / 44.4440.744.24[31.02, 50.46]
    DeepSeek-V4-Pro-081341.67 / 34.72 / 36.1137.503.67[28.24, 47.22]
    GLM-5.237.50 / 30.56 / 36.1134.723.67[25.46, 44.44]
    Kimi-K336.11 / 37.50 / 30.5634.723.67[25.93, 43.52]

    We mark small single-attempt gaps in §4.3, §4.4, and §6 as preliminary observations, such as GLM-5.2's 5.56-point scaffold gap, while keeping large-margin claims such as Qwen-3.8-Max's 18.05-point gap.

Reviewer_FkvFOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GameLogicBench, a benchmark of 72 Godot gameplay-logic implementation tasks spanning isolated mechanics, interactions among mechanics, and features integrated into existing repositories. The evaluator deterministically checks runtime state and event history at every simulation tick across 403 hand-designed scenarios and 1,451 seeded test cases.

Benchmark admission requires that a proper solution and a behavior-preserving alternative pass, while naive solutions and single-capability mutants fail. Across 20 model–scaffold configurations, the best single run solves 52.78% of tasks. The study also examines task tiers, scaffold effects, rerun stability, test-protocol ablations, and repository-source retrieval contamination.

Critical strengths

A strong deterministic oracle design. Inspecting runtime invariants throughout the simulation avoids the obvious failure mode in which a game reaches an acceptable final state after violating essential rules in intermediate ticks (Secs. 2.1 and 3.1).

Unusually serious evaluation validation. Proper solutions, behavior-preserving rewrites, naive solutions, and targeted single-capability mutants are used as benchmark-admission controls (Secs. 2.2–2.3). The audited 36-task ablation shows that terminal-only and preview-only evaluation can substantially overestimate success (Table 5).

Transparent contamination analysis. The sealed-network protocol is supported by a paired open-network audit showing direct upstream-code reuse and higher Repo scores in several configurations (Appendix D). This is an important practical consideration for code benchmarks.

Useful breadth within the chosen domain. The suite spans 21 Atom, 28 Combo, and 23 Repo tasks, twelve source genres, multiple runtime capability classes, and 20 model–scaffold configurations (Table 2; Table 3).

Critical weaknesses

Generalization beyond the chosen engine is uncertain. All tasks use Godot 4.4/GDScript, and the protocol excludes visually judged mechanics, multiplayer synchronization, and subjective player experience. Only 23 tasks are repository-tier tasks (Table 2). These restrictions are reasonable for reliable scoring but limit the scope of conclusions about general game development.

The criterion-validity gate may not cover the diversity of correct implementations. Each task uses a proper solution and a behavior-preserving alternative, which is a strong start, but one alternate rewrite cannot establish that the runtime-state oracle is implementation independent. Mutants are constructed from the anticipated capability decomposition, potentially missing unrelated failure modes or rejecting unexpected legitimate mechanics. Independent solutions from expert developers would offer stronger validation.

Model ranking and tier interpretations lack comprehensive uncertainty analysis. Most of the 20 configurations are evaluated once; only three configurations have three-attempt stability results (Table 4). A one-task difference changes the headline solve rate by 1.39 points. Furthermore, tier comparisons are observational: Repo tasks differ from Atom tasks in project size, mechanism type, and implementation burden, not integration scope alone (Table 2). The results support a difficulty gradient but not a clean causal attribution.

The most compelling scoring ablations use a 36-task audit subset. The terminal-only, preview-only, and mutant-hardening comparisons are strong evidence on that subset but should not be described as exhaustively demonstrating error rates over all 72 tasks (Sec. 4.6).

Questions for authors

Have any tasks been evaluated using independently authored correct implementations not involved in judge calibration? What is the observed false-rejection rate, and which internal state contracts, if any, proved implementation specific?

How representative are the 36 audited tasks of the full 72-task library, especially by tier and capability class? Can the main scoring ablations be extended or accompanied by sampling uncertainty?

Can repeated attempts be reported for the leading model–scaffold combinations, with paired task-level comparisons and confidence intervals? How often do adjacent rankings reverse?

Could the authors include a matched subset controlling for implementation size or mechanism type to better isolate the effect of cross-system integration?

2 comments

We thank the reviewer for recognizing our tick-level assertions, evaluation validation, and contamination analysis.

W1. Scope boundaries and generalization beyond Godot.

Although our experiments cover a single engine, the assertion method can transfer to any engine with headless, fixed-timestep execution and per-tick state access (for example, Unity batch mode with manually stepped physics). The benchmark includes mechanics whose rules can be checked on runtime state, such as collision, line-of-sight occlusion, root motion, and animation-event timing. When a mechanic also involves rendered appearance or subjective player experience, tick-level assertions can verify the functional rules first, and human, multimodal-model, or interactive agent judges can then score qualities such as control responsiveness and pacing. We state these scope boundaries in the Limitations and Future Work section and detail the engine requirements in Appendix J.1.


W2 & C1. Implementation diversity, false-rejection rate, and state-contract case study.

  1. False-rejection rate (W2, C1). Across all 1,440 main-table submissions, our pre-submission review found a 3.56% false-rejection rate and a 2.32% false-positive rate. None of these submissions were used to calibrate the judge. An isolated AI agent first labeled each submission from the task brief and code diff without seeing the judge. Whenever its label disagreed with the judge, it inspected the judge code to locate the discrepancy, and we verified every judge error by hand. The confusion matrix below compares the calibrated pre-fix judge against the confirmed ground truth across all 1,440 submissions:

    Ground truth \ Pre-fix judgeJudge PASSJudge FAILTotalJudge error rate
    Actually correct433164493.56% (false-negative / false-rejection rate)
    Actually incorrect239689912.32% (false-positive rate)
    Total4569841,4402.71% overall disagreement (39 / 1,440)

    We fixed all 39 errors before submission, so Table 3 already uses the corrected judge, and we now report this review in §2.3 and Appendix G.2.

  2. Control diversity (W2). Of the 72 behavior-preserving controls, 56 use a different algorithm or design and 16 perturb intermediate state. For example, a pathfinding control replaces Godot's built-in navigation with custom grid search, whereas the 16 state-perturbed controls keep the same method as the proper solution and check that the judge does not rely on its internals.

  3. Independent expert solutions (W2). Writing an independent human expert solution for every task would substantially increase authoring cost, since Repo tasks sit inside existing game projects averaging 17,599 lines of code. During construction, a discoverability check and 72 alternative controls verify specification completeness and implementation neutrality without a human-written solution: every rule and constant checked by the judge must be stated in the brief or readable in the starter codebase (Appendix G.1). Beyond construction, our review of all 1,440 agent submissions above measured false rejection on unseen solutions directly, and after release we will also accept community misjudgment reports, fix confirmed judge bugs, and rescore affected runs (Appendix J.2).

  4. Implementation-specific state contracts (C1). False rejections arose when checks assumed monotonic progress or solution-specific intermediate state: in combo_corridor_deadlock, an early-stop check assumed every unit moves continuously toward its goal and wrongly rejected a valid solution. Four units must cross a narrow corridor in opposite directions within 560 ticks. One submission backed a unit up to let oncoming units pass, and the judge stopped the run as a deadlock even though its schedule brings all four units to their goals by tick 368. We removed this check and rescored affected runs (Appendix G.3).

W3 (Ranking) & C3. Repeated attempts, paired comparisons, and rank reversals.

Across three attempts for each of the top six Claude Code configurations, the mean rank order shows no reversals relative to Table 3. Claude-Opus-5 leads or ties in every attempt. The table below reports each configuration's three single-attempt solve rates, mean, standard deviation across the three attempts (SD, as in Table 4), and 95% CI bootstrapped over the 72 tasks (Attempt 1 is the Table 3 run):

ModelSolve rate, attempts 1 / 2 / 3 (%)MeanSD95% CI of the mean
Claude-Opus-552.78 / 50.00 / 52.7851.851.60[42.59, 61.57]
Qwen-3.8-Max44.44 / 50.00 / 41.6745.374.24[36.11, 55.09]
GPT-5.6-Sol41.67 / 36.11 / 44.4440.744.24[31.02, 50.46]
DeepSeek-V4-Pro-081341.67 / 34.72 / 36.1137.503.67[28.24, 47.22]
GLM-5.237.50 / 30.56 / 36.1134.723.67[25.46, 44.44]
Kimi-K336.11 / 37.50 / 30.5634.723.67[25.93, 43.52]

We also added paired task-level comparisons between adjacent ranks, reporting the mean pass-rate difference, bootstrap 95% CI, and per-task win/loss/tie counts:

Adjacent pairMean diff. (pts)95% CIWin / Loss / Tie
Claude-Opus-5 − Qwen-3.8-Max+6.48[−1.85, +15.28]18 / 14 / 40
Qwen-3.8-Max − GPT-5.6-Sol+4.63[−2.78, +12.50]18 / 13 / 41
GPT-5.6-Sol − DeepSeek-V4-Pro-0813+3.24[−5.09, +11.57]20 / 14 / 38
DeepSeek-V4-Pro-0813 − GLM-5.2+2.78[−5.09, +10.65]15 / 12 / 45
GLM-5.2 − Kimi-K30.00[−6.94, +6.94]15 / 17 / 40

Larger gaps are statistically significant, for example Claude-Opus-5's 11.11-point lead over GPT-5.6-Sol (95% CI [+0.93, +21.30]). Adjacent ranks are close, and single attempts swap them in only 2 of 15 comparisons. We expand Table 4 to all six configurations, report the paired comparisons in Appendix I, and mark small gaps among the remaining single-attempt configurations as preliminary observations.


W3 (Tiers) & C4. Observational tier gradient and same-mechanism cross-tier pairs.

  1. Causal wording (W3). We revised §4.1 to describe tier differences observationally, replacing the causal heading with "Solve rates decline as task scope broadens."

  2. Same-mechanism pairs (C4). Across 20 same-mechanism cross-tier pairs, configurations passing the lower-tier check failed the identical higher-tier check in 39 of 132 cases (29.5%). During construction, 21 higher-tier tasks reused a lower-tier check unchanged and recorded the link (20 with check-level logs). Because total code size still differs within each pair, we report these results as supporting observations in Appendix H.


W4 & C2. 36-task subset representativeness and full 72-task scoring ablations.

  1. Representativeness (C2). The 36 tasks are a stratified sample of roughly half of each tier (Atom 10/21, Combo 13/28, Repo 13/23), covering all seven capability classes.

  2. Full-benchmark coverage (W4, C2). Both scoring ablations now cover all 72 tasks, with results close to the 36-task estimates. The table below compares Terminal-only and Preview-only scoring from the 36-task subset to all 72 tasks across all 893 mutants and 991 failed main-table submissions:

    MetricTerminal-only (36 tasks → all 72 tasks)Preview-only (36 tasks → all 72 tasks)
    Escaped mutants236 / 666 (35.4%) → 300 / 893 (33.6%)508 / 666 (76.3%) → 688 / 893 (77.0%)
    Tasks with escapes34 / 36 → 55 / 7236 / 36 → 72 / 72
    FAIL → PASS submissions64 / 488 (13.1%) → 157 / 991 (15.8%)418 / 488 (85.7%) → 833 / 991 (84.1%)
    Mean Δ solve rate+8.9 points → +10.9 points+58.1 points → +57.8 points

    Comparing judges before and after mutant validation stays on these 36 tasks because we archived pre-validation judges only for them, and we state this scope explicitly in §1, §4.6, and §6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.