acceptodds
Under review as a conference paper at ICLR 2027

A2Z GameSpec-Bench: How Faithfully Can Coding Agents Generate Games from Game Design Specifications?

Abstract

Delegating complete application development to coding agents requires preserving the intended design rather than simply producing plausible outputs through naive prompting. Game development provides a demanding testbed, as long-form Game Design Documents (GDDs) describe requirements that must work together across game logic, visual rendering, and player interactions. However, existing game-development benchmarks typically use compact specifications and provide limited support for evaluating interdependent requirements across these aspects in long-form GDDs. We introduce , a benchmark of 100 long-form GDDs for evaluating end-to-end game development by coding agents. We measure by checking whether the resulting game satisfies the GDD requirements and preserves the relationships among them. Each GDD is turned into a that contains rules, constraints, and prerequisite relations. Following game-development practices, we combine source-code inspection with agent-generated for scenario-based replay and adaptive playtesting. The contract remains fixed across agents and revision rounds, while judgments and evidence linked to the same requirements support consistent comparison and failure diagnosis. Our evaluations show that current agents struggle to jointly satisfy interdependent requirements across code implementation and actual play. Requirement-specific feedback improves GDD Fidelity by 10.9% relative gain over self-revision after two rounds. A2Z GameSpec-Bench assesses end-to-end specification-following ability beyond individual implementation judgments and provides targeted feedback to support more faithful game development.

Primary area:
datasets and benchmarks
Keywords:
agentic game generation, benchmark, specification-driven evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_a9PsOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a benchmark based on implementing games that fulfill detailed game design documents by integrating game components along the document requirements. First, 100 generated game design documents were generated and converted into contracts specifying their components and their relationships. Benchmark evaluation requires the tested game rules to satisfy the contract (which is not provided to the agent), tested by playtesting from various scenarios including some targeted to the design rules and using visual judges for rendered behavior.

Evaluations show current models do not fully solve the benchmark. Providing feedback based on the errors compared to the contract improves model performance.

Critical strengths

Ensuring testing addresses component relationships in a design is important for progress on game and general software generation.

Critical weaknesses

The goal contract is hidden from models. While this makes sense from an evaluation perspective it is odd if the primary goal is to improve on that contract, not any external criteria withheld from the model. I would have liked to see results on a standard benchmark (GameDev-Bench) for comparison, even if that requires inferring the GDDs needed for those games.

It is not clear the GDDs or goal contracts match human game design documents. Without this, it is not clear the benchmark targets what it intends to. Showing specifications have causal language is a useful correlate without addressing the intended goal.

Questions for authors

What effort were the models that implemented the code run at? This makes a substantial difference to performance, particularly given the rapid recent advancements in the field like GPT 6 astra.

What variability was there over the three runs? The three evaluations? This helps determine how large the gains are compared to fluctuations.

5 comments
AuthorsOpenReview ↗

Response to W1

[W1] External Validation of Contract-Based Feedback

We appreciate your suggestion to validate the usefulness of our feedback against external evaluation criteria. We extend our evaluation to GameDevBench [1] in Appendix G.7 and Table S27 to compare revisions guided by our evaluator feedback with retries and self-review by the GPT-5.6-Sol agent.

ConditionPass rate (%)Gain (pp)
Initial build49.0-
Retry52.0+3.0
Self-review53.0+4.0
Ours57.0+8.0
  • Setup: We evaluate the held-out test pass rate on Godot development tasks from GameDevBench after one revision round. All conditions use GPT-5.6-Sol as the coding agent and start from the same initial build for each task. Our evaluator derives verifiable requirements from the provided task instructions and generates feedback through source-code inspection and test-policy-based runtime checks. GameDevBench's held-out test scripts remain unavailable to both the coding agent and our evaluator and are used strictly for final evaluation.

  • Results: Our evaluator feedback increases the held-out test pass rate from 49.0% to 57.0%, compared with 52.0% for retry and 53.0% for self-review. The 8.0 percentage-point improvement from our evaluator feedback is twice the gain achieved by self-review. Thus, our requirement-level feedback improves task success under independent external tests withheld from both the coding agent and our evaluator.

Additionally, [Common Response 2] demonstrates that our source-code evaluator accurately detects execution-verified implementation defects at the requirement level. Taken together, these experiments provide strong evidence for both reliable requirement-level error detection and the practical utility of our benchmark.


Reference

[1] Wayne Chi et al., "GameDevBench: Evaluating Agentic Capabilities Through Game Development," ICML, 2026.

AuthorsOpenReview ↗

Response to W2

[W2] Representativeness of GDDs and Evaluation Objectives of the Benchmark

Thank you for pointing out the need to clarify the benchmark's purpose. As discussed in Section 1, our focus is on how faithfully coding agents implement the requirements specified in a GDD. Accordingly, the primary objective of our benchmark is requirement-level implementation fidelity from specifications, rather than performance on human-written GDDs specifically. However, we appreciate that this evaluation is meaningful if those requirements capture important aspects of game design. With this in mind, we design our GDDs around the following goals:

  • To support the integrity of requirement-based evaluation, we construct specifications with clear and consistent evaluation targets rather than directly using human-authored GDDs. Starting from human-provided game ideas, our dataset generation pipeline produces structured GDDs and checks them for omissions, contradictions, and incorrect formulas as described in Appendix A.1–A.4. This approach is motivated by requirements-engineering work on translating game-design documents into requirements suitable for production [1]. Therefore, we use a systematic authoring and validation process to make the requirements explicit and assessable.

  • For agentic game development, we emphasize whether the requirements convey the intended game design rather than the surface naturalness of their representation. Our GDDs specify core gameplay loops, mechanics, progression, and rewards as described in Appendix A. Also, our goal contracts formalize the corresponding implementation requirements so that we can evaluate whether generated games faithfully implement these design elements, as described in Appendix B. Recent frameworks incorporate agent-based specification authoring [2, 3], and this highlights that a faithful implementation of agent-generated specifications is closely relevant to these emerging agentic workflows. Related benchmarks such as WebGameBench [4] and WebRISE [5] also structure task requirements for evaluation.

    • For example, in traces\texttt{traces}_$\texttt{left}$, players place sensors at two of three locations to gather clues for an inference puzzle, as shown in Figure S2 in Appendix A.6. Allowing sensors at all three locations would remove the need to decide which location to leave unobserved. Our contract explicitly encodes the two-sensor restriction, preserving this intended gameplay choice as explained in Appendix B.4.

We revise Section 1 to clarify this intended use of our benchmark. We appreciate your valuable feedback.


Reference

[1] David Callele et al., "Requirements Engineering and the Creative Process in the Video Game Industry," IEEE RE, 2005.

[2] Yixu Huang et al., "GUI Agents for Continual Game Generation," Findings of EMNLP, 2026.

[3] Xiangxin Zhou et al., "Game-Creation-Agents: Autonomous Game Production," arXiv:2610.16776, 2026.

[4] Wenyu Zhang et al., "WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games," arXiv:2605.17637, 2026.

[5] Yuxin Meng et al., "WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts," EMNLP, 2026.

AuthorsOpenReview ↗

Response to Q1

[Q1] Reasoning Effort of Coding Agents

Thank you for highlighting the importance of coding agents' reasoning effort. We clarify the agent effort settings in Appendix C. Additionally, we analyze the effect of different reasoning-effort settings on coding-agent performance in Table S11. We reproduce Table S11 below, and scores indicate GDD Fidelity on a 0–100 scale.

Coding agentmediumxhighGain (points)
Claude-Opus-593.094.9+1.9
Claude-Opus-4.880.184.9+4.8
GPT-5.555.587.3+31.8
GPT-5.6-Sol55.282.5+27.3
  • In our main experiments in Section 4 and Table 2, coding agents with configurable reasoning effort use xhigh or the equivalent maximum supported setting, including GPT-6-Astra at xhigh. No effort parameter is assigned to Kimi-K2.7-Code because its vLLM serving setup does not expose this control.

  • Effort comparison (medium vs. xhigh): We also compare medium and xhigh settings for four coding agents on the same 10 Big GDDs, generating one build per task and setting. Increasing effort improves GDD Fidelity by 31.8 points for GPT-5.5 and 27.3 points for GPT-5.6-Sol, compared with 1.9 points for Claude-Opus-5 and 4.8 points for Claude-Opus-4.8. We observe that reasoning effort can substantially affect performance, with the magnitude of this effect varying across the evaluated models. We use the maximum supported reasoning-effort setting, such as xhigh, to assess agents' ability to implement complex, long-form specifications without limiting performance through a restrictive reasoning-effort setting.

Reviewer_byNnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents A2Z GameSpec-Bench, a benchmark of 100 long-form game design documents (GDDs) for evaluating end-to-end game development by agents. Each GDD is turned into a fixed dependency-aware contract that contains rules, constraints, and prerequisite relations. The authors combine source-code inspection with agent-generated Test Policies for scenario-based replay and adaptive playtesting. Experiments show that current agents struggle to jointly satisfy interdependent requirements between runnable builds and faithful implementations. Three-axis feedback improves GDD fidelity compared with self-revision after two rounds.

Critical strengths
  • It is well-motivated to evaluate coding agents' ability to faithfully implement long-form game design specifications with interdependent requirements.
  • The authors evaluate a broad range of frontier coding models across games of varying complexity, with detailed feedback ablations examining which forms of guidance help agents correct specification violations.
  • Playtest reveals problems in how individual mechanics interact during gameplay, including failures in rules that the source-code judge had marked as fully implemented.
  • The revision results show that feedback on specific requirements helps agents achieve higher GDD Fidelity than direct review of GDD the source code.
Critical weaknesses
  • The paper does not validate whether the Dependency-Aware Contract preserves complex gameplay requirements. An ending that depends on the order of earlier choices may be misrepresented if the extracted conditions record only which choices occurred.
  • Failed and unverified requirements both receive zero credit, but the main results lack per-model playtest coverage to clarify what drives the score differences between agents.
  • The revision experiment reports only overall GDD Fidelity, so the results do not show which evaluation axis improves more with feedback.
  • Potential same-family bias in the GPT-based judges is not assessed. The authors should verify the reported rankings using a judge from another model family.
Questions for authors
  • Could the authors provide a breakdown of GDD Fidelity by game type to identify which types are most challenging for coding agents?
  • For each model in Table 2, what proportions of rules are satisfied, violated, or unverified?
9 comments
AuthorsOpenReview ↗

Response to W1

[W1] Preservation of Order-Dependent Gameplay Requirements

Thank you for raising the question of whether our dependency-aware contract preserves requirements that depend on the order of player choices. We provide an explanation and an ablation in Appendix G.1.3 with Figures S21 and S22.

Our contract represents these dependencies through state-valued preconditions and effects. For example, the first choice records its value in state, and the next rule checks that state together with the subsequent action. As illustrated in Figure S21, choosing A then B and choosing B then A therefore produce different states and can trigger different endings. The same mechanism appears in our benchmark games (e.g., Safe Dial\texttt{Safe Dial}, Shadow Pair\texttt{Shadow Pair}, and Chord Snap\texttt{Chord Snap}).

We additionally evaluate 385 order-dependent rules across the 50 Small GDDs with and without their state preconditions. The table below shows the satisfaction rates from Figure S22, using the same adaptive-playtest traces in both settings.

Coding agentWith state preconditions (%)Without state preconditions (%)Increase (pp)
Claude-Fable-5.183.686.5+2.9
Claude-Opus-584.487.3+2.9
Claude-Opus-4.876.180.8+4.7
GPT-6-Astra82.185.7+3.6
GPT-5.6-Sol79.281.6+2.4
GPT-5.581.683.9+2.3
GLM-5.370.773.6+2.9
DeepSeek-V4-Pro30.132.2+2.1
Kimi-K2.743.851.1+7.3

Removing these preconditions increases satisfaction rates by 2.1-7.3 percentage points across all nine coding agents. This shows that our evaluator accounts for the specified prior state when assigning scores to sequential requirements under our contract.

AuthorsOpenReview ↗

Response to W2

[W2] Per-model Playtest Coverage and Score Differences

Thank you for requesting per-model playtest coverage to clarify what drives the score differences between coding agents. We address this concern through verdict decomposition and cross-evaluator comparisons as follows:

  1. Separating observed violations from unverified requirements: In [Common Response 4] and Figure S17 in Appendix F.3, we report the satisfied\texttt{satisfied}, violated\texttt{violated}, and unverified\texttt{unverified} proportions for every coding agent. Although violated and unverified requirements both receive zero credit, this breakdown explicitly distinguishes observed implementation violations from requirements without conclusive evidence. Additionally, Table S16 in Appendix F.3 shows that adversarial playtesting substantially expands requirement coverage by approximately 40–59 percentage points, ranging from 53.3% to 88.0% across all nine coding agents.

  2. Coding agent comparisons remain consistent across different judges: As shown in [Common Response 4] and Figure S17 in Appendix F.3, the satisfied\texttt{satisfied} proportion, which determines the playtest score, differs by at most 5.4 percentage points from the GPT-5.6-Luna judge, and playtest rankings remain identical across all three evaluators. Additionally, overall GDD Fidelity rankings remain identical, as reported in [Common Response 1] and Table S17 in Appendix F.4. These results show that evaluator choice does not significantly change the ordering of coding agents.

AuthorsOpenReview ↗

Response to W3

[W3] Axis-Wise Improvements from Revision Feedback

We appreciate your request for a breakdown of the revision gains by evaluation axis. We add an axis-wise decomposition of the score improvements after two revision rounds in Appendix E.3. We reproduce Table S13 below. Score values are absolute changes on a 0–1 scale. The source-code columns report changes in unweighted rule satisfaction and invariant pass rates.

Revision feedbackSource code: RuleSource code: InvariantPlaytestReplay
Self-revision+0.23+0.10+0.04−0.01
Source+0.27+0.26+0.17−0.01
Source + Playtest+0.25+0.26+0.22+0.00
Source + Playtest + Replay+0.26+0.25+0.23+0.01

Our three-axis feedback increases playtest fidelity by 0.23, compared with 0.04 under self-revision. Rule satisfaction and invariant pass rates also improve by 0.26 and 0.25, respectively. However, replay improvement remains small at 0.01. As analyzed in Appendix E.3, 96% of replay feedback items concern visual rendering omissions rather than logic errors. These results suggest that visual requirements remain difficult for GPT-5.6-Sol to improve during revision in our benchmark. Please refer to Appendix E.3 and Table S13 for further details.

Reviewer_tfeoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a benchmark containing 100 long-form game design specifications, focusing on how the generative code acts behaviorally aligned to the proposed specifications. It also designed a benchmarking protocol that converts the specification document into a set of validation rules, checking it by source code inspection, scenario replay, and adaptive play testing.

Critical strengths
  • This paper has a clear benchmarking goal with a fine-grained consideration about the importance of the rules (dependency-weighted contract scoring).
  • It offers a comprehensive benchmark pipeline, ranging from source code inspection and play testing.
Critical weaknesses
  • Implementation quality is not sufficiently disentangled from evaluation effectiveness. Although replay scenarios are fixed, the game-building agent supplies the input sequences and their timing, while an LLM-based judge selects the frames to inspect. Adaptive playtesting similarly relies on model-generated test programs and model-based interpretation of collected evidence. Consequently, a correctly implemented requirement may receive no credit because the test fails to exercise it or collect sufficient evidence. The appendix provides evaluator comparisons and human calibration, including separate correlations for replay, playtest, and overall scores. However, these correlations use the same aggregate human rating, cover games generated by only one coding-agent configuration, and do not directly validate source-code judgments. They therefore do not establish requirement-level accuracy or identify which stages cause evaluation errors. More controlled validation is needed to attribute low scores to implementation defects rather than test-generation, evidence-collection, or judgment failures, and to establish whether these uncertainties affect model rankings.

  • The relationship between dependency complexity and generation difficulty remains insufficiently characterized. The Small/Big comparison shows lower generation quality for broader game designs, but does not distinguish the effect of having more rules from that of composing rules through more complex dependencies. The appendix provides evidence that dependency information affects assessment and helps identify potential failures. However, the paper does not examine whether games with comparable rule counts become harder to generate as their dependency structures become more complex. Given the central role of dependency-aware contracts, such an analysis would help connect the evaluation methodology to a substantive empirical finding about agents’ limitations in implementing interacting rules.

Questions for authors
  • Can you validate the evaluation pipeline using controlled cases with independently established outcomes? For example, following the mutation-testing approach in TestGenEval [1], one could keep the GDD fixed, introduce known rule violations into verified implementations, and check whether the corresponding scores reliably decrease while the original implementations receive credit. Alternatively, inspired by Cardiverse [2], Appendix B.4, one could keep gameplay records fixed and compare scores against matching versus deliberately inconsistent requirements, with the behavioral mismatches independently verified. Such experiments would provide more direct evidence of error-detection accuracy than agreement between evaluators alone.

[1] Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2025. TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark. In Proceedings of the Thirteenth International Conference on Learning Representations (ICLR 2025).

[2] Danrui Li, Sen Zhang, Samuel S. Sohn, Kaidong Hu, Muhammad Usman, and Mubbasir Kapadia. 2025. Cardiverse: Harnessing LLMs for Novel Card Game Prototyping. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 29735–29762. Association for Computational Linguistics.

  • Is the final metric, GDD fidelity, only correlated with the number of the rules? Or if we control the number of the rules, does the composition complexity of the rules—in other words, the complexity of the dependency—also affect the GDD fidelity metric?

  • How robust are the evaluated coding agents’ rankings to the choice of evaluator base model? Appendices F.2 and G.2 report agreement and score differences across evaluator configurations, but do not establish whether these differences affect the final rankings. Could you re-evaluate the same generated builds using an alternative evaluator model and report changes in overall scores and rankings, particularly for closely performing agents?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1

[W1] Disentangling Implementation Quality from Evaluation Effectiveness

Thank you for highlighting the need for controlled validation that distinguishes implementation defects from evaluation errors. We conduct direct requirement-level validation and comparisons across all nine coding agents in Appendix G.6 with Tables S25 and S26.

As detailed in [Common Response 2], we follow the mutation-testing approach in TestGenEval and introduce targeted defects into game builds from our benchmark dataset while keeping the GDD and evaluation items fixed. Deterministic execution checks verify that the original build satisfies the targeted outcome and that the injected defect causes the modified build to violate it. These mutations cover numerical errors, boundary conditions, state-dependent behaviors, and procedural generation, as described in Appendix G.6. As shown in Table S26, our evaluator both awards full credit to the original target and reduces its score after mutation in 84.1% of the 728 contract items. This supports that our evaluator detects confirmed implementation defects at the requirement level.

We additionally compare source code scores with deterministic execution-based scores across all nine coding agents in Table S25. The mean absolute difference is 1.4 percentage points, and Spearman's ρ\rho is 0.98. We find that our source-code evaluation produces scores and rankings closely consistent with deterministic execution checks. Furthermore, there are identical overall rankings across three different evaluator agents as shown in [Common Response 1].

For playtesting, we report the satisfied\texttt{satisfied}, violated\texttt{violated}, and unverified\texttt{unverified} proportions for every coding agent in [Common Response 4]. Specifically, the satisfied proportion differs by at most 5.4 percentage points across the alternative evaluators relative to GPT-5.6-Luna, while both the playtest ranking and the overall GDD Fidelity ranking remain identical.

AuthorsOpenReview ↗

Response to Weakness 2

[W2] Dependency Complexity and Implementation Difficulty

We appreciate that rule count and dependency complexity should be examined separately. We add comparisons within rule-count groups in Appendix G.1.2 and Figure S20.

As explained in [Common Response 3], we define dependency complexity through the number of prerequisite links in the longest chain of rules, because we observe that deeper chains require more upstream conditions to be satisfied for downstream behaviors to work.

From our analysis, longer dependency chains are associated with lower GDD Fidelity within all three rule-count groups, with a pooled within-group correlation of -0.42. These results show that complex dependencies with deeper chains of rules are harder for agents to implement faithfully, even among games with comparable rule counts.

Please refer to [Common Response 3] for the analysis and Appendix G.1.2 for further details.

AuthorsOpenReview ↗

Response to Question 1

[Q1] Controlled Validation with Independently Established Outcomes (TestGenEval)

Thank you for suggesting controlled mutation testing. As detailed in our response to [W1] and [Common Response 2], we follow the mutation-testing approach in TestGenEval and conduct the proposed validation in Appendix G.6 and Table S26.

Please refer to [Common Response 2] for the complete results and protocol.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Initial Comment with Common Responses

Dear Reviewers,

We sincerely appreciate your valuable feedback and constructive comments. Following suggestions, we strengthen the manuscript with the following experiments and analyses:

  • We evaluate the generated games using two additional Claude judges to assess ranking robustness and potential same-family bias.
  • We validate source-code judgments through controlled mutations with execution-verified outcomes and comparisons against deterministic execution-based scores.
  • We analyze implementation difficulty among games with comparable rule counts to examine the role of dependency complexity.
  • We report satisfied, violated, and unverified requirement proportions for each coding agent to distinguish observed violations from requirements without conclusive evidence.

We present these results in the four common responses below and address individual reviews separately.

AuthorsOpenReview ↗

Common Response 1 & 2

[Common Response 1] Cross-Family Judge Comparison (tfeo-Q3, byNn-W4)

We examine ranking robustness and potential same-family bias by additionally evaluating the games generated by all nine coding agents in Table 2 using Claude-Opus-5.5 and Claude-Sonnet-5.5. Appendix F.4 presents the comparison with our original GPT-5.6-Luna evaluator.

The following results correspond to Table S17. Each cell lists source-code, replay, and playtest scores, respectively, on a 0-100 scale.

Coding agentGPT-5.6-LunaClaude-Opus-5.5Claude-Sonnet-5.5Overall GDD Fidelity rank
Claude-Fable-5.186.6 / 64.3 / 80.092.1 / 60.9 / 76.092.2 / 59.1 / 77.61
Claude-Opus-583.2 / 62.8 / 75.788.1 / 59.9 / 72.588.2 / 57.9 / 73.52
GPT-6-Astra79.3 / 61.0 / 74.586.7 / 58.1 / 69.186.5 / 56.3 / 70.53
GPT-5.568.5 / 51.7 / 69.671.2 / 47.4 / 65.371.6 / 46.7 / 67.64
GPT-5.6-Sol62.4 / 56.1 / 67.068.3 / 52.0 / 62.868.2 / 50.5 / 64.55
Claude-Opus-4.861.1 / 52.0 / 57.466.3 / 47.7 / 53.466.7 / 46.5 / 55.26
GLM-5.368.4 / 46.1 / 52.668.7 / 41.9 / 50.472.6 / 41.1 / 51.67
DeepSeek-V4-Pro64.8 / 39.1 / 47.562.9 / 35.2 / 46.567.7 / 34.5 / 46.98
Kimi-K2.749.1 / 33.4 / 36.048.4 / 29.1 / 34.451.6 / 28.8 / 35.99

All three evaluators yield identical overall rankings across the nine coding agents, with Kendall's τ\tau = 1.00. Overall score changes relative to GPT-5.6-Luna are at most approximately 2.3 points, without changing the ordering of even closely performing agents (e.g., GPT-5.5 vs. GPT-5.6-Sol). Thus, the agent comparison remains stable when evaluation changes from GPT to Claude-family judges.


[Common Response 2] Controlled Evaluation with Deterministic Execution Checks (tfeo-W1, tfeo-Q1, a9Ps-W1)

We provide two complementary validations in Appendix G.6 with Tables S25 and S26. These experiments examine whether the source-code evaluator responds to independently verified implementation defects, and whether its comparison of coding agents agrees with deterministic execution-based evaluation.

1. Requirement-level validation through controlled mutations

Following the mutation-testing approach in TestGenEval [1], we keep the GDD and evaluation items fixed and introduce targeted code mutations into game builds from our benchmark dataset to assess whether our source-code evaluator detects known behavioral defects. Deterministic execution checks establish that the original build passes the targeted requirement, the mutated build fails it, and both builds pass the unrelated control checks. Then, GPT-5.6-Luna at high reasoning effort evaluates each version without receiving mutation descriptions, execution results, or information identifying the mutated version.

Table S26 shows the results below. "Both conditions" counts cases where the original target receives full credit and its score decreases after mutation.

Requirement typePairsOriginal full creditScore decreasedScore unchangedScore increasedBoth conditions
Claims611555 (90.8%)508 (83.1%)967501 (82.0%)
Invariants117114 (97.4%)111 (94.9%)60111 (94.9%)
Total728669 (91.9%)619 (85.0%)1027612 (84.1%)

In 84.1% of pairs, the source-code evaluator both awards full credit to the original target and reduces its score after mutation. These results support that our source-code judgments detect actual implementation defects at the requirement level.

2. Agreement with deterministic execution-based scores

For each requirement, execution checks set up its preconditions, trigger the relevant action, and assert the expected postcondition. We aggregate the resulting pass or fail outcomes using the source-code fidelity formulation in Equation (4), and average the game-level scores.

Table S25 compares these execution-based scores with our source-code evaluation across all nine coding agents.

Coding agentExecution-based score (%)Source-code fidelity (%)Absolute difference (pp)
Claude-Fable-5.185.486.61.2
Claude-Opus-583.083.20.2
Claude-Opus-4.863.261.12.1
GPT-6-Astra78.779.30.6
GPT-5.6-Sol65.262.42.8
GPT-5.570.168.51.6
GLM-5.369.068.40.6
DeepSeek-V4-Pro64.964.80.1
Kimi-K2.752.849.13.7
Mean70.369.31.4

Accordingly, seven of the nine agents maintain identical rank positions with Spearman's ρ\rho = 0.98. We observe that our source-code evaluation produces scores and rankings closely consistent with deterministic execution checks.


Reference

[1] Kush Jain et al., "TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark," ICLR, 2025.

AuthorsOpenReview ↗

Common Response 3 & 4

[Common Response 3] Dependency Complexity and Implementation Difficulty (tfeo-W2, tfeo-Q2)

We examine whether agents struggle more with interacting rules even when games have comparable rule counts. We define dependency complexity as the number of prerequisite links in the longest chain of rules, because deeper chains require more upstream conditions to be satisfied for downstream behaviors to work. This rationale is supported by the strong correlation between rule depth and the cumulative number of conditions required to trigger a rule (Spearman's ρ\rho = 0.85) as shown in Appendix G.1.2. We group GDDs into three rule-count ranges and examine the relationship between dependency depth and GDD fidelity within each group.

We extract the results from Figure S20.

Number of rulesNumber of GDDsSpearman's ρ\rho (dependency depth and GDD fidelity)
≤1223-0.19
13-2023-0.16
≥2144-0.48

Longer dependency chains are associated with lower fidelity in all three groups, with a pooled within-group correlation of -0.42. Our analysis shows that complex dependencies with deeper chains of rules are harder for agents to implement faithfully, even among games with comparable rule counts.


[Common Response 4] Requirement Verdicts Across Evaluators (tfeo-W1, byNn-W2, byNn-Q2)

We examine the requirement-level playtest verdicts in Figure S17 to distinguish observed violations from requirements without a conclusive verdict.

Each cell below lists satisfied\texttt{satisfied} / violated\texttt{violated} / unverified\texttt{unverified} requirements (%).

Coding agentGPT-5.6-LunaClaude-Opus-5.5Claude-Sonnet-5.5
Claude-Fable-5.180.0 / 8.1 / 12.076.0 / 3.3 / 20.777.6 / 2.6 / 19.8
Claude-Opus-575.7 / 9.4 / 14.972.5 / 4.2 / 23.473.5 / 3.4 / 23.1
GPT-6-Astra74.5 / 9.0 / 16.669.1 / 3.0 / 27.970.5 / 2.6 / 26.9
GPT-5.569.6 / 12.1 / 18.365.3 / 6.9 / 27.867.6 / 6.3 / 26.1
GPT-5.6-Sol67.0 / 14.8 / 18.262.8 / 7.1 / 30.164.5 / 6.7 / 28.8
Claude-Opus-4.857.4 / 14.2 / 28.553.4 / 6.3 / 40.355.2 / 5.5 / 39.2
GLM-5.352.6 / 13.4 / 34.050.4 / 6.7 / 42.951.6 / 5.9 / 42.4
DeepSeek-V4-Pro47.5 / 13.3 / 39.246.5 / 13.1 / 40.446.9 / 13.0 / 40.0
Kimi-K2.736.0 / 17.3 / 46.734.4 / 9.9 / 55.735.9 / 9.0 / 55.1

The breakdown separates the zero-credit share into observed violations and unverified requirements for each coding agent. The satisfied proportion determines the playtest score and differs by at most 5.4 percentage points from GPT-5.6-Luna. Remarkably, the playtest score ranking remains identical across all three evaluators. This shows that comparisons of requirement satisfaction are consistent across evaluator agents in our playtests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.