acceptodds
Under review as a conference paper at ICLR 2027

GameEngineBench: Full-Session Verification of Coding Agents in a Networked Game Engine

Abstract

Coding agents face new challenges in game development, where they must produce code that is not only syntactically correct but also consistent with engine-managed lifecycle, state, and networking behavior. Existing game-development benchmarks, however, do not combine assertions over engine-based runtime states with cross-client consistency, failing to evaluate how well generated code adheres to the desired behavioral specifications. To address these limitations, we present GameEngineBench, a benchmark of 110 C++ implementation tasks derived from 9 existing Unreal Engine 5 projects. Each task requires an agent to implement missing behavior within an established gameplay architecture and a restricted editable scope. We also introduce Full-Session Verification, a testing framework that injects hidden tests after the agent finishes and evaluates the resulting implementation inside networked Play-in-Editor (PIE) sessions. The tests evaluate Unreal's game loop, actor lifecycle, subsystems, and replication stack while directly inspecting runtime state. Across 17 benchmark tasks, tests additionally inspect a distinct client world to determine whether the authoritative server state is correctly replicated. To our knowledge, this is the first coding benchmark to evaluate cross-client replication. A detailed analysis of 768 mutated implementations finds that 74% of distributed-property bugs are detected only by end-to-end tests involving full PIE sessions. Across model–harness configurations, the strongest achieves 55.5% pass@1, while 31 tasks remain unsolved by every evaluated configuration. These results demonstrate that compilation and shallow verification overestimate agents' ability to implement behavior that remains correct during full engine execution.

Primary area:
datasets and benchmarks
Keywords:
coding agents, benchmark, verification, game engine

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_aPGdOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

GameEngineBench evaluates coding agents on 110 C++ feature-implementation tasks derived from nine Unreal Engine 5 repositories. Its hidden verification suites progress from static and unit checks to synthetic-world integration tests and full networked Play-in-Editor sessions. Separate server and client worlds enable checks of lifecycle ordering and distributed state consistency; 17 tasks directly inspect client-world behavior. The paper evaluates 12 model-harness configurations, with the strongest achieving 55.5% pass@1, and analyzes 768 known-wrong mutants across 108 tasks. Full-session tests detect substantially more injected defects than shallower tiers, particularly distributed failures, illustrating a useful gap between plausible code and correct engine behavior.

Critical strengths
  1. The benchmark targets a consequential capability that compilation and isolated function tests cannot establish: correct behavior over a running engine lifecycle and across networked worlds (Sections 3-4). Hidden post-solve tests, reference-pass/starter-fail validation, and review by two Unreal Engine experts are meaningful safeguards. The resulting tasks offer a concrete evaluation setting for long-horizon, repository-level implementation.
  2. The tier-by-tier mutation analysis provides evidence for the verification design, rather than only reporting a leaderboard (Section 5.5; Table 3). E2E tests catch 90% of all mutants versus 68% for integration tests; on distributed mutants, the corresponding rates are 92% and 15%. This is a valuable empirical demonstration of when deeper execution changes the observability of defects.
Critical weaknesses
  1. The reliability of the final E2E verifier needs further quantification (Section 5.5; Appendix B). The 195-run comparison uses human-reviewed E2E verdicts as its reference, but does not explain whether reviewers independently established correctness or how disagreements were resolved. Its error-rate interpretation therefore depends on an insufficiently described adjudication process. Approximately 10% of known-wrong mutants also escape E2E detection, without a detailed account of those blind spots. Appendix B.5 describes sensible stability controls, but does not report repeated-run verdict consistency. These gaps matter because verification errors or unstable outcomes could affect benchmark rankings.
  2. Result provenance is not consistently documented. Section 4.3 describes an execution-based pass rule, whereas Figure 6(e) refers to missing judge labels being treated as non-passes and is captioned as the task-level matrix behind the aggregate ranking. Appendix A also presents Task 19 as unsolved, while Appendix D says it is solved by Fable 5. These could be stale annotations rather than errors in the official results, but the manuscript needs a reconciled task-by-configuration ledger before readers can confidently interpret its pass rates and complementarity analysis.
Questions for authors
  1. What exactly are the judge labels in Figure 6(e), and does that panel use the same task set, configurations, and pass predicate as the main leaderboard? Please reconcile the Task 19 descriptions and identify any affected aggregate results.
  2. How were the 195 runs manually reviewed, and were disagreements with E2E verdicts independently adjudicated? Please characterize undetected known-wrong mutants and report repeated-run verdict consistency for fixed reference and agent solutions.
1 comment
AuthorsOpenReview ↗

Response to Reviewer aPGd

We thank the reviewer for the careful review. We respond to the concerns and questions brought up below.

W1 / Q2 (verifier reliability). Two experts reviewed 195 graded submissions (39 tasks sampled at random with a fixed seed, stratified by project and client-world type, with five fixed configurations per task) against the task specification, behavioral contract, and code changes, each recording an assessment without consulting the other; 81 of these were rejected by an end-to-end test. On the 30 submissions reviewed by both experts, they initially agreed on 27 (90%), and the three disagreements, which arose from ambiguous specifications, were resolved through discussion. No end-to-end verdict was overturned (Section 4.3). The submitted per-tier error rates on these runs, which used the end-to-end verdict as their reference, are withdrawn. We also quantify reliability on the suite that defines pass@1: every reference solution passes in both grading runs (220/220, after two first attempts of one load-sensitive test failed and passed on rerun), in 819 of the 821 submissions that build and fail a test the failing test maps to a named requirement, and regrading changed no verdict (0/1,210 submissions; 0/100 repeats; Appendix B.5, Table 5). The submitted mutation table (768 mutants, E2E 90%) graded separate per-tier suites and could not be reproduced. In its place, on real submissions, 209 of the 493 that pass every static, unit, and integration test fail an end-to-end test (69 of 269 on tasks that also have integration tests), and removing end-to-end tests would raise pass@1 by 12.7–20.9 points per configuration (Section 5.4); on the frozen suite, distributed mutants are killed by 0% of unit tests, 11.9% of integration tests, and 85.1% of end-to-end tests, although integration tests kill more local-logic mutants than end-to-end tests (50.7% vs 42.0%; Table 3). Against the single frozen suite, 17 of 567 mutants (3.0%) survive: 13 weak assertions (the code runs but no assertion distinguishes the fault), 3 untested paths, and 1 fixture limit (the distinguishing condition cannot be produced deterministically). Four further surviving mutants were judged after grading to be likely equivalent or not constrained by the instruction, and were excluded. Appendix F lists all surviving mutants.

W2 / Q1 (result provenance, Task 19). The "judge labels" in Fig. 6(e) were a remnant of an LLM-based adjudication component in the submitted version, which is withdrawn; one ledger of 1,210 verdicts (110 tasks × 11 configurations, compiled tests only) now feeds every figure, including Fig. 6(e), which uses the same tasks, configurations and pass rule as the leaderboard. No configuration solves Task 19, so Appendix A was right, and the complementarity examples (submitted Appendix D, now Appendix E) are rewritten. The aggregates that changed are best pass@1 55.5% → 46.4%, unsolved tasks 31 → 43, and configurations 12 → 11 (Kimi was not part of the evaluated run).

Presentation. We merged the scoring definitions into one section (Section 4.3), removed duplicated text, and explain technical terms at first use.

Reviewer_P28LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GameEngineBench contains 110 C++ feature-implementation tasks from nine Unreal Engine 5 projects. Hidden tests run after the agent finishes, using networked Play-in-Editor sessions; 17 tasks also inspect a separate client world. Twelve model–harness configurations are evaluated once per task, with the best reaching 55.5% pass@1. A mutation study and 195 human-reviewed runs compare different verification tiers.

Critical strengths
  1. The tasks capture temporal and distributed behavior that compilation and isolated unit tests can miss. They also sit inside real projects rather than small standalone exercises.
  2. The verification engineering is thoughtful: hidden-test injection after solving, editable-scope checks, connected client worlds, readiness polling and positive/negative controls.
  3. The paper tests its own verifiers. The reviewed runs show substantial over-acceptance by unit tests, and Fable 5's lead is supported by the paired task counts. The aggregate figures are mostly internally consistent.
Critical weaknesses
  1. The final labels and task set are ambiguous. The text says scoring is entirely executable, with no model in the loop, yet Figure 6(e) refers to missing “judge labels.” That panel also does not match Figure 3: nine of its ten rows have one extra pass, and removing Task 66 reconciles them. The paper mentions an “active” task set without listing excluded IDs or when exclusions were decided. This could be a stale figure, but reviewers need a frozen manifest and an explanation of Task 66. The models used for calibration and test construction are also unnamed.
  2. The mutation result says less than the headline suggests. “E2E only” means missed by unit and integration tests, not missed by every shallower verifier. In the Distributed row, Static detects 60% and “E2E only” detects 74%, so these sets must overlap. Further, unit tests have no networking and integration tests stay within one world, making part of the distributed-mutant gap expected by design. The useful missing number is the kill rate of the full suite that actually defines pass@1. Mutant generation, equivalent-mutant filtering and whether static checks affect final labels also need explanation.
  3. Reliability and the added value on agent solutions are underreported. The 195-run human review has no stated selection protocol or human-versus-E2E disagreement rate. Each configuration is run once, and verdict stability is not measured despite acknowledged nondeterminism. There is no separate pass rate for the 17 client-world tasks or breakdown of actual agent failures by property class. Most close ranking differences are also unresolved by the paired counts.
  4. The solver setup leaves possible confounds. It is unclear whether agents can run PIE or automation tests, access the web, or inspect git history containing the removed public implementation. Compile failures cluster strongly by harness, so model and tooling effects are hard to separate. The benchmark and task manifest are not yet released.
Questions for authors
  1. What are “calibrated pass” and “judge labels”? Can any model or human judgment change an executable verdict? Please list inactive task IDs, explain Task 66, and state when tasks/tests were frozen and which models helped calibrate them.
  2. What is the verdict flip rate on repeated tests of reference solutions and archived submissions? How were the human-reviewed runs selected, and how often did human review overturn E2E?
  3. Which suite defines pass@1, does it include static checks, and what is its per-class mutant kill rate? Please clarify mutant construction and revise “E2E only” accordingly.
  4. Please report results on the 17 client-world tasks and integration-versus-E2E outcomes for actual submissions. What tools and source history could agents access, and what artifacts can be released?
3 comments
AuthorsOpenReview ↗

Response to Reviewer P28L

We thank the reviewer for the detailed and constructive review. We respond to the concerns and questions brought up below.

W1 / Q1 (labels, task set, freezing, models). "Calibrated pass" and "judge labels" referred to an LLM-judge component of the submitted pass label; it is withdrawn, and every revised result comes only from compiled hidden tests, so no model or human can change a verdict. The 10 inactive tasks (16, 32, 33, 40, 41, 44, 66, 73, 76, 84) were deactivated before the final graded run and are not part of it; the frozen 110-task manifest is Table 14 (Appendix H). Fig. 6(e) still included Task 66, which explains the extra pass in nine rows, and all figures are now regenerated from one ledger. Before the suites were frozen, tests and instructions were revised after inspecting failures from preliminary runs of the evaluated configurations, so the results are not a held-out evaluation; after the freeze, all 11 configurations re-solved all 110 tasks, and only these submissions are graded. Claude Opus 4.8 assisted in writing the hidden tests, GPT-5.5 served as the calibration judge, and LLM agents helped triage failures and draft the revisions made before the freeze.

W2 / Q3 (mutation analysis). The frozen suite that defines pass@1 kills 550 of 567 mutants (97.0%, an in-sample upper bound); by class: local logic 97.1%, cross-file 96.7%, ordering 94.7%, distributed 97.0%, hollow 98.2%. "E2E only" is relabelled "killed, but by no unit or integration test" (194 mutants); of the 67 distributed mutants, 65 are killed, 55 by no unit or integration test, and 51 only by client-world assertions. Static and reflection assertions sit inside behavioral tests and count toward the verdict, but decide only 5 of 1,210 verdicts on their own. Mutants are LLM-drafted single faults, reviewed against a named requirement (998 drafted, 642 reviewed, 567 validated); four further surviving mutants were excluded after grading as likely equivalent or not constrained by the instruction. The submitted mutation table (768 mutants) could not be reproduced on the frozen suite and is replaced.

W3 / Q2 (reliability, real failures, rankings). The submitted per-tier comparison on the 195 runs, which scored each verification tier against the end-to-end verdict, is withdrawn. Separately, two experts reviewed these 195 graded submissions (selection recorded in Section 4.3: 39 tasks sampled at random with a fixed seed, stratified by project and client-world type, five fixed configurations per task) against the task specification, behavioral contract, and code changes, each without consulting the other; 81 were rejected by an end-to-end test. On the 30 reviewed by both, they initially agreed on 27 (90%) and resolved the three disagreements, which arose from ambiguous specifications, through discussion; no end-to-end verdict was overturned. Regrading all 1,210 submissions changed no verdict, 100 repeat grades changed none, and each of the 110 reference solutions passed in both grading runs (220/220, after two first attempts of one load-sensitive test failed and passed on rerun; Appendix B.5, Table 5). Classifying each failing test by the property it checks (with LLM assistance), 615 of the 821 submissions that build and fail a test fail a non-local property (ordering 316, distributed 280, cross-file 227; categories overlap). With Holm-corrected paired tests, 27 of 55 configuration pairs differ significantly and the top four are tied (Table 13); each configuration solves each task once, so solve-to-solve variance is not measured.

W4 / Q4 (solver setup, release). Agents could compile but were instructed not to run the editor or tests; none ran automation tests or a PIE session, and none had hidden tests, reference solutions, version history, or working web access (140 solves made 274 network attempts; none succeeded), so results measure implementation under compile-time feedback, and training-data exposure to the public projects cannot be excluded. Compile failures are 24 of 1,210 (Claude Code 4/550, Codex 4/330, Antigravity 3/110, Qwen Code 13/220), and results are reported per model–harness configuration rather than per model. On the 17 cross-client tasks, pass@1 is 20.9% versus 23.9% on the other 93; on the 53 tasks with client-world tests, pass@1 is 19.4% versus 27.3% on the other 57 (Table 9). On real submissions, 209 of the 493 that pass every static, unit, and integration test fail an E2E test (69 of 269 on tasks that have integration tests). The anonymised release (https://anonymous.4open.science/r/game-engine-bench-iclr27) contains the grader, frozen test suites, per-submission verdicts, mutants, the task manifest with licences, and setup scripts that rebuild each task from its public upstream repository.

Ethics. Licences and upstream provenance are documented in Appendix H, and LLM-assisted test construction is disclosed in the AI-use statement.

Thank you for the response and the release. I recomputed the main numbers from results/verdicts.csv and mutants/mutant_verdicts.csv: 51/110 for fable5-max, 43 unsolved tasks and 550/567 mutants killed. They match your reply. Moving to compiled-test verdicts, together with the stability and client-world results, addresses several of my concerns. However, the evaluation pipeline and all headline numbers changed after submission, and your replies conflict on human validation. I am therefore keeping my score at 2 for now and will reconsider once the points below are resolved.

  1. Your reply to me says the 195-run comparison is withdrawn and that no verdict has been validated by humans. Your reply to Reviewer QKUA says experts reviewed 195 submissions and overturned none. Which is correct? If the selection was not recorded, I would remove this analysis rather than cite it as validation.
  2. The main text should state that the submitted results used an LLM-judge label, and that the tests were revised after looking at failures of the evaluated configurations.
  3. Were all 110 tasks re-solved after the freeze, or only the tasks whose instructions changed? Your two replies differ on this.
  4. Please break down the 79 editor_crash_or_no_run failures by configuration, and say whether each was caused by the submission or by the environment.
AuthorsOpenReview ↗

Response to P28L

We thank the reviewer for recomputing our numbers from the release. Our previous reply to you was incorrect on two points, which we fix here; our earlier reply has been updated accordingly. We apologize for the confusion.

Q1: The expert review described in our reply to Reviewer QKUA is correct. Our earlier reply was drafted from an outdated source and inadvertently omitted the expert review. The submitted per-tier comparison is withdrawn, which scored tiers against the E2E verdicts, but the human review is a separate analysis to support the reliability. Two Unreal experts reviewed 195 graded submissions: 39 tasks sampled at random with a fixed seed, stratified by project and client-world type, with the submission of each of five fixed configurations. Each expert assessed the task specification, behavioral contract, and code changes without consulting the other. Expert A reviewed 112 submissions and Expert B 113, with 30 reviewed by both; they initially agreed on 27 of the 30 (90%) and resolved the three disagreements, which arose from ambiguous specifications, through discussion. Of the 195, 81 were rejected by an E2E test, and no verdict was overturned (Section 4.3).

Q2: The revised paper already states both points. The AI use statement notes that the submitted results used a pass label with an LLM-judge component, which is withdrawn. Section 4.3 states that tests and instructions were revised after inspecting failures from preliminary runs of the evaluated configurations before the suites were frozen.

Q3: All 11 configurations re-solved all 110 tasks after the suites were frozen, and only these submissions are graded. Before the freeze, tests and instructions were revised after inspecting failures from preliminary runs of the same configurations, so the results are not a held-out evaluation.

Q4: None of the 79 editor crashes is attributable to the grading environment. Every reference solution runs cleanly on the same machine image (220/220), and all 79 submissions also fail when regraded. In 37, the crash occurs in agent-written code or in an agent shader that fails to compile. In 25, an assertion or access violation fires in unchanged project, engine, or test code because of the submission's behavior (e.g., null components, direct calls to interface events, unbounded recursion). The remaining 17 are a latent bug in one project's upstream plugin (ALSXT), never triggered by references; we report them separately.

ConfigurationCrashesAgent code / shaderUnchanged code, caused by submissionUpstream plugin bugEnvironment
Gemini 3.1 Pro198650
Qwen 3.7 Plus146620
DeepSeek V4 Pro105230
GPT-5.5 (medium)72320
Claude Opus 4.8 (high)63210
Claude Sonnet 4.6 (high)64110
GPT-5.5 (high)51220
GPT-5.5 (xhigh)44000
Claude Opus 4.7 (high)32100
Claude Opus 4.8 (max)32100
Claude Fable 5 (max)20110
Total793725170

We hope this resolves the inconsistency!

Reviewer_QKUAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces GameEngineBench, a benchmark of 110 C++ implementation tasks from nine Unreal Engine 5 projects. It proposes Full-Session Verification, which evaluates agent-generated code in networked Play-in-Editor (PIE) sessions to verify lifecycle-dependent behavior and cross-client state consistency. Across 12 model–harness configurations, the best achieves 55.5% pass@1. A mutation study involving 768 faulty implementations shows that E2E tests detect 90% of injected bugs, including 74% of distributed-property bugs missed by unit and integration tests. The results highlight the limitations of shallow verification for coding agents operating in stateful game engines.

Critical strengths

S1. Realistic and challenging benchmark. The tasks require agents to modify existing UE5 projects while respecting engine lifecycle, subsystem dependencies, and networking constraints, extending coding-agent evaluation beyond isolated programming tasks.

S2. Empirical evidence for deeper verification. The mutation analysis and comparisons across verification tiers demonstrate that shallow tests can miss behavioral failures and substantially affect measured agent performance.

Critical weaknesses

W1. Limited differentiation from prior work. Prior studies such as PlayCoder (FSE 2026) and GameLogicBench have explored runtime behavioral verification for game-related coding tasks. The paper does not adequately distinguish its contributions from these approaches. Its most distinctive feature, cross-client replication verification, directly applies to only 17 of 110 tasks, and its additional impact on evaluating real agent-generated solutions remains unclear.

W2. Insufficient validation of verifier reliability. The mutation study relies on intentionally injected defects, which may not represent realistic agent failures. Moreover, the human-reviewed evaluation uses E2E verdicts as reference labels without clearly establishing their independent correctness. Quantitative evidence of verdict stability across repeated executions is also missing.

Questions for authors

Q1. For the 17 cross-client tasks, how many real agent-generated solutions pass single-world tests but fail cross-client verification? How does this affect model rankings?

Q2. How does Full-Session Verification differ technically from prior runtime behavioral evaluation approaches such as PlayCoder and GameLogicBench?

Q3. How were the E2E reference verdicts independently validated, and how stable are the results across repeated runs?

7 comments
AuthorsOpenReview ↗

Response to Reviewer QKUA

We thank the reviewer for the constructive feedback. We respond to the concerns and questions brought up below.

W1 / Q2 (differentiation from PlayCoder and GameLogicBench). We now cite and compare PlayCoder. Its tasks regenerate a single function (at least 28 lines) from a given signature and an LLM-written docstring, judged by unit tests and then an LLM playtester; our tasks require implementing multi-file features in existing UE5 C++ projects (median 389 lines) from a behavioral specification, judged only by compiled tests. GameLogicBench is a concurrent preprint (arXiv, September 2026) that asserts on the state of a single Godot process and does not test network synchronization. Both evaluate behavior within a single game process. Our tests run inside a Play-in-Editor session and assert directly on engine-owned state in the server world and in each client world, so a client-side check passes only if the server produces the state and Unreal's replication delivers it.

On "17 of 110": the submitted figure counted the tasks whose suites asserted on a client world when the paper was written; the submitted scores predated the expansion of these tests. Before the final evaluated run, 36 further tasks gained client-world tests, so the revised suites contain 119 client-world tests in 53 tasks, and 71 tasks start at least one client. The suites were frozen before that run, and all 11 configurations re-solved all tasks and were graded with the same frozen suites (Section 4.3). Their impact on real submissions is reported under Q1.

W2 (verifier reliability). Beyond mutants, we now measure the deeper tests on real submissions: 209 of the 493 submissions that pass every static, unit and integration test fail an end-to-end test (69 of 269 on tasks that also have integration tests), and removing end-to-end tests would raise pass@1 by 12.7-20.9 points per configuration (Section 5.4). To validate the verdicts, two experts reviewed 195 graded submissions (39 tasks sampled at random with a fixed seed, stratified by project and client-world type, with five fixed configurations per task) against the task specification, behavioral contract, and code changes, each without consulting the other; 81 of these were rejected by an end-to-end test. On the 30 submissions reviewed by both experts, they initially agreed on 27 (90%) and resolved the three disagreements, which arose from ambiguous specifications, through discussion. No end-to-end verdict was overturned (Section 4.3). Stability is reported under Q3.

Q1 (client-world checks on real submissions). On the 17 cross-client tasks named in the submission, 24 of 63 would-be passes (38.1%) fail only on client-world tests. On the 53 tasks with client-world tests, 64 of 177 would-be passes (36.2%) fail only on client-world tests (64 of 348 across all 110 tasks), and pass@1 averaged over configurations is 19.4% versus 27.3% on the other 57 tasks. Removing client-world tests raises pass@1 by 3.6-7.3 points and changes the ranking of three configurations; the leader is unchanged.

Q3 (validation and stability). Verdicts come only from compiled tests, and we validate those tests directly: every reference solution passes its frozen suite in both grading runs (220/220, after two first attempts of one load-sensitive test failed and passed on rerun; Appendix B.5, Table 5), in 819 of the 821 submissions that build and fail a test the failing test maps to a named requirement, the suites kill 550 of 567 seeded faults (97.0%, an in-sample upper bound), and expert review overturned no verdict (see W2). Regrading all 1,210 submissions changed no verdict, and 100 repeat grades changed none.

Thank you for the detailed response! The new results on real agent submissions and verdict stability substantially address my previous concerns.

I have two remaining questions:

  1. The number of tasks with client-world assertions increased from 17 to 53. Could you clarify whether this reflects a correction in reporting or newly added tests? If tests were added, were they designed independently of the evaluated agent submissions, and were all configurations re-evaluated using the same frozen suites?

  2. While repeated grading demonstrates stability, it does not necessarily establish oracle correctness. Have you considered manually validating a representative sample of real agent submissions rejected by the E2E tests?

Overall, the additional evidence significantly strengthens the paper, particularly the demonstrated impact of cross-client verification on real agent evaluations.

Thank you for the follow-up and for the positive assessment of the revision. We address both questions below.

Q1: Mostly newly added tests. The submitted figure of 17 counted the tasks whose suites asserted on a client world when the paper was written, but the submitted scores predated the expansion to 53 tasks. Before the final evaluated run, 36 further tasks gained client-world tests, giving 119 tests in 53 tasks, each labelled with two independent methods (Cohen's κ = 0.934; the 34 disagreements were resolved by manual inspection) (Section 4). The tests were written from the task specifications and, before the suites were frozen, revised after inspecting failures from preliminary runs of the evaluated configurations, so they are not independent of agent behavior in general. They are independent of every graded submission: after the freeze, all 11 configurations re-solved all 110 tasks, and every submission was graded with the same frozen suites (Section 4.3). On the original 17 tasks alone, 24 of 63 would-be passes (38.1%) fail only on client-world tests, compared with 64 of 177 (36.2%) across all 53.

Q2: Yes, out of the 195 graded submissions reviewed by experts, 81 were rejected by an E2E test. These experts manually examined task specifications, behavioral contracts, and code changes, and no verdict was overturned (Section 4.3). We agree that stability does not establish correctness. Each reference solution, derived from the original open-source implementation, passes its frozen suite in both grading runs (220/220), and each start state fails on the removed behaviors. Two engineers with industry Unreal Engine experience reviewed every task and its tests, and tests were revised based on their feedback (Section 3.2). For rejections, a failing test maps to a named requirement in 819 of the 821 submissions that build and fail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank the reviewers for their careful reading and constructive feedback. Several comments stem from a common issue: parts of the original submission reflected an earlier evaluation pipeline that used an LLM-judge-calibrated pass label. In the revision, we replace these results, and all analyses derived from them, with verdicts determined exclusively by compiled hidden tests.

Revised Results

  • Model performance: We evaluate 11 model–harness configurations. The highest-scoring configuration, Claude Fable 5 (max), achieves 46.4% pass@1 (95% CI: 37.3–55.5%). After correction for multiple comparisons, the top four configurations are statistically tied.
  • Task difficulty: Of the 110 tasks, 43 remain unsolved by all evaluated configurations.

New Analyses and Improvements

Regrading all 1,210 submissions produced zero verdict changes (0/1,210), and an additional 100 repeat grades also yielded zero changes, providing evidence of grading reproducibility. We conduct a revised mutation analysis using the frozen test suites. The previously reported figures of 768 mutants and a 90% mutation score could not be reproduced on the frozen suite and have therefore been withdrawn. We also expand client-world test coverage from 17 to 53 tasks, providing a broader assessment of end-to-end behavior. Additionally, we add examples from actual model submissions demonstrating failures that are detected only by end-to-end tests, highlighting the importance of evaluating observable behavior beyond isolated unit tests. Finally, we expand the discussion of related work, including PlayCoder, to better contextualize our contributions relative to existing approaches.

Together, these revisions strengthen the reliability, transparency, and scope of our evaluation. They also clarify which conclusions are supported by the corrected results, while preserving the central motivation for evaluating agents against executable, end-to-end behavioral tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.