- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
GameEngineBench evaluates coding agents on 110 C++ feature-implementation tasks derived from nine Unreal Engine 5 repositories. Its hidden verification suites progress from static and unit checks to synthetic-world integration tests and full networked Play-in-Editor sessions. Separate server and client worlds enable checks of lifecycle ordering and distributed state consistency; 17 tasks directly inspect client-world behavior. The paper evaluates 12 model-harness configurations, with the strongest achieving 55.5% pass@1, and analyzes 768 known-wrong mutants across 108 tasks. Full-session tests detect substantially more injected defects than shallower tiers, particularly distributed failures, illustrating a useful gap between plausible code and correct engine behavior.
Critical strengths
- The benchmark targets a consequential capability that compilation and isolated function tests cannot establish: correct behavior over a running engine lifecycle and across networked worlds (Sections 3-4). Hidden post-solve tests, reference-pass/starter-fail validation, and review by two Unreal Engine experts are meaningful safeguards. The resulting tasks offer a concrete evaluation setting for long-horizon, repository-level implementation.
- The tier-by-tier mutation analysis provides evidence for the verification design, rather than only reporting a leaderboard (Section 5.5; Table 3). E2E tests catch 90% of all mutants versus 68% for integration tests; on distributed mutants, the corresponding rates are 92% and 15%. This is a valuable empirical demonstration of when deeper execution changes the observability of defects.
Critical weaknesses
- The reliability of the final E2E verifier needs further quantification (Section 5.5; Appendix B). The 195-run comparison uses human-reviewed E2E verdicts as its reference, but does not explain whether reviewers independently established correctness or how disagreements were resolved. Its error-rate interpretation therefore depends on an insufficiently described adjudication process. Approximately 10% of known-wrong mutants also escape E2E detection, without a detailed account of those blind spots. Appendix B.5 describes sensible stability controls, but does not report repeated-run verdict consistency. These gaps matter because verification errors or unstable outcomes could affect benchmark rankings.
- Result provenance is not consistently documented. Section 4.3 describes an execution-based pass rule, whereas Figure 6(e) refers to missing judge labels being treated as non-passes and is captioned as the task-level matrix behind the aggregate ranking. Appendix A also presents Task 19 as unsolved, while Appendix D says it is solved by Fable 5. These could be stale annotations rather than errors in the official results, but the manuscript needs a reconciled task-by-configuration ledger before readers can confidently interpret its pass rates and complementarity analysis.
Questions for authors
- What exactly are the judge labels in Figure 6(e), and does that panel use the same task set, configurations, and pass predicate as the main leaderboard? Please reconcile the Task 19 descriptions and identify any affected aggregate results.
- How were the 195 runs manually reviewed, and were disagreements with E2E verdicts independently adjudicated? Please characterize undetected known-wrong mutants and report repeated-run verdict consistency for fixed reference and agent solutions.
Response to Reviewer aPGd
We thank the reviewer for the careful review. We respond to the concerns and questions brought up below.
W1 / Q2 (verifier reliability). Two experts reviewed 195 graded submissions (39 tasks sampled at random with a fixed seed, stratified by project and client-world type, with five fixed configurations per task) against the task specification, behavioral contract, and code changes, each recording an assessment without consulting the other; 81 of these were rejected by an end-to-end test. On the 30 submissions reviewed by both experts, they initially agreed on 27 (90%), and the three disagreements, which arose from ambiguous specifications, were resolved through discussion. No end-to-end verdict was overturned (Section 4.3). The submitted per-tier error rates on these runs, which used the end-to-end verdict as their reference, are withdrawn. We also quantify reliability on the suite that defines pass@1: every reference solution passes in both grading runs (220/220, after two first attempts of one load-sensitive test failed and passed on rerun), in 819 of the 821 submissions that build and fail a test the failing test maps to a named requirement, and regrading changed no verdict (0/1,210 submissions; 0/100 repeats; Appendix B.5, Table 5). The submitted mutation table (768 mutants, E2E 90%) graded separate per-tier suites and could not be reproduced. In its place, on real submissions, 209 of the 493 that pass every static, unit, and integration test fail an end-to-end test (69 of 269 on tasks that also have integration tests), and removing end-to-end tests would raise pass@1 by 12.7–20.9 points per configuration (Section 5.4); on the frozen suite, distributed mutants are killed by 0% of unit tests, 11.9% of integration tests, and 85.1% of end-to-end tests, although integration tests kill more local-logic mutants than end-to-end tests (50.7% vs 42.0%; Table 3). Against the single frozen suite, 17 of 567 mutants (3.0%) survive: 13 weak assertions (the code runs but no assertion distinguishes the fault), 3 untested paths, and 1 fixture limit (the distinguishing condition cannot be produced deterministically). Four further surviving mutants were judged after grading to be likely equivalent or not constrained by the instruction, and were excluded. Appendix F lists all surviving mutants.
W2 / Q1 (result provenance, Task 19). The "judge labels" in Fig. 6(e) were a remnant of an LLM-based adjudication component in the submitted version, which is withdrawn; one ledger of 1,210 verdicts (110 tasks × 11 configurations, compiled tests only) now feeds every figure, including Fig. 6(e), which uses the same tasks, configurations and pass rule as the leaderboard. No configuration solves Task 19, so Appendix A was right, and the complementarity examples (submitted Appendix D, now Appendix E) are rewritten. The aggregates that changed are best pass@1 55.5% → 46.4%, unsolved tasks 31 → 43, and configurations 12 → 11 (Kimi was not part of the evaluated run).
Presentation. We merged the scoring definitions into one section (Section 4.3), removed duplicated text, and explain technical terms at first use.