- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces GRACEBench (60 extension and repair tasks in existing Godot 2D projects, with 1,270 executable checks organized into gate, behavior, and integration layers) to valuates whether coding agents can meet real-world game-development requirements in which several systems must work together. The evaluation is conducted across 12 models, 180 answers validly pass every gate and behavior check.
Critical strengths
- The benchmark is designed for real-world, complete game-development requirements in practical settings that comprises 60 extension and repair tasks in existing Godot 2D projects across ten gameplay genres, with 1,270 executable checks at three layers.
- The evaluation is comprehensive across 12 models and this work also conducted in-depth cross-system failures that can bring more insights: gate and behavior checks alone would count 180 answers as complete, of which 65 pass every integration check; the other 115 span 34 tasks.
Critical weaknesses
- As this is a benchmark paper, it should provide more details to support reproducibility, beyond describing the 60 tasks and how they were constructed. The link in the “Reproducibility Statement” is inaccessible, and the paper should make the reported results and the corresponding scripts and code available. This is my main concern.
- It is also unclear why coordination among multiple game systems is an important scenario. Is this a key challenge in real-world game development?
Questions for authors
- Is game systems must work together is an important scenarion?
- Why the fraction of tasks where every check validly passes are so low?
3 comments
Q1. Importance of coordination among game systems
We sincerely thank the reviewer for asking us to clarify the connection to practical game development. A player-visible feature often depends on several systems working together. In Task 48, for example, collecting a fuel can involves collision handling, resource accounting, spawn quotas, and vehicle position. A collected object must continue occupying a quota slot until it leaves the scene. When that slot becomes available, a deferred spawn must use the vehicle’s current position. Implementing the individual mechanisms is therefore only part of the work. Their states and transitions must also agree.
Failures at these boundaries have direct gameplay consequences, such as items that never respawn, cooldowns that change with frame rate, or state that incorrectly persists after death or a level transition. Adding or repairing features in an existing game thus requires coordinating mechanisms with different responsibilities and lifecycles. This motivates our choice of base projects with at least three interacting runtime systems.
Our results show a substantial gap in this setting. Of the 180 answers that pass every gate and behavior check, 115 still fail at least one integration check. These answers satisfy the individually tested mechanisms but miss a required interaction or consistency condition.
Q2. Reasons for the low task-resolution rate
We sincerely thank the reviewer for raising this question about the low task-resolution rate.
A task counts as resolved only when every check validly passes. Each task has about 21 checks on average, covering both ordinary behavior and demanding conditions such as boundaries, event ordering, and consistency at 60 Hz and 120 Hz. Across the 720 evaluation runs, 65 fully resolve the corresponding task.
Two factors help explain this rate. First, full resolution requires the interactions between mechanisms to work correctly alongside the mechanisms themselves. The 180 answers passing all gate and behavior checks fall to 65 once integration checks are included. Second, the benchmark was deliberately screened for difficulty. Task versions that screening models solved too frequently were redesigned, leaving substantial room for improvement by future agents.
Many unresolved answers nevertheless come close to completion. Among submitted answers, 75 fail only one check and another 66 fail only two. We therefore use the dimension-weighted score as our primary metric, which credits a diagnostic dimension when all its associated checks pass. We report the resolved rate alongside it to show how often an agent satisfies the complete set of task requirements.
Concern: Accessibility and completeness of reproducibility materials
We sincerely thank the reviewer for highlighting the accessibility issue and emphasizing the need for reproducibility materials beyond the task descriptions. We agree that a benchmark should make its construction process and reported results available for inspection and reproduction.
We have now open-sourced the task synthesis pipeline and released the recorded model results in the reproducibility package, available at https://figshare.com/s/4202f0d2bae53844122f. These materials accompany the released tasks and make both the construction process and the empirical results available for inspection and reuse. We have updated the Reproducibility Statement to describe these materials and provide the package link.