acceptodds
Under review as a conference paper at ICLR 2027

GRACE-Bench: Can Coding Agents Make Game Systems Work Together?

Abstract

Coding agents are increasingly used to implement features and repair defects in existing software projects. Game projects make this work especially demanding: a single requirement often spans player input, physics, gameplay rules, object lifecycles, and visual feedback, so meeting it requires these systems to work together through shared runtime state and event sequences. To test whether coding agents can meet such complete game-development requirements, we introduce GRACE-Bench: 60 extension and repair tasks in existing Godot 2D projects, with 1,270 executable checks organized into gate, behavior, and integration layers. The layers cover execution prerequisites, designated mechanisms, and required cross-system interactions of the same task, so each answer is judged against its complete requirement and each failure is located in a layer. Across 12 models, 180 answers validly pass every gate and behavior check, so an evaluation without the integration layer would count all of them as complete; only 68 also pass every integration check. The other 112 (62.2%; task-resampling 95% interval 50.8–72.6%) fail checks of cross-system behaviors required by their task specifications. These failures span 33 tasks and all 12 models, and the proportion stays above one half when any single model or task is left out or when the ten most frequently failed integration checks are removed. These results expose a gap in current coding agents: they often implement the requested mechanisms without making them work together, a gap that checks of individual mechanisms alone would not reveal. Recorded runtime evidence and targeted replay trace these failures to unmet requirements and their gameplay consequences, showing the diagnostic value of the layered checks. Because every answer is graded on all checks of its task, GRACE-Bench can be rerun to test whether future agents genuinely improve at meeting complete development requirements.

Primary area:
datasets and benchmarks
Keywords:
coding agents, game development, diagnostic benchmarks, behavioral correctness, cross-system reliability

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_HjttOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GRACEBench (60 extension and repair tasks in existing Godot 2D projects, with 1,270 executable checks organized into gate, behavior, and integration layers) to valuates whether coding agents can meet real-world game-development requirements in which several systems must work together. The evaluation is conducted across 12 models, 180 answers validly pass every gate and behavior check.

Critical strengths
  • The benchmark is designed for real-world, complete game-development requirements in practical settings that comprises 60 extension and repair tasks in existing Godot 2D projects across ten gameplay genres, with 1,270 executable checks at three layers.
  • The evaluation is comprehensive across 12 models and this work also conducted in-depth cross-system failures that can bring more insights: gate and behavior checks alone would count 180 answers as complete, of which 65 pass every integration check; the other 115 span 34 tasks.
Critical weaknesses
  • As this is a benchmark paper, it should provide more details to support reproducibility, beyond describing the 60 tasks and how they were constructed. The link in the “Reproducibility Statement” is inaccessible, and the paper should make the reported results and the corresponding scripts and code available. This is my main concern.
  • It is also unclear why coordination among multiple game systems is an important scenario. Is this a key challenge in real-world game development?
Questions for authors
  • Is game systems must work together is an important scenarion?
  • Why the fraction of tasks where every check validly passes are so low?
3 comments
AuthorsOpenReview ↗

Concern: Accessibility and completeness of reproducibility materials

We sincerely thank the reviewer for highlighting the accessibility issue and emphasizing the need for reproducibility materials beyond the task descriptions. We agree that a benchmark should make its construction process and reported results available for inspection and reproduction.

We have now open-sourced the task synthesis pipeline and released the recorded model results in the reproducibility package, available at https://figshare.com/s/4202f0d2bae53844122f. These materials accompany the released tasks and make both the construction process and the empirical results available for inspection and reuse. We have updated the Reproducibility Statement to describe these materials and provide the package link.

AuthorsOpenReview ↗

Q1. Importance of coordination among game systems

We sincerely thank the reviewer for asking us to clarify the connection to practical game development. A player-visible feature often depends on several systems working together. In Task 48, for example, collecting a fuel can involves collision handling, resource accounting, spawn quotas, and vehicle position. A collected object must continue occupying a quota slot until it leaves the scene. When that slot becomes available, a deferred spawn must use the vehicle’s current position. Implementing the individual mechanisms is therefore only part of the work. Their states and transitions must also agree.

Failures at these boundaries have direct gameplay consequences, such as items that never respawn, cooldowns that change with frame rate, or state that incorrectly persists after death or a level transition. Adding or repairing features in an existing game thus requires coordinating mechanisms with different responsibilities and lifecycles. This motivates our choice of base projects with at least three interacting runtime systems.

Our results show a substantial gap in this setting. Of the 180 answers that pass every gate and behavior check, 115 still fail at least one integration check. These answers satisfy the individually tested mechanisms but miss a required interaction or consistency condition.

AuthorsOpenReview ↗

Q2. Reasons for the low task-resolution rate

We sincerely thank the reviewer for raising this question about the low task-resolution rate.

A task counts as resolved only when every check validly passes. Each task has about 21 checks on average, covering both ordinary behavior and demanding conditions such as boundaries, event ordering, and consistency at 60 Hz and 120 Hz. Across the 720 evaluation runs, 65 fully resolve the corresponding task.

Two factors help explain this rate. First, full resolution requires the interactions between mechanisms to work correctly alongside the mechanisms themselves. The 180 answers passing all gate and behavior checks fall to 65 once integration checks are included. Second, the benchmark was deliberately screened for difficulty. Task versions that screening models solved too frequently were redesigned, leaving substantial room for improvement by future agents.

Many unresolved answers nevertheless come close to completion. Among submitted answers, 75 fail only one check and another 66 fail only two. We therefore use the dimension-weighted score as our primary metric, which credits a diagnostic dimension when all its associated checks pass. We report the resolved rate alongside it to show how often an agent satisfies the complete set of task requirements.

Reviewer_6MA2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRACE-Bench evaluates coding agents on 60 extension and repair tasks in existing Godot 2D projects. Its 1,270 executable checks cover execution prerequisites, designated behaviors, and required interactions between mechanisms. Across 12 models, 115 of the 180 submissions passing all gate and behavior checks still fail integration checks. Reference implementations, deliberately faulty implementations, and selected runtime case studies support the evaluation.

Critical strengths
  • A useful benchmark with actionable diagnostics. Checking different requirements within the same implementation makes it possible to identify what remains broken after the designated mechanisms pass.

  • The main empirical finding is well supported. The conditional integration failures span 34 tasks and all 12 models.

Critical weaknesses
  • The novelty is useful but fairly incremental. Previous benchmarks already test runtime behavior and include integration checks. The results establish the diagnostic value of the additional checks, but less clearly isolate cross-system difficulty from the difficulty of those checks: behavior checks can already involve multiple systems, and the layers are not matched controls.

  • Grader validation is stronger against false positives than false negatives. Rejecting all 275 faulty implementations is reassuring evidence against accepting incorrect code. Evidence against rejecting correct code is thinner. This matters because overly restrictive checks could inflate the reported integration failure rate.

Questions for authors
  1. What operational rule distinguishes a behavior check involving several systems from an integration check? How was consistency of this distinction checked across tasks?

  2. How many tasks have independently produced correct alternative implementations? Beyond the construction references, was there an independent review of model submissions to assess false negatives, particularly among the 115 conditional integration failures?

4 comments
AuthorsOpenReview ↗

Q1. Operational rule and consistency of layer assignments

We sincerely thank the reviewer for asking us to clarify the operational rule and its application across tasks.

We classify a check by the requirement exercised by its assertion. A mechanism is a task-defined function, such as quota management, vehicle movement, or spawn placement.

  • Behavior: the assertion tests whether a designated mechanism produces the specified output or state transition under given inputs and conditions, including its boundary, timing, and lifecycle rules.
  • Integration: the assertion tests a required dependency between mechanisms during their joint operation. For example, one mechanism must use another’s current state, respond correctly to its events, or coordinate a transition in the required order.

The number of participating objects or scripts does not determine the layer. For example, Task 48’s quota-retention check involves both the collected object and the spawner, but its assertion tests the quota-management rule: the object occupies its slot until it leaves the scene. The deferred-placement check instead requires the spawner to use the car’s current position when a slot becomes available, testing the dependency between quota availability, vehicle movement, and spawn placement.

Across tasks, layer assignments follow the same construction guide. The grader author assigns the layer when writing the rubric, and the S7 audit reviews every rubric entry, including its layer assignment, against the corresponding task statement.

We also examined the sensitivity of the main finding to the treatment of frame-rate checks, which can assess either a designated mechanism or interactions between mechanisms. Assigning all frame-rate checks to behavior gives a conditional integration-failure proportion of 52.6%. Assigning all to integration gives 65.4%, compared with 63.9% originally. Each proportion uses the answers passing all gate and behavior checks under the corresponding assignment as its denominator. Under both alternatives, more than half of these answers still fail an integration check.

AuthorsOpenReview ↗

Q2. Alternative implementations and independent review of potential false rejections

We sincerely thank the reviewer for emphasizing independent acceptance evidence and the examination of potential false rejections.

We constructed alternative implementations for 15 tasks. Each started from an evaluated answer that failed exactly one check. An isolated development session repaired the answer using the public task statement, without access to the reference implementation or probe source code. All 15 alternative implementations passed every check. The calibration procedure requires each alternative reference to pass every check. We also compared each alternative with its corresponding primary reference. After removing comments and whitespace, normalizing identifiers and literals, and excluding shared starter-code fragments, the median 5-token and 10-token Jaccard similarities are 21.7% and 7.7%, respectively. These comparisons document structural differences between the alternative and primary implementations.

Beyond the construction references, eleven of the twelve evaluated models independently produced 65 answers that pass every check, spanning 27 tasks. We also examined the recorded outcomes for the 115 conditional integration failures. For 91 of these 115 answers, each failed integration check is passed by at least one other model’s answer to the same task. This provides direct evidence that the corresponding checks accept implementations beyond the designated references.

As a conservative sensitivity analysis, we excluded all 46 checks that no evaluated answer passed. Among the 195 answers that then pass every remaining gate and behavior check, 116 still fail at least one remaining integration check: 59.5%, compared with 63.9% originally. The main finding therefore remains consistent under this exclusion.

During the rebuttal period, two authors manually cross-checked a sample of 30 answers drawn from the 115 conditional integration failures. No false rejections were found in this sample. We will conduct a more comprehensive and systematic manual review of failed answers, with particular attention to these conditional integration failures, by comparing their observed behavior with the public task requirements and the corresponding checks.

Thank you for addressing my concerns. I will keep my positive scores.

Thank you so much for your encouraging response! We’re very glad we could address your concerns, and we really appreciate the time and care you’ve put into reviewing our work.

Reviewer_pcctOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRACE-Bench evaluates coding agents on 60 extension and repair tasks in existing Godot 2D projects. Its 1,270 executable checks are organized into gate, behavior, and integration layers, supported by public interface contracts, private runtime probes, and seven diagnostic dimensions. The benchmark is calibrated using reference implementations, deliberately faulty implementations, and witnesses for critical scenarios.

The main empirical finding is that, across 12 models and 720 task–model outcomes, 180 submissions pass every gate and behavior check, but 115 of these still fail at least one integration check: a conditional failure rate of 63.9%, with a task-resampling 95% interval of 52.7–74.1%. The authors provide sensitivity analyses and two detailed failure cases, including a targeted replay that identifies an incorrect runtime-state lookup. The principal contributions are the benchmark, its calibration and diagnostic records, and an empirical analysis of requirements missed when integration checks are omitted.

Critical strengths
  1. The evaluation is carefully designed. The checks are tied to explicit task requirements and validated using reference and deliberately faulty implementations. The specification audits and human review provide further support for the grading decisions.

  2. The experiments reveal useful failures that behavior checks miss. The paper shows that agents can pass the designated behavior checks while still violating interaction requirements in the same task. The concrete examples explain how these failures affect game behavior, and the sensitivity analyses support the broader pattern. This makes the results useful for understanding where apparently successful implementations remain incomplete.

Critical weaknesses

The added value of the behavior/integration split is unclear.

Runtime and integration testing already exist in related work, so the value of the proposed split depends partly on the additional insight it provides. Table 1’s behavior example itself couples timing, object lifetime, and quota state, leaving it unclear how mechanism boundaries are defined consistently across tasks. The reported failures are credible, but the additional insight provided by this split beyond identifying gaps in test coverage needs clearer justification.

Questions for authors
  1. How are behavior and integration checks distinguished? Please provide a few borderline examples and explain how the assignments were reviewed. If some checks could reasonably belong to either layer, would reassigning them materially change the main result?

  2. What explains the gap between check-level and task-level success? Astra passes approximately 87% of behavior checks and 80% of integration checks, but fully resolves only 30% of tasks (Figure 2, Table 3). Among the unresolved tasks, how many fail only one or a few checks? Do these failures concentrate on a small number of repeated error patterns or involve diverse requirements? This would help assess the benchmark’s remaining difficulty and diagnostic value.

  3. How well do the graders accept different correct implementations? How many tasks were tested with alternative correct implementations beyond the designated reference? Were any apparently contract-compliant implementations rejected, and how were those cases resolved?

8 comments
AuthorsOpenReview ↗

Q1. Distinguishing behavior and integration checks

We sincerely thank the reviewer for this thoughtful and constructive question. We agree that the layer definitions, their review, and sensitivity to borderline assignments are important for interpreting the main finding.

Definitions and review. We distinguish the layers by the requirement that determines whether a check passes. A mechanism is a function specified by the task, such as quota management, vehicle movement, or spawn placement.

Behavior checks assess whether a designated mechanism follows its specified rules under given inputs and conditions: whether it produces the required output, updates its state correctly, and handles its prescribed boundary cases. These checks can include timing and lifecycle requirements within that mechanism.

Integration checks assess whether a required relationship between mechanisms holds when they operate together. For example, one mechanism must use another’s current state, an event produced by one mechanism must trigger the appropriate response in another, or transitions across mechanisms must occur in the required order.

The grader author assigns the layers when writing the rubric, and the S7 audit reviews every entry, including its layer assignment, against the task statement.

Borderline examples. In Table 1, Task 48’s quota-retention check is assigned to behavior because it tests the quota-management rule: a collected object continues to occupy its slot until it leaves the scene. When a slot becomes available, placing the deferred spawn using the car’s current position tests the dependency between quota availability, vehicle movement, and spawn placement and is classified as integration. Thus, involving multiple objects or testing a timing boundary does not by itself determine the layer. The deciding factor is the functional rule or cross-mechanism dependency being checked.

Frame-rate checks provide another useful case for examining this boundary. Testing at 60 Hz and 120 Hz can assess either a designated mechanism’s frame-rate independence or the consistency of interactions between mechanisms. The test frequency alone does not determine the layer. We therefore examined whether assigning these checks uniformly to either layer would change the main finding.

Sensitivity to reassignment. We recomputed the conditional failure proportion under two alternative assignments. Each proportion uses the answers passing all gate and behavior checks under the corresponding assignment as its denominator. The original result is 115 of 180 answers, or 63.9%.

  • Moving all 92 frame-rate integration checks to behavior: 52.6%.
  • Moving all 60 frame-rate behavior checks to integration: 65.4%.

Under both alternative assignments, more than half of the answers that pass every gate and behavior check still fail an integration check. The main finding therefore remains consistent under these reassignments.

AuthorsOpenReview ↗

Q2. Gap between check-level and task-level success

We sincerely thank the reviewer for highlighting this distinction and asking how it relates to the benchmark’s remaining difficulty.

A task counts as resolved only when every check passes, whereas the check-level rate also credits requirements satisfied by partially complete answers. High check-pass rates can therefore coexist with a much lower task-resolution rate. Of GPT-6-Astra’s 42 unresolved tasks, 17 fail only one or two checks: these answers satisfy most requirements but remain unresolved under the all-check criterion.

The remaining failures show both breadth and recurrence. They span all seven diagnostic dimensions, particularly timing, lifecycle, and state. Meanwhile, more than half of GPT-6-Astra’s failed checks are also failed by at least nine of the twelve models on the same task. Thus, the results identify both near-complete answers and requirements that remain difficult across models. The check records locate these remaining gaps, while the paper’s case analyses explain how selected failures arise and affect gameplay.

AuthorsOpenReview ↗

Q3. Acceptance of alternative implementations and review of potential false rejections

We sincerely thank the reviewer for this thoughtful and constructive question. We fully agree that acceptance evidence beyond the designated references is important for assessing whether the graders accommodate different correct implementations.

We constructed alternative implementations for 15 tasks. Each started from an evaluated answer that failed exactly one check. An isolated development session repaired the answer using the public task statement, without access to the reference implementation or probe source code. All 15 alternative implementations passed every check. The calibration procedure requires each alternative reference to pass every check. We also compared each alternative with its corresponding primary reference. After removing comments and whitespace, normalizing identifiers and literals, and excluding shared starter-code fragments, the median 5-token and 10-token Jaccard similarities across the fifteen pairs are 21.7% and 7.7%, respectively. These comparisons document structural differences between the alternative and primary implementations.

The evaluated answers provide further acceptance evidence. Eleven of the twelve models independently produced 65 answers that pass every check, spanning 27 tasks. Furthermore, among the 115 answers that pass every gate and behavior check but fail an integration check, 91 have the following property: for each failed integration check, at least one other model’s answer to the same task passes that check. Together, these results provide direct evidence that the corresponding checks accept implementations beyond the designated references.

Of the 1,270 checks, 46 (3.6%) were not passed by any evaluated answer. Among these, 35 concern boundary conditions, event ordering, lifecycles, repeated operations, or frame-rate consistency. The benchmark was deliberately designed to assess these scenarios. As a conservative sensitivity analysis, we treated all 46 checks as invalid and recomputed the result after excluding them. Among the 195 answers that then pass all remaining gate and behavior checks, 116 still fail at least one remaining integration check: 59.5%, compared with 63.9% originally. The main finding therefore remains consistent even under this exclusion.

During the rebuttal period, two authors manually cross-checked a sample of 30 answers drawn from the 115 conditional integration failures. No false rejections were found in this sample. The acceptance evidence and sensitivity analysis above support the graders’ ability to accommodate different implementations and the stability of the main finding. We will conduct a more comprehensive and systematic manual review of failed answers, comparing their observed behavior with the public task requirements and the corresponding checks.

Reviewer_2GRcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

GRACE-Bench evaluates coding agents on 60 extension and repair tasks in existing Godot 2D projects. Its 1,270 executable checks cover runtime prerequisites, designated behaviors, and required interactions among game systems. Public contracts define how implementations are tested, while private probes record deterministic outcomes. Across 12 models, 180 answers pass all gate and behavior checks. Only 65 also pass every integration check. Reference solutions, faulty implementations, sensitivity analyses, and selected runtime cases support the benchmark's diagnostic value. The main finding is that passing individual behavior checks often leaves task-required interactions incomplete.

Critical strengths
  1. The benchmark evaluates complete requirements within the same implementation. Public contracts connect specifications to runtime observations. This makes failures in state sharing, event ordering, and resource accounting concrete and inspectable (Sections 3.1–3.2).

  2. The main result extends across all 12 models and 34 tasks. Of 180 answers passing earlier layers, 115 fail integration checks. Removing the ten most frequently failed checks still leaves 97 failures, supporting the finding's breadth (Section 4.3).

  3. Calibration covers 60 accepted references, 275 rejected faulty records, and targeted witnesses for all 432 critical scenarios. Unavailable checks are handled explicitly. The case studies also distinguish observed discrepancies from established causes (Sections 3.4–5).

Critical weaknesses
  1. Calibration includes strong fault-detection tests and a procedure for accepting alternative references. However, the paper does not summarize how often alternative references were available or whether acceptance disputes arose. Such records would strengthen the evidence that the grader accommodates varied correct implementations.

  2. The difficulty-screening procedure is not fully specified. Repeated trials and a preset acceptance limit are described, but their values and the task-redesign history are missing. These details would help reproduce benchmark construction and explain how screening shaped the final task set.

Questions for authors
  1. The headline failures span 34 tasks, but tasks can share game bases. Appendix E reweights the overall layer gap by template path. Could you summarize how the 115 conditional failures are distributed across those groups? This would clarify the breadth of the headline finding beyond individual tasks.

  2. The calibration procedure requires alternative references to pass every check. How many tasks had such references, and how were they obtained? Please summarize any existing review of apparently correct agent answers that failed, if available. This would clarify acceptance evidence beyond the primary reference solutions and deliberately faulty implementations.

  3. The construction pipeline specifies one S6 trial and repeated S8 screening with Claude Opus 4.6. What threshold and repeat count were used in S8? Please summarize existing records of task redesign or exclusion. These details would make the selection procedure reproducible and clarify the resulting challenge set.

3 comments
AuthorsOpenReview ↗

Q1. Distribution of conditional failures across template paths

We sincerely thank the reviewer for this insightful and constructive suggestion. We fully agree that the distribution across shared game bases should have been made clearer in the original manuscript.

The 180 answers that pass every gate and behavior check come from 42 tasks on 27 of the 34 template paths. The 115 answers among them that fail an integration check come from 34 tasks on 25 paths. No single path accounts for more than 13 of the 115 failures (11.3%), and the five paths with the most failures account for 51 (44.3%). Leaving out one path at a time, together with every task built on it, gives a conditional failure proportion of 61.8% to 66.9%, compared with 63.9% overall. Averaging the within-path conditional failure proportions equally across the 27 paths with eligible answers gives 66.4%.

Resampling all 34 template paths instead of tasks, while retaining all tasks and model answers within each sampled path, gives a 95% percentile interval of 52.4% to 74.7% (10,000 resamples), close to the task-resampling interval of 52.7% to 74.1%. The conditional failures span many template-path groups, and the failure proportion remains similar under path-level reweighting and leave-one-path-out analysis. We have added the per-path counts and the resampling details in Appendix E.2.

AuthorsOpenReview ↗

Q2. Acceptance of alternative implementations and review of potential false rejections

We sincerely thank the reviewer for this insightful and constructive suggestion. We agree that acceptance evidence beyond the primary references is important for assessing whether the probes accommodate different implementations.

We constructed alternative implementations for 15 tasks. Each started from an evaluated answer that failed exactly one check. An isolated development session repaired the answer using the public task statement, without access to the reference implementation or probe source code. All 15 alternative implementations passed every check. After normalizing identifiers and literals, removing comments and whitespace, and excluding shared starter-code fragments, the median 5-token and 10-token Jaccard similarities between the alternatives and their corresponding primary references are 21.7% and 7.7%, respectively.

In the evaluation, 11 of the 12 models independently produced 65 answers that pass every check, spanning 27 tasks. Furthermore, for 91 of the 115 conditionally failing answers, every failed integration check is passed by at least one answer from another model on the same task. This provides direct evidence that these probes accept implementations beyond the primary references.

Of the 1,270 checks, 46 (3.6%) were not passed by any evaluated answer. Among these, 35 concern boundary conditions, event ordering, lifecycles, repeated operations, or frame-rate consistency. The benchmark was deliberately designed to assess these challenges. A zero pass count therefore does not, by itself, establish that a check is invalid. As a conservative sensitivity analysis, even treating all 46 checks as invalid and removing them leaves the main finding intact: of the 195 answers that then pass all remaining gate and behavior checks, 116 still fail at least one remaining integration check (59.5%, compared with 63.9% originally).

Taken together, these results provide substantial evidence that the probes accept different implementations and that the main finding is robust. During the rebuttal period, two authors manually cross-checked a sample of 30 answers drawn from the 115 conditional integration failures. No false rejections were found in this sample. We will extend this work with a more comprehensive and systematic manual review of failed answers.

AuthorsOpenReview ↗

Q3. Screening thresholds, repeat counts, and redesign records

We sincerely thank the reviewer for this insightful and constructive suggestion. We agree that the screening thresholds, repeat counts, and redesign history should have been described more clearly.

Our previous description did not clearly distinguish the models used in S6 and S8. S6 used Claude Opus 4.6, while S8 used Kimi K2.6, Qwen3.7-Max, and GPT-5.6 Sol.

S6. Each task version receives one trial attempt, using only the starter project and TASK.md. The task returns to S0 for redesign if the attempt fails fewer than three checks. Otherwise, a review session examines whether the failures indicate a problem with the probe or task statement, and suspected problems are referred to human review.

S8. The strongest tier, GPT-5.6 Sol, runs first and makes up to three attempts. The task returns to S0 if more than one attempt passes every check. Each of the two lower tiers initially makes three attempts. If none passes every check, that tier stops. If exactly one does, it is extended to eight attempts in total, with at most one fully passing attempt allowed. A second fully passing attempt in either lower tier triggers redesign. Invalid attempts are rerun and are not counted.

Redesigns. Among the 60 retained tasks, 36 were redesigned at least once, with 75 redesigns in total and at most four per task. Each redesign revised the S0 design and rebuilt the task from S1. S6 triggered 29 redesigns and S8 triggered two. The remaining 44 arose from a pre-build check: Qwen3.7-Max received the parameter file with numeric values hidden and described how it would implement each requirement. A review session returned the design if it judged that this approach would likely fail fewer than three checks in S6.

Qwen3.7-Max and GPT-5.6 Sol, which participated in S8 screening, also appear on our leaderboard. Because tasks were retained only when these models rarely passed every check, their scores may be biased downward by this selection. We have marked their screening role in Table 3. Excluding both models, 99 of the 159 answers that pass every gate and behavior check still fail an integration check (62.3%, compared with 63.9% for all models), so the main finding remains consistent.

We have added these procedural details and redesign counts to Appendix G. To make the task construction process clearer and easier to reproduce, we have open-sourced the construction pipeline and included it in the reproducibility package.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.