- Rating
- 3
- Confidence
- 1(−1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Gauntlet, a develop–freeze–evaluate protocol for studying coding agents that autonomously construct standalone game-playing programs. Intermediate programs are frozen and independently evaluated on hidden instances, allowing the agents’ experiments and self-assessments to be compared with actual performance. The study spans a private roguelike, ported games, StarCraft II, and Freeciv, and reports roughly 5,000 scored versions. Its main findings concern generational improvements, early performance plateaus, stale final validation, and a model-dependent scale of stopping. A paired intervention that refreshes validation panels improves held-out clear rate by 12.4 percentage points in nine completed Fable 5 pairs. The Opus 5 and Sonnet 5 experiments give mixed or null evidence on their primary endpoints.
Critical strengths
-
The protocol makes autonomous development empirically inspectable. Hidden exams and replayed intermediate programs connect what an agent tested with what its code actually achieved, rather than relying on final scores or persuasive transcripts alone. The separation between independent development sessions and held-out game instances is appropriate. The cross-fitted checkpoint analysis is particularly useful: the median apparent advantage over shipping shrinks from +0.05 to +0.01 in utility, showing how the framework can correct a misleading process-level conclusion (Section 5).
-
The construction controls substantiate the value of interaction with the environment. The spec-only arm retains the harness, tools, prompt, and budgets while withholding the executable engine; all twenty such sessions construct their own simulators or test harnesses. The reported 10–78 percentage-point gains with the full environment therefore provide stronger evidence than a leaderboard alone. The complete-game StarCraft II and Freeciv results also demonstrate that the protocol reaches substantial control-program development. Together, these contributions support weak acceptance.
Critical weaknesses
-
The stopping claims are overstated. Per Appendix K the race score is an unnormalized pseudo-likelihood, and the race model gets saturation info the baseline doesnt, so its LOSO edge isn’t a calibration win. The hazard result only shows stops follow evaluations. Section 6 already calls the budget a predictive description, but the discussion’s statement that agents stop when this budget “runs out” suggests a stronger explanation. Please move the appendix caveats into the main text.
-
The refresh gain holds only for the 9/12 completed Fable 5 pairs, and the Appendix L bounds span −7.9 to +25.4 pp. Opus 5 is null on R2 and Sonnet 5 is inconclusive. Since R2 still varies at 80/80, a ceiling doesn’t explain this. Headroom, uptake, model and fork timing are confounded, so what’s left is a model-specific existance result.
Questions for authors
-
Could you provide a comparison of stopping predictors that uses the same information available at the same prediction time, with normalized probabilities or a directly interpretable prediction error? If this is outside the scope of the study, would you revise the main-text out-of-sample and budget-dominance claims to distinguish the descriptive scale regularity, exploratory race score, and registered hazard result? This would resolve my main soundness concern.
-
For the Fable intervention, can you report an all-initiated sensitivity analysis using the last executable artifact available in interrupted branches, with one common recovery/stopping rule, or explain why that estimand is unavailable? Please clarify when the recovery and restart rules in Table 6 were fixed relative to inspection of branch exam scores. This would help assess whether the positive effect is robust beyond the completed-pair estimand.
5 comments
Thank you for the additional analyses and clarifications. While these address several concerns, the substantial correction to the hazard result and the incorrectly reported primary-endpoint result leave me concerned about the rigor of the empirical reporting. I therefore move my rating to 2 weak rejection.
Thank you for the follow-up. We understand the doubt that arises when a reviewer is told that something needs to be corrected during revision. However, we humbly disagree that these two points undermine the empirical basis of the paper. One is purely a matter of naming, and the other concerns a supporting remark. For the revision, we added six experiments and recomputed the stopping, intervention, and cost statistics from the session records; in all of this, the analysis behind that remark is the only statistical analysis whose conclusion changed. We therefore believe that the findings of the submission stand and that the revision makes them more solid, and we hope that you will reconsider your assessment. We discuss the two points in detail below.
The primary endpoint: a matter of naming. It affects neither the data nor the conclusion. The submitted text used the name "R2" for two things, the utility endpoint and the 1,000-seed reference exam, and so attached "registered endpoint" to the utility on the reference exam (+0.0102, the value you quoted). The registration names the same utility on the 80-seed exam, which gives +0.016 (CI −0.003 to +0.034). Both values come from the same twelve pairs, and both are null, as your review concluded.
The hazard analysis: a supporting remark. In the submission, its result appeared in one parenthetical clause of the main text ("a registered hazard analysis agrees") and in one paragraph of Appendix K. Nothing else in the paper builds on it: no figure, no table, and no other result. Your comment led us to re-examine it, and we are grateful for that. Its input, the list of evaluation results that each agent had read, was incomplete. We rebuilt the list from the transcripts and re-ran the registered code unchanged; the result is null, and the main text now says so (§7).
The main findings. They do not depend on either point. The steps between model generations, the early plateau, and the comparison of shipped and best versions come from held-out exam scores. The stopping results rest on where sessions stop, on the predictor comparison you requested, and on the new control experiments. The Fable 5 intervention effect was recomputed and is unchanged (+9.9 clears, CI +3.7 to +16.1). The strengths you noted in your review (the cross-fitted analysis, the construction controls, and the complete-game results) are not affected either.
Our effort in the revision went into the additional experiments and the rewriting of the paper as requested by the reviewers, and we kept the rebuttal direct, transparent, and brief. That brevity may have given the impression that much had to be changed. We apologize for the possible confusion, and we would be glad to address any further concerns.
Thank you for the precise review. We have added the requested comparison of stopping predictors and an analysis of all initiated Fable 5 intervention pairs, and revised the main-text claims accordingly.
Weakness 1 and Question 1 (stopping). We have added this comparison. Each predictor now uses the same information available at the same prediction time and returns a normalized probability. At each point in a held-out session, it predicts the probability of stopping next using only the model's other sessions and whether the agent has already seen a result showing that the task is solved. Adding this indicator to the model's typical stopping length raises the probability assigned to the observed stop from 0.118 to 0.140 (+0.17 nats per session, 95% interval 0.10 to 0.24; 331 sessions) and removes the miscalibration by saturation status (§7; table in Appendix K).
We have also revised the claims as you suggest. The main text now distinguishes a descriptive regularity (sessions in which the agent never sees the task solved stop at a model-dependent length) from an association (stopping is more likely after a result showing that the task is solved). We make no mechanistic claim and have removed "transcript budget" and "runs out". The race model appears only in the appendix, where it is explicitly described as a score that uses hindsight. We report the registered hazard analysis as null: re-running it on the complete record of the agents' evaluations gives −3 [−12, +6]. The submitted value of −33 came from an incomplete log.
Weakness 2 and Question 2 (the Fable 5 intervention). We can estimate the effect across all initiated pairs because every interrupted branch left an executable program. When we score the last executable program from every branch in both arms, refresh leads by +7.1 clears out of 80 across all twelve pairs (95% CI +1.0 to +13.2; 11 of 12 positive; §8, Appendix L).
Regarding timing, the rules for attempts and re-forks were fixed before the corresponding exam scores were known. Two recovery steps were taken after scores from other branches were known: rewinding and resuming two Fable 5 pairs, and establishing the restart rule for Opus 5 pairs after an infrastructure failure. The seven Fable 5 pairs affected by neither step show +9.9 clears (CI +2.7 to +17.0), with all seven positive.
We agree with your interpretation of the Opus 5 null and the limits on generality. We also need to correct an error: the registered primary endpoint is utility on the 80-seed exam (+0.016, CI −0.003 to +0.034). The +0.0102 reported under that label in the submitted text is the same utility measured on a 1,000-seed reference exam. Both results are null. We also tested the intervention under two additional conditions (introducing the tool mid-session and using a second vendor's CLI), neither of which provided clear evidence of the effect. The paper now presents the Fable 5 result as an existence result for one model under one condition.