acceptodds
Under review as a conference paper at ICLR 2027

Compiled Agency: Coding Agents as Game AI Researchers — from a Roguelike to StarCraft II and Civilization

Abstract

Coding agents are increasingly capable of sustained engineering and empirical research. Greater autonomy makes their research decisions themselves a target for evaluation: what to investigate, which experiments to run, and when to stop. Games provide a controlled, interactive setting with objectively measurable outcomes. We introduce Gauntlet, a develop–freeze–evaluate protocol for studying these decisions as agents build standalone game-playing programs. Starting from a game description, a raw observation/action interface, and an empty policy file, an off-the-shelf agent develops a controller from bare interaction in one autonomous session. We call the capability under study compiled agency: the shipped program plays with zero model calls. Every intermediate version is frozen for later scoring on held-out instances, connecting the agent's research process to independently measured performance. The capability is real and advancing: environment access adds 10 to 78 percentage points of held-out success over construction-only controls, and progress across model generations comes in steps, with tiers that defeat one generation entirely falling to the next. The protocol yields StarCraft II controllers that defeat every fair built-in AI, and in the newest generation the strongest cheating tier as well, and Civilization controllers that win complete games by conquest. Replaying more than 5,000 frozen versions exposes the research behind the programs: gains that plateau early; rigorous local investigation beside sparse validation of what actually ships; and stops that follow a race between the agent's own evidence and a model-specific transcript budget it was never given. When validation panels are refreshed mid-session, so that only the evidence changes, shipped success rises 12.4 points in nine of nine completed pairs of twelve initiated. The experiments an agent designs are part of the capability it delivers; Gauntlet makes them measurable and improvable—a step toward agents whose research practice, not just whose code, can be engineered. We release the benchmark, the corpus, and the replay tooling.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, coding agents, autonomous research agents, agent evaluation, develop-freeze-evaluate, program synthesis, game AI, learning dynamics, validation feedback, StarCraft II, Civilization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fLfiOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Gauntlet, a develop–freeze–evaluate protocol for studying coding agents that autonomously construct standalone game-playing programs. Intermediate programs are frozen and independently evaluated on hidden instances, allowing the agents’ experiments and self-assessments to be compared with actual performance. The study spans a private roguelike, ported games, StarCraft II, and Freeciv, and reports roughly 5,000 scored versions. Its main findings concern generational improvements, early performance plateaus, stale final validation, and a model-dependent scale of stopping. A paired intervention that refreshes validation panels improves held-out clear rate by 12.4 percentage points in nine completed Fable 5 pairs. The Opus 5 and Sonnet 5 experiments give mixed or null evidence on their primary endpoints.

Critical strengths
  1. The protocol makes autonomous development empirically inspectable. Hidden exams and replayed intermediate programs connect what an agent tested with what its code actually achieved, rather than relying on final scores or persuasive transcripts alone. The separation between independent development sessions and held-out game instances is appropriate. The cross-fitted checkpoint analysis is particularly useful: the median apparent advantage over shipping shrinks from +0.05 to +0.01 in utility, showing how the framework can correct a misleading process-level conclusion (Section 5).

  2. The construction controls substantiate the value of interaction with the environment. The spec-only arm retains the harness, tools, prompt, and budgets while withholding the executable engine; all twenty such sessions construct their own simulators or test harnesses. The reported 10–78 percentage-point gains with the full environment therefore provide stronger evidence than a leaderboard alone. The complete-game StarCraft II and Freeciv results also demonstrate that the protocol reaches substantial control-program development. Together, these contributions support weak acceptance.

Critical weaknesses
  1. The stopping claims are overstated. Per Appendix K the race score is an unnormalized pseudo-likelihood, and the race model gets saturation info the baseline doesnt, so its LOSO edge isn’t a calibration win. The hazard result only shows stops follow evaluations. Section 6 already calls the budget a predictive description, but the discussion’s statement that agents stop when this budget “runs out” suggests a stronger explanation. Please move the appendix caveats into the main text.

  2. The refresh gain holds only for the 9/12 completed Fable 5 pairs, and the Appendix L bounds span −7.9 to +25.4 pp. Opus 5 is null on R2 and Sonnet 5 is inconclusive. Since R2 still varies at 80/80, a ceiling doesn’t explain this. Headroom, uptake, model and fork timing are confounded, so what’s left is a model-specific existance result.

Questions for authors
  1. Could you provide a comparison of stopping predictors that uses the same information available at the same prediction time, with normalized probabilities or a directly interpretable prediction error? If this is outside the scope of the study, would you revise the main-text out-of-sample and budget-dominance claims to distinguish the descriptive scale regularity, exploratory race score, and registered hazard result? This would resolve my main soundness concern.

  2. For the Fable intervention, can you report an all-initiated sensitivity analysis using the last executable artifact available in interrupted branches, with one common recovery/stopping rule, or explain why that estimand is unavailable? Please clarify when the recovery and restart rules in Table 6 were fixed relative to inspection of branch exam scores. This would help assess whether the positive effect is robust beyond the completed-pair estimand.

5 comments

Thank you for the precise review. We have added the requested comparison of stopping predictors and an analysis of all initiated Fable 5 intervention pairs, and revised the main-text claims accordingly.

Weakness 1 and Question 1 (stopping). We have added this comparison. Each predictor now uses the same information available at the same prediction time and returns a normalized probability. At each point in a held-out session, it predicts the probability of stopping next using only the model's other sessions and whether the agent has already seen a result showing that the task is solved. Adding this indicator to the model's typical stopping length raises the probability assigned to the observed stop from 0.118 to 0.140 (+0.17 nats per session, 95% interval 0.10 to 0.24; 331 sessions) and removes the miscalibration by saturation status (§7; table in Appendix K).

We have also revised the claims as you suggest. The main text now distinguishes a descriptive regularity (sessions in which the agent never sees the task solved stop at a model-dependent length) from an association (stopping is more likely after a result showing that the task is solved). We make no mechanistic claim and have removed "transcript budget" and "runs out". The race model appears only in the appendix, where it is explicitly described as a score that uses hindsight. We report the registered hazard analysis as null: re-running it on the complete record of the agents' evaluations gives −3 [−12, +6]. The submitted value of −33 came from an incomplete log.

Weakness 2 and Question 2 (the Fable 5 intervention). We can estimate the effect across all initiated pairs because every interrupted branch left an executable program. When we score the last executable program from every branch in both arms, refresh leads by +7.1 clears out of 80 across all twelve pairs (95% CI +1.0 to +13.2; 11 of 12 positive; §8, Appendix L).

Regarding timing, the rules for attempts and re-forks were fixed before the corresponding exam scores were known. Two recovery steps were taken after scores from other branches were known: rewinding and resuming two Fable 5 pairs, and establishing the restart rule for Opus 5 pairs after an infrastructure failure. The seven Fable 5 pairs affected by neither step show +9.9 clears (CI +2.7 to +17.0), with all seven positive.

We agree with your interpretation of the Opus 5 null and the limits on generality. We also need to correct an error: the registered primary endpoint is utility on the 80-seed exam (+0.016, CI −0.003 to +0.034). The +0.0102 reported under that label in the submitted text is the same utility measured on a 1,000-seed reference exam. Both results are null. We also tested the intervention under two additional conditions (introducing the tool mid-session and using a second vendor's CLI), neither of which provided clear evidence of the effect. The paper now presents the Fable 5 result as an existence result for one model under one condition.

Thank you for the additional analyses and clarifications. While these address several concerns, the substantial correction to the hazard result and the incorrectly reported primary-endpoint result leave me concerned about the rigor of the empirical reporting. I therefore move my rating to 2 weak rejection.

Thank you for the follow-up. We understand the doubt that arises when a reviewer is told that something needs to be corrected during revision. However, we humbly disagree that these two points undermine the empirical basis of the paper. One is purely a matter of naming, and the other concerns a supporting remark. For the revision, we added six experiments and recomputed the stopping, intervention, and cost statistics from the session records; in all of this, the analysis behind that remark is the only statistical analysis whose conclusion changed. We therefore believe that the findings of the submission stand and that the revision makes them more solid, and we hope that you will reconsider your assessment. We discuss the two points in detail below.

The primary endpoint: a matter of naming. It affects neither the data nor the conclusion. The submitted text used the name "R2" for two things, the utility endpoint and the 1,000-seed reference exam, and so attached "registered endpoint" to the utility on the reference exam (+0.0102, the value you quoted). The registration names the same utility on the 80-seed exam, which gives +0.016 (CI −0.003 to +0.034). Both values come from the same twelve pairs, and both are null, as your review concluded.

The hazard analysis: a supporting remark. In the submission, its result appeared in one parenthetical clause of the main text ("a registered hazard analysis agrees") and in one paragraph of Appendix K. Nothing else in the paper builds on it: no figure, no table, and no other result. Your comment led us to re-examine it, and we are grateful for that. Its input, the list of evaluation results that each agent had read, was incomplete. We rebuilt the list from the transcripts and re-ran the registered code unchanged; the result is null, and the main text now says so (§7).

The main findings. They do not depend on either point. The steps between model generations, the early plateau, and the comparison of shipped and best versions come from held-out exam scores. The stopping results rest on where sessions stop, on the predictor comparison you requested, and on the new control experiments. The Fable 5 intervention effect was recomputed and is unchanged (+9.9 clears, CI +3.7 to +16.1). The strengths you noted in your review (the cross-fitted analysis, the construction controls, and the complete-game results) are not affected either.

Our effort in the revision went into the additional experiments and the rewriting of the paper as requested by the reviewers, and we kept the rebuttal direct, transparent, and brief. That brevity may have given the impression that much had to be changed. We apologize for the possible confusion, and we would be glad to address any further concerns.

Reviewer_YQUuOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
3
Read review
Paper summary

This work proposes Gauntlet, a benchmark to evaluate agent's "Compiled Agency" - their capability to write game-playing algorithms that perform well offline. Within the Gauntlet framework, they show that the limit of agentic game-algorithm writing is limited by an uninstructed transcript budget, and that providing the models with diverse evidence can improve their performance.

The authors frame this as an autoresearch task and investigate how models treat the challenges with regards to hypotheses testing and iterative improvement of their methods.

Critical strengths
  1. Massive Empirical Scale and Bit-Exact Reproducibility: The underlying experimental campaign represents an immense amount of engineering. Reconstructing, sandboxing, and replaying >5,000 intermediate policy versions across 19 games and ~140 sessions (including full-game StarCraft II and CivRealm conquests) is a major technical accomplishment. Crucially, the supplementary verification suite is genuine and fully functional: scripts for re-scoring policies, regenerating figures, and auditing forked intervention pairs run out-of-the-box and match the paper's reported hashes and statistics exactly.
  2. Compelling Conceptual Paradigm and Methodological Rigor: The formulation of "compiled agency"—evaluating an agent not via prompt-scaffolded test-time inference, but as an empirical engineer that compiles reasoning into deterministic, standalone code—is a timely and clean benchmark paradigm. Methodologically, the paper displays rare scientific honesty: the authors pre-registered their analysis pipelines (methods_preset.json pinned by SHA256), used cross-fitting to dismantle the illusion of checkpoint-selection gains on reused seeds, and transparently reported intervention nulls (e.g., Opus 5 at ceiling and Sonnet 5 failing to adopt the tool).
Critical weaknesses
  1. Severely Degraded, Opaque LLM-Generated Prose: The paper reads as though it were drafted or heavily rewritten by an LLM in a single pass to maximize density, resulting in an obscure, elliptical essay rather than a precise scientific paper. The prose is saturated with inverted syntax, semicolon-stacked fragments, and anthropomorphic fluff (e.g., "the agent writes an optimistic abstract", "ritual of confirmation", "where headroom and uptake meet", "programs worth ranking"). Crucial concepts are used pages before they are defined (or never defined in the text at all), such as "forks", "uptake", and "the diagnosis above." This hyper- compressed, conversational phrasing places an unreasonable cognitive burden on the reader and undermines trust in the presentation.
  2. Confusing Terminology Collisions and Obscure Jargon: The manuscript uses established machine learning terminology in highly non-standard, misleading ways. Most egregiously, the authors describe standalone rule-based controllers as playing "model-free" (e.g., page 8, line 111). In an ICLR venue centered on reinforcement learning, "model-free" universally designates algorithms that learn without an explicit environment transition model P(s'|s, a). Using it here to mean "without calling a Large Language Model at runtime" is a jarring collision. Similarly, borrowing terms like "uptake" (to mean validator tool call frequency) without immediate definitions creates unnecessary ambiguity.
Questions for authors
  1. Manuscript Drafting and Scientific Clarity: The supplementary repository, reconstruction pipelines, and verification tools represent hundreds of hours of high-quality, genuine software engineering. Why, then, does the manuscript text read like an unedited, zero-shot LLM pass filled with telegraphic, stylized, and sometimes meaningless phrasing? Was the text compressed using an LLM to fit the 9-page limit at the expense of technical precision, and do the authors plan a complete human rewrite into standard scientific English?
  2. Disentangling the "Transcript Budget" from Vendor CLI Scaffolding: The finding that agents terminate hard sessions at a consistent, uninstructed "transcript budget" across different games is intriguing. However, Claude models ran under the Claude Code CLI, whereas OpenAI models ran under the Codex CLI. Given that these proprietary harnesses employ fundamentally different system prompts, hidden history trimming, and context-compaction routines, how can you disentangle whether this stopping regularity is an intrinsic property of the underlying foundation models or merely an artifact of the vendor CLIs' context-management scaffolding?
  3. Generalizability of the Panel-Refresh Intervention: The panel-refresh intervention produced a statistically significant effect in only one model (Fable 5), while returning null results on Opus 5 (due to ceiling effects) and Sonnet 5 (due to low tool adoption). Beyond noting that success requires both "headroom and uptake," what concrete evidence demonstrates that serving fresh validation data can be developed into a general, robust harness mechanism rather than an isolated finding in a single model family?
2 comments

Thank you for reading the paper and supplement so carefully and for recognizing the work behind them. The study draws on about 1,200 agent sessions and 2,600 hours of agent work, with 7,812 program versions scored on held-out exams. We apologize that the writing made this work difficult to assess. We have rewritten the main text and thoroughly edited it by hand, and the paper should now read much more clearly.

Weaknesses 1 and 2, Question 1 (writing and terminology). Yes. To fit nine pages, we compressed the submitted text with AI assistance, at the expense of clarity and precision. For the revision, we rewrote the main text and edited it thoroughly by hand, and we used the additional page to explain material that had been too compressed. Each section now begins by stating what it examines, and the result paragraphs lead with their finding in a full sentence before giving the numbers. Section 3 describes the testbed in one continuous account, starting with what Gauntlet is and the scale of the study. Terms such as fork, snapshot, shipped program, natural stop, and transcript length are defined at first use. We have replaced "model-free" with "plays without calling a model", and replaced "uptake" and the other phrases you quote with plain descriptions.

Question 2 (model or vendor CLI). To distinguish model effects from vendor-CLI effects, we wrote a minimal ReAct-style harness of our own and ran six models on the same task through the API, in place of the vendor's CLI. This custom harness is identical for all models, with three tools, no system prompt, and no history trimming or compaction. Stopping length follows the model: five of the six models stop within a factor of 1.25× of their stopping length under the vendor's CLI (ratios across all six: 0.81 to 1.61×), and their ordering is largely preserved (Spearman 0.89; §7, Figure 5b).

We also tested context compaction directly. When Claude Code was forced to compact starting at about 168k tokens, sessions underwent two to eight compactions, with no detectable shortening (1.24×, 95% CI 0.77 to 1.85; Figure 5c). Our custom harness still specifies tools and execution, and we did not vary the context-window size. We therefore claim a regularity across the tested settings, not a mechanism.

Question 3 (generality of the panel refresh). We have no evidence that the effect generalizes, and the revision makes this limitation explicit. We tested the intervention under two additional conditions, neither of which provided clear evidence of the effect: introducing the tool mid-session in ordinary Fable 5 sessions (+1.2 clears out of 80, CI −2.4 to +4.8), and using GPT-5.6-sol under the Codex CLI (+2.8 clears, CI −3.3 to +8.9). The paper now presents the Fable 5 result (+9.9 clears, CI +3.7 to +16.1, positive in nine of nine pairs) as an existence result for one model under one condition, not a general mechanism (§8).

We hope the revised writing makes the work easier to assess. We would appreciate any further comments and your reconsideration of the paper.

​Thank you for the substantial rewrite, the custom ReAct harness control, and the transparent framing of the §8 intervention limits. The manuscript is now much clearer, and the trajectory analysis across 7,812 reconstructed versions provides valuable empirical insights into long-horizon coding agents. I am raising my rating to 3.

​For the camera-ready version, I encourage two presentation improvements to maximize clarity: ​1. Terminology ("Open-Ended"): In RL and agent theory, open-endedness denotes unbounded novelty generation without a fixed terminal goal, whereas Gauntlet's games have static, well-defined win conditions (clearing seeds, conquest, defeating built-in AIs). Replacing "open-ended" with "unbudgeted" or "agent-terminated"—which matches your own Table 1 definition—will avoid conceptual ambiguity.

​2. Decluttering: Moving minor accounting details and null protocol tweaks to the appendix, and simplifying dense multi-panel figures (Figs. 2, 3, 5, 6), will sharpen the core findings.

Reviewer_He71OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates coding agents (primarily Claude Code and Codex) given an open-ended session to write a standalone Python game bot from scratch. Agents get the game rules, a raw observation-action API, an empty policy file, and standard shell tools (with no starter bot, evaluation wrapper, token budget, or clock), and are graded on whatever LLM-free script is in policy.py when the session ends, a setup the authors call "compiled agency." Across custom roguelike games, StarCraft II, and Freeciv, their Gauntlet harness reconstructs intermediate policy edits from the session logs and scores them offline on held-out seeds and scenarios.

Impressively, the newest frontier models write heuristic bots that beat built-in StarCraft II and Civilization AIs. Controls show that live testing against the game engine accounts for most of that gain. Replaying intermediate edits shows that progress on reachable tiers flattens out early into seed churn, agents repeatedly reuse the same small set of test seeds, and unsolved runs tend to quit around a model-specific token scale.

Critical strengths
  1. Clean benchmark setup that others can build upon. Wrapping roguelike games, StarCraft II, and Civilization behind the same minimal interface (an empty policy file, the game API, and standard shell tools, without a bunch of heavy agent scaffolding getting in the way) makes this a nice, practical contribution. Seeing standalone Python scripts with zero runtime LLM calls beat built-in StarCraft II and Civilization AIs is a notable result on its own.

  2. Useful trajectory analysis. Instead of only grading whatever file is sitting on disk at the end, replaying intermediate edits on held-out seeds gives a much clearer look at what the agents are actually doing during a run. Combined with the clean one-shot and spec-only controls (which isolate how much live game execution actually buys), the version-by-version analysis identifies interesting patterns about how agents work in practice, especially how early the real gains happen before runs flatten into sideways seed churn, and how much agents lean on reusing the same small set of test seeds.

Critical weaknesses
  1. Unannounced stopping: the agent is never told a timer exists and is given zero warning before killing the container. The harness kills the session right then and there, recovers whatever untested, mid-edit policy.py happened to be sitting on disk that moment, and grades it as the model's final answer. Now, in some sense it's interesting to see how models plan and behave under such uncertainty, but it is a big confounder and seems unfair to me. At minimum, the model should know the eval might terminate at any time and they should always have a submission.py that reflects their best current version. That seems minimally invasive and eliminates the confounder.

  2. The above scenario is a real issue: consider the three GPT-5.6-sol runs on Civilization easy 1v1 in Table 4. Run i3 went 0 for 16 after working for far more active hours. A tiny footnote under Table 4 explains: i3's policy.py differed by 336 lines from the last tested build, so it seems likely that it got caught mid change.

  3. The confounding effects of manual "continue" prompts to resume stalled responses. The issue is not with using "continue", which presumably does not alter the immediate trajectory of the model too much, but rather with the potential for that to influence how long the model spends on the task, given that there is no time horizon specified. It seems very plausible that a model that has been repeatedly told to continue might spend longer on the task than a model that has not. There is no analysis in the paper to address this possible effect.

  4. The paper would benefit from more clearly positioning itself within the autoresearch literature. In this paper, a coding agent is given a mutable policy.py file, shell access to run the game and see scores, and it sits in an open-ended loop trying to hillclimb a game bot. This seems exactly like unstructured autoresearch (or iterative heuristic/program search). This is basically the same loop people run in Karpathy-style autoresearch repos, just with the pre-built evaluator and keep/revert wrapper removed so the agent has to manage its own testing. Calling that "compiled agency" makes it sound like a brand-new paradigm. Meanwhile, the word autoresearch appears only once (inside a single reference title), and autonomous research agents get brushed past in two sentences in Related Work. This strikes me as unfortunate because owning the autoresearch connection would make the paper stronger. Anyone who has run autoresearch loops in practice has hit the exact failure modes this paper measures: fast early gains followed by hours of wasted churn, overfitting to a small set of reused validation seeds, and messy stopping behavior.

Questions for authors
  1. What defines a "snapshot"? I could not find a snapshotting policy anywhere in the paper. The text only says "every snapshot the agent left in its workspace must byte-match the reconstruction at that moment."

  2. Would the byte reconstructions have been necessary if there had been some automatic snapshotting policy, or if agents were working in a git repo and simply told to commit their work?

  3. Matched-spend curves: how do held-out win rates compare as a function of cumulative input tokens and active wall time?

  4. How many of the 367 "natural stops" in Table 5 and the long-running StarCraft and Civilization sessions received continue messages after stalls, and do your stopping budgets vary between runs with and without continues.

  5. Panel-refresh seed variety: over ~15–20 validator calls, Refresh exposes the agent to roughly 500–800 unique dungeons with per-seed feedback versus only 50 unique dungeons total in Reuse. Wouldn't a fixed 500-seed Reuse panel separate holdout freshness from simply seeing 10× more map layouts?

3 comments

Thank you for the detailed review. We have excluded all externally terminated sessions from the main results, tested your proposed remedy and the effect of continue messages, audited session i3, added matched-spend curves, and reframed the paper around autoresearch.

Weakness 1 (unannounced stopping). The agent decides when to stop by design. The cap is a background safety measure that is never disclosed to the agent, so it cannot affect a session that does not reach it. Most sessions do not: the cap terminated 15 of the 125 roguelike sessions (14 from one slow model, Opus 5) and none in StarCraft II or Civilization. We agree that grading an interrupted session is unfair. We have therefore excluded all sessions that reached the cap from the main results and re-run Opus 5 with a longer cap, which no session reached. Every session in Figure 2 ended when the agent chose to stop.

We also tested your proposed remedy. Eight Sonnet 5 sessions were instructed that they could be terminated at any moment and that the policy file must always contain the best tested version; six received the standard prompt. We found no detectable change in session length (ratio 0.99, 95% CI 0.86 to 1.18) or exam score (+2.8 clears out of 80, CI −13.2 to +19.5; §7, §8).

Weakness 2 (session i3). Our footnote was incorrect: session i3 was not interrupted in the middle of a change. The reported 336-line difference came from comparing the wrong pair of files. The shipped program differs from the last program the agent played by 41 lines: an exception guard and a renamed constant. We also reconstructed 13 earlier builds from i3, including every build that had won a development game, and evaluated each on the 16 held-out games. Like the shipped build, they won 0 to 2 of 16 games (Appendix F). Performance was poor throughout the session. Because we terminated i3 manually, we have also excluded it from the main results.

Weakness 3 and Question 4 (continue messages). We tested this directly on Sonnet 5 using one or two consecutive continue messages. We paused eight sessions and forked each from the same state into branches that resumed with or without the message. We detected no change in session length: the ratios relative to the branch without a message range from 0.94 to 1.00, with all intervals within 0.92 to 1.06 (§7, Figure 5c).

On the counts (Question 4): of the 359 naturally stopped sessions, 13 had received a continue message (24 messages in total, each that single word). In Figure 2, these account for none of the 59 roguelike sessions, 3 of 26 StarCraft II sessions, and 5 of 21 Civilization sessions; the affected sessions are now marked. Excluding these sessions leaves the stopping results essentially unchanged: the hazard ratio in §7 is 5.9 rather than 6.0, and the range of model-specific stopping lengths spans 5.1× rather than 5.4×.

Weakness 4 (autoresearch). We agree, and we have revised the framing accordingly. The abstract, introduction, and related work now present Gauntlet as a study of autoresearch in a controlled setting, and the related work has a dedicated paragraph on research agents. The three failure modes you identify are what the paper measures: early gains that flatten into churn (§6, Figure 3), validation on a small set of reused seeds (§6, Figure 4), and stopping behavior (§7, Figure 5). We retain "compiled agency" only as a name for the capability being tested, and §2 and Table 1 place Gauntlet beside earlier work in which a model writes a game-playing program.

Question 1 (snapshot). A snapshot is an intermediate policy version that the agent saved in its workspace, such as policy_v14.js. We do not use Git, and the harness does not save snapshots; they exist only when an agent chooses to keep one. We use these files to cross-check the versions reconstructed from the session transcript. The term is now defined in §3.

Question 2 (automatic snapshots or git). Automatically saving a snapshot after every write to the policy file would have made reconstruction unnecessary, and Appendix B now notes this for future sessions. We did not ask agents to commit their work because that instruction would itself become part of the task.

Question 3 (matched spend). Appendix E now plots held-out scores against cumulative input tokens, active wall time, and dollar cost. The differences between model generations persist at matched spend. At 200k input tokens, the median Fable 5 session clears 60 of 80 games, compared with 19 for Opus 4.8. After 15 minutes of active time, the median Astra session clears 70.5, compared with 48 for GPT-5.6-sol.

Question 5 (seed variety). Yes, a fixed 500-seed panel would separate freshness from variety; we have not run that arm. The refresh and reuse arms differ in both respects (a median of 440 distinct seeds versus 50). Section 8 now states that the comparison estimates their combined effect.

We would be glad to clarify anything further.

Thank you for the thorough and thoughtful rebuttal. I appreciate the substantial extra work and am raising my rating from 2 to 3.

Thank you for the positive assessment and for raising the rating. We appreciate your careful review and are glad that the additional experiments and analyses addressed your concerns.

P.S. On our side, the review form still shows the earlier rating at the moment. Please ignore this if you have already updated it.

Comments on the paper

1 comment

We thank all three reviewers for their careful reading. We have uploaded the revised paper. Because the main text has been rewritten throughout, individual changes are not highlighted. The main revisions are:

  • New experiments. We added an announced-termination control and a controlled test of continue messages (He71). We wrote a minimal ReAct-style harness and ran six models on the same task through the API, in place of their vendors' command-line agents, and we added a forced-compaction control (YQUu). We also tested the panel-refresh intervention under two additional conditions (YQUu, fLfi).
  • New analyses. We compared stopping predictors using the same information at the same prediction time, and estimated the Fable 5 intervention effect across all initiated pairs (fLfi). We also added matched-spend curves and audited earlier builds from session i3 (He71).
  • Positioning. We now frame the paper as a study of autoresearch in a controlled setting and provide a full related-work section (He71).
  • Writing. The main text has been rewritten and thoroughly edited by hand, with the additional page used to improve clarity (YQUu).

We give further details in the individual responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.