acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

Abstract

A generally intelligent system should acquire new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language-model agents depend strongly on human-engineered harnesses that determine perception, memory, tools, recovery, and inference allocation. ARC-AGI-3 offers a concrete test of this gap: human-steered systems already solve all public levels while using fewer actions than human baselines, so effective designs are known to exist. We ask whether frontier models can build and improve such a harness autonomously. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across the study, models construct diverse and sometimes highly capable harnesses, but substantial feedback-driven revisions do not consistently improve held-out performance. Development gains often fail to transfer, and reductions in inference cost can come at the expense of task performance. For models evaluated in both settings, even revised harnesses fall well short of reported human-steered systems. These findings reveal a gap between autonomous harness construction and reliable adaptation: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
LLMs, AI agents, harness adaptation, autonomous self-improvement, generalization, ARC-AGI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qWXYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper studies whether frontier models can autonomously build and improve their own harness used on ARC-AGI-3. The authors have each of 9 combinations of models and reasoning strengths independently build 3 harnesses, revise each harness twice based on feedback from 5 development games, and evaluate all versions on 20 held-out games. The results show that initial harness performance varies widely, that the held-out average improves by only 2.80 points after two rounds of revision, and that gains on the development set clearly overestimate gains on held-out. The authors also conduct a coding analysis of revision behavior and compare with systems designed under human leadership.

Critical strengths

The experimental protocol is relatively tightly controlled, held-out games take no part in building or revision, every version of each lineage is evaluated, and reasoning cost is also fully recorded.

Critical weaknesses

The experimental scope and the conclusions of the paper are both insufficient to answer the question posed by the title. All experiments are run only on 25 public games from a single benchmark, each harness is revised only twice, development feedback comes only from 5 games, and the authors also acknowledge that these games may already have appeared in the models' training data. Under this setting, the core results still cannot be distinguished from noise, with a held-out average gain of 2.80 points after two rounds of revision, a confidence interval that includes zero, and a standard deviation that is already 3.99 for repeated evaluations of the same harness, while each snapshot is evaluated only once. The main finding that the development set overestimates held-out performance has already been reported in existing work such as SEAGym and HarnessOpt-Bench, and the paper also proposes no improvement method. Therefore, this work is more like a descriptive observation under a specific protocol, and the contribution is not enough to support a title as broad as whether an LLM can improve its own harness.

Questions for authors
  1. The average gain from two rounds of revision falls within the range of variation from repeated evaluation of a single harness, and overestimation of held-out by the development set has already been reported, so what new conclusion about the question posed by the title can this study actually draw under a protocol that fixes two rounds of revision and uses only 5 development games?
  2. How do the results distinguish weak ability to improve the harness from weak ability to execute in the games for conditions such as Terra, Luna, and Sonnet that always score near zero, when in each condition the model that builds the harness and the model that plays the games in the harness are the same?
3 comments

Thank you for the review.

Scope. “Can LLMs improve their own harness?” is a research question that will take many studies to answer. This paper tests one part of it: whether models that build their own harness can improve it through two revisions using feedback from five games. We repeat this across nine model–effort conditions, with three independent constructions each. The subtitle, “A Controlled Study on ARC-AGI-3,” and the Limitations section state that scope. We do not claim to settle what longer runs or other benchmarks would show. The general response explains why we chose breadth over more revisions, and why public games do not undermine a within-lineage comparison.

Noise. The review sets the 2.80-point mean gain against the 3.99-point SD of repeated evaluations. These are different quantities:

  • The 3.99 is how much one harness's score varies between runs.
  • The 2.80 is a mean over 27 lineages. Its uncertainty is the interval we report, [−1.44, 7.35], which is computed from the observed spread of the 27 lineage changes (SD 11.92) and so already contains run-to-run variation.

If every snapshot varied as the measured one does, evaluation noise would account for under a quarter of that spread's variance. Most of it reflects real differences between lineages: the same procedure helps some harnesses substantially and harms others.

We tested this directly for the rebuttal. Before collecting new results we selected a large gain, a large loss and a near-zero case, then evaluated each initial and revised harness three times on all held-out games (360 evaluations). The large improvement and the large regression both reproduced in all three fresh pairs (+30.8 and −16.6 on average); the near-zero case changed direction between runs (Table 26). The average gain is small because substantial gains and losses offset each other, not because nothing happens. Repeating every evaluation ten times would have cost about USD 145,000, so we spent the budget on independent constructions, which expose differences that repeating one harness cannot show.

The development–held-out gap is 5.62 [1.78, 9.77] in the main analysis. After excluding game comparisons affected by execution recovery, the estimate remains positive (4.21) but its interval includes zero. We report that sensitivity explicitly.

Prior work. The studies differ in what they start from and what they score:

  • SEAGym compares existing self-evolution methods and update schedules.
  • HarnessOpt-Bench lets an optimizer search from a supplied seed and select a final candidate. It measures that candidate's gain over the seed.
  • Our study first lets models build their starting harnesses independently. Each becomes the baseline for two revisions, and we evaluate all three stages. This measures variation in construction and whether the same update procedure improves or damages the resulting systems.

The gap also differs: HarnessOpt-Bench compares the best validation score during search with the selected candidate's test score. We compare development and held-out gains from the same revision.

No method. This is an empirical study. Its contribution is evidence about what models can build, when their revisions help or harm performance, and whether development gains transfer to other games. Proposing a new optimization method is a different contribution.

Q1. Four conclusions under this protocol:

  1. Construction is strong and highly variable.
  2. Development progress overstates transfer.
  3. Updaters do not abstain, but development evidence helps choose what to deploy.
  4. Lower cost is not evidence of improvement.

Q2 (builder vs. player). The original design could not separate the two, so we ran a crossed evaluation that swaps only the runtime model in frozen snapshots; generated code is byte-identical (Table 25, three lineages per arm, mean held-out RHAE):

  • Terra and Luna are limited by their gameplay. Playing Sol-built harnesses, they score only 1–6 points above their own harnesses and 19–26 points below Sol in the same harnesses. A better harness does not lift them.
  • Sonnet is limited by the harnesses it builds. They run without errors, but Sol playing them scores only 1–2 points more than Sonnet does, about 25 points below Sol's own harnesses.
  • Weak players compress revision effects: the Sol lineages' H2−H0 changes (+33.18, −15.18, −10.70) lie between −5.51 and +8.20 under Terra or Luna.

With one evaluation per cell, these comparisons are descriptive; the reverse cells (e.g. a Terra-built harness played by Sol) were not run.

Thanks the authors for their improvement, which solved some of my problems about the soundness. I decided to increase my score a bit. However, I still think the contribution of this work is too incremental and too small to be accepted by ICLR. Thus I will maintain my score.

Thank you for reading our response and for raising the soundness score.

We disagree on one point: no single paper will settle whether models can improve their own harness. Longer runs, more benchmarks and repeated evaluations need more time and money than one paper can usually cover (see general response). The picture will come from several controlled studies that each add a piece (your main criticism is that we overclaim, but the restriction to ARC-AGI 3 is right in the title). This paper is one such piece, and the study took several months of work. To give an idea of the scale: the models built 147 harness candidates, and we evaluated the 81 frozen versions in 2,025 games, with about 512,000 game actions, 133,000 model responses, 23,000 USD cost if we include the further rebuttal experiments and 1,278 hours of gameplay. We do not consider this "too small", and we believe harness improvement is one of the most important questions in the current debate around AI self-improvement.

Reviewer_W3QdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether language models can autonomously construct and improve the executable harnesses through which they solve interactive tasks. With model weights and a general-purpose coding bootstrap fixed, each model builds an ARC-AGI-3 harness and revises it twice using feedback from five development games. All three snapshots are evaluated on twenty games excluded from adaptation. The study covers nine model–effort conditions, 27 independent construction lineages, 81 snapshots, and 2,025 game evaluations, alongside an outcome-blind analysis of 54 revision episodes.

Critical strengths

The study evaluates every prescribed snapshot rather than reporting only successful descendants, and the twenty held-out games provide no feedback for construction, revision, or selection. Independent construction replicates and whole-lineage bootstrap resampling preserve the paired structure of the comparisons. This design makes the study useful for assessing the behavior of an update procedure beyond its best observed outcome.The paper distinguishes revisions that improve performance while reducing cost from those that become cheaper by solving less effectively. It also reports checkpoint-selection diagnostics, execution variability, practical change thresholds, and annotation limitations. These disclosures strengthen the study's value and make its claims more assessable. An informative empirical contribution of this kind does not require a new optimization algorithm or a state-of-the-art task score.

Critical weaknesses
  1. The primary endpoint always deploys the second revision. Yet Appendix D, Table 5 shows that selecting among the three existing snapshots using only development RHAE yields a mean held-out improvement of 4.53 points [0.81, 8.68], compared with 2.80 points for deploying the final revision. It also reduces observed regressions from twelve lineages to five, with nine lineages retaining the initial harness. The paper acknowledges this result and correctly describes it as post hoc, but it materially qualifies the interpretation of unreliable self-improvement: generating revisions and deciding which version to deploy are distinct capabilities. I would like the selection result to become a coequal part of the main conclusion, with clarification of how and when the rule was chosen. The current evidence is strongest as an evaluation of the specified update-and-deployment protocol; it supports a broader assessment of autonomous improvement less directly. This matters because that broader distinction is central to the paper's significance.

2.Each main-study snapshot is evaluated once. Appendix D.1 repeats one post hoc selected final harness ten times, obtaining a suite-level standard deviation of 3.99 RHAE points, but does not repeat its parent or establish variability across other harnesses. Consequently, the fifteen gains, twelve losses, and nine strict score–cost improvements remain classifications of observed outcomes. Table 9's five-point threshold changes the counts to nine improvements, three regressions, and fifteen indeterminate cases; Figure 4 even includes a roughly 0.0001-point decline. The authors explicitly acknowledge this limitation, and the lineage bootstrap is not thereby invalid, but reliable improvement and capability preservation are the central quantities the study seeks to illuminate. Targeted paired repetitions would provide more direct evidence. Recovery adds a related uncertainty: Appendix G records 45 additional incomplete attempts without an outcome reason and seventeen selected fresh restarts. There is no evidence of choosing the best among multiple clean scores, but a sensitivity analysis is needed to determine whether unresolved interruptions affect these relatively small aggregate effects.

Questions for authors
  1. When was the development-based selection rule in Table 5 specified, and were alternative rules examined after held-out results were available? How should the positive result under this rule change the main interpretation of autonomous improvement? I am not treating development-based selection itself as test leakage; the issue is the timing of the rule's specification and the distinction between revision generation and deployment.

  2. Can you provide paired repeated evaluations of parent and descendant snapshots for a small, prespecified sample spanning near-zero changes, substantial improvements, and substantial regressions? Which regression and score–cost classifications remain supported after accounting for execution variability? A limited validation would be sufficient; I am not requesting repetition of the entire campaign.

  3. How are the 45 unexplained incomplete attempts and seventeen selected fresh restarts distributed across stages and development/held-out games? Do the mean improvement, adaptation gap, and regression counts change materially under paired exclusion or conservative bounds for affected evaluations? Confirmed external infrastructure failures should remain distinguished from model-generated software failures.

2 comments

Thank you for an exceptionally careful review. We agree with both critical weaknesses and acted on each. Selection is now a co-equal result, with its provenance documented (this reply). We also ran the paired repeats and the recovery analyses you proposed (second reply).

Q1a (timing). The rule was specified post hoc. Version history fixes the order:

  • The 24 lineages of eight conditions ran 17–28 Aug 2026, and their held-out scores were tabulated on 31 Aug.
  • The rule first appears on 3 Sep (local time, UTC+2), already labeled a post hoc diagnostic.
  • No analysis plan was preregistered. The campaign manifest fixes replicates, prompts and the replacement policy, but no endpoint or selection rule.

Q1b (alternatives). None were computed. The rule (highest five-game development RHAE, ties to the earlier snapshot) is unchanged since 3 Sep. To test how much it depends on particular games, we added leave-one-game-out variants for this response:

  • Over H0: +3.95 to +4.16.
  • Over always deploying H2: +1.15 to +1.36, with four of five intervals including zero.

Only the three Astra lineages (13–15 Sep) postdate the rule. For them it gains +0.08 over H2, consistent with a near-ceiling H2 of 94.37, so it is no validation.

Q1c (interpretation). We agree that generating revisions and deciding which to deploy are distinct capabilities, and the revised paper makes selection a co-equal result:

  • Abstract: "No revision retains its parent, yet a post hoc rule that selects among existing snapshots by development score raises the held-out gain to 4.53 points ([0.81, 8.68]), separating the generation of revisions from the decision to deploy them."
  • Section 5 adds that the rule cuts observed regressions from 12 to 5 and remains to be evaluated prospectively.
  • Conclusion: our evidence is strongest for the specified update-and-deploy protocol and speaks less directly to autonomous improvement in general.

The selection result refines our reading; it does not reverse it. The updaters could have kept their parent and never did, in 54 of 54 revisions, so the deployment decision is a capability they did not exercise. With selection, five lineages still regress, and the gain of 4.53 remains small against the headroom that human-steered harnesses show for the same models. We therefore conclude, as you suggest, that revision generation is unreliable under this protocol, and that an explicit deployment decision recovers part of the benefit.

AuthorsOpenReview ↗

Repeats and Discovery

Why one evaluation per snapshot. Evaluating all 81 snapshots ten times would cost about USD 145,000 at list prices (general response), which is why the ten-run diagnostic covered a single harness. A small prespecified paired sample, as you propose, is the design we could afford, and we ran it.

Q2 (paired repeats). Three lineages were prespecified from historical outcomes before any new inference. Each H0 and H2 was re-evaluated three times on all 20 held-out games (360 games), with H0 and H2 run concurrently. Fresh H0 is compared with fresh H2, because provider-side changes since August cannot be excluded.

Lineage (original H0 → H2)Fresh H2−H0, three pairsMean [95%, df = 2]
Sol-high r1, gain (20.38 → 53.56)+35.05, +36.25, +21.15+30.82 [9.97, 51.67]
Sol-high r2, loss (27.92 → 12.74)−21.96, −14.02, −13.74−16.57 [−28.16, −4.99]
Fable-high r1, near-zero (16.05 → 13.77)−4.41, +2.40, −1.45−1.15 [−9.64, 7.33]
  • Large changes reproduce: all three fresh pairs agree for the gain and for the regression, so neither is an execution artifact. The same update procedure substantially improves one harness and damages another. The loss interval ends at −4.99, just inside the ±5 band, so under our preset rule its practical class is unresolved; the frozen plan anticipated this case as borderline.
  • Near-zero stays unresolved, as the frozen plan expected.
  • Cost: no strict score–cost classification is supported, because every cost interval includes zero.
  • Recoveries: 15 of the 360 games hit an explicit provider failure and were resumed once under the original campaign's policy (Table 26).

The revised paper therefore leads with the ±5-point counts (9/3/15) rather than strict signs, for cost as well: of 18 cheaper final harnesses, 5 gain, 3 lose and 10 change by less. The 0.0001-point decline in Figure 4 is a near-floor Sonnet lineage and is indeterminate at any threshold above zero.

Q3 (recovery). In short, recovery handling leaves the held-out gain and the gain/loss counts essentially unchanged, and it widens the uncertainty of the gap.

  • Scores: all 2,025 selected scores were recomputed from the original scorecards, and all match. Each slot has exactly one clean completed attempt.
  • The 105 additional attempts:
    • 49 carry an explicit provider-infrastructure reason (42 HTTP 504, 7 timeouts).
    • 11 are blocked by replay or scorecard checks.
    • 45 are incomplete, with a pending request and no recorded cause, so we do not attribute them.
    • None is a model-software failure: harness error exits occur within clean attempts and are scored as outcomes.
  • Distribution: the incomplete attempts and the 17 fresh restarts (all on held-out games) affect 44 slots in eight lineages; Table 28 gives the stage × split counts.
  • Paired exclusion: removing affected games at both endpoints changes the held-out gain from 2.80 to 2.56 and the gap from 5.62 to 4.21. Gains and losses stay at 15/12. The gap's interval then includes zero, [−0.05, 8.87].
  • Bounds: replacing affected scores with any value in [0, 100] gives a gain in [−0.21, 6.46], a gap in [−1.38, 10.47] and 12–15 losses.

The revised paper states that the gap's point estimate is stable while its interval is sensitive to recovery handling (Tables 28–29).

Reviewer_LxE4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether LLMs can improve their own agent harness on ARC-AGI-3. Starting from an OpenCode bootstrap, each of 9 model-effort conditions runs 3 lineages of two self-revisions (27 lineages, 81 snapshots, 2,025 game evaluations), using 5 development games and 20 held-out games, scored by RHAE. Held-out improvement from H0 to H2 is +2.80 (95% CI [-1.44, 7.35]), against +8.42 on development games, giving a generalization gap of 5.62 [1.78, 9.77]. Fifteen lineages improve and twelve decline (p = 0.70); repeat-evaluation SD is 3.99. Revisions reduce cost per game (mean -$2.72, median -$0.54). Of 54 revisions, 22 of 27 first revisions reorient the harness. Human-steered harnesses (Tycho, VISTA, AVO) reach 100 RHAE. Total compute exceeds $18,129.

Critical strengths
  • Important question. Self-improving harnesses are widely discussed; a careful, pre-specified test is valuable.
  • Honest reporting. The paper reports confidence intervals, sign tests and the development/held-out gap rather than cherry-picked lineages, and states the null result clearly.
  • Scale. 2,025 game evaluations across 9 model-effort conditions is a substantial investment.
  • Useful descriptive findings. The characterization of what revisions change (mostly reorientation) and the cost reductions are informative.
Critical weaknesses
  • Low statistical power. Three lineages per condition and a single evaluation per snapshot, with evaluation noise SD about 4 relative to effects about 3, mean the study cannot distinguish small real gains from zero.
  • Narrow development signal. Only five development games guide revisions. This limits the range of adaptation evidence and is a plausible explanation to test for the generalization gap, not an established causal explanation.
  • Only two revisions. Two revision steps may be too few to observe compounding improvement.
  • Selection policy remains worth analyzing. The protocol explicitly permits preserving the parent, so forced revision is not a valid criticism. A useful follow-up is how often preservation occurs and whether development-based best-snapshot selection improves held-out outcomes without selection bias.
  • Contamination risk. ARC-AGI-3 games are public and may be in training data.
  • LLM-coded qualitative analysis. Revision behavior is coded by GPT-5.6 Sol in two passes, low-kappa codes are dropped, and there is no human validation.
  • Replication burden. The reported compute expenditure exceeds $18,129, and the evaluation uses proprietary models. This makes independent replication resource-intensive. Model identifiers, access dates and settings should be documented precisely.
Questions for authors
  1. With the observed noise, what effect size could the study detect with 80% power? How many lineages or repeated evaluations would be needed?
  2. Since preserving the parent is allowed, how often does that occur? How does development-based best-snapshot selection affect held-out outcomes?
  3. Does the post hoc dev-based selection (+4.53) hold up under cross-validation across development games?
  4. Can you validate the LLM-coded revision categories on a human-labeled subset?
  5. What distinguishes the human-steered harnesses that reach 100 RHAE from the self-revised ones?
1 comment

Thank you for a careful review.

Q1 (power). The study is powered for gains of the size the question is about. Human-steered harnesses reach 100 RHAE with the same models (Opus 5 xhigh: 100, against 39.95 for our revision-2 harnesses), so an update procedure that reliably turned feedback into capability had tens of points of room. With the observed SD of the 27 paired H2−H0 changes, 11.92, the design has 80% power for a mean gain of 6.4–6.7 RHAE, and the interval ends at 7.35. The result is thus informative: two revisions do not deliver gains near what the headroom allows. Whether a gain of a few points exists stays open. Resolving a 3-point gain would need about 126 lineages (about USD 85,000); repeats per snapshot help only modestly, because lineages genuinely differ (Table 21). The gap, 5.62 [1.78, 9.77], excludes zero in the primary analysis. Its interval is sensitive to recovery handling (see W3Qd).

Five development games. The signal is larger than the count suggests (Table 19):

  • Levels and budgets: the games have 6–9 levels, with human baselines of 7–163 actions per level and a 5× action budget per level.
  • Feedback received: the 54 feedback bundles cover 270 parent-game evaluations and 666 completed level instances, with about 42,700 actions, traces and costs.
  • Split comparison: before revision, both splits score alike (26.13 development vs. 24.54 held-out); the gap opens only after revision.
  • Game composition: the gap does depend on it. Omitting r11l from the development summary gives 3.02; omitting any other game gives 4.68–7.84.

Two revisions, contamination. See the general response.

Q2 (preservation). None of the 54 revisions retains its parent; each changes at least 84 lines (median 449). Development-based selection retains H0 in 9 lineages (3 by ties). It reduces regressions from 12 to 5, gains +1.72 [0.38, 3.67] over always deploying H2, and raises cost from USD 7.57 to USD 9.15 per game. It was specified post hoc (see W3Qd and Table 23).

Q3 (cross-validation of selection). Omitting each development game in turn, gains over H0 stay at +3.95 to +4.16 (5–6 regressions). Gains over H2 are weaker, +1.15 to +1.36, with four of five intervals including zero. Because all candidates were revised on all five games, this checks the selector, not adaptation.

Q4 (LLM coding). We had no independent human annotators, so we tested the labels differently. Both codes we interpret are defined by counts and event order, which a deterministic, non-LLM script can re-derive. Its rules were frozen before any comparison.

  • SCOPE: the script matches the LLM consensus in 51/54 episodes (quadratic κ = 0.90). The scope contrast holds: −3.77 [−9.35, 1.62] for contained minus broad, against −3.94 published.
  • ADAPT (reactive vs. hand-off): it matches in 46/54 (κ = 0.64; 0.74 if smoke repairs whose edits are cut from the packets also count). This contrast is not robust: +0.48 [−5.09, 7.05] under the frozen rules and −2.39 to −0.17 under the two variants, against −3.24 published.

The revised paper presents the ADAPT contrast as dependent on its operationalization. No quantitative result depends on these labels.

Replication. The cost is intrinsic to the question, since each data point is a complete build–play–revise trajectory; it also limits further revisions and repeats (general response). To lower the burden, the artifact contains all 81 snapshots, all 2,025 game rows and scripts that reproduce the submission's statistics without inference. Coding costs USD 1,352 of the USD 18,129; the rest is evaluation and qualification. Table 20 gives model versions and run dates.

Q5 (human-steered harnesses). No controlled contrast isolates one cause, but the public reports and our snapshots show three differences (revised Section 7):

  • What the model sees during play. The reference systems keep the full interaction record available to the model (Tycho: a typed history; VISTA: every frame; Retrodict: a log it replays in code). To save cost, our Sol harnesses use stateless calls or short chains with bounded notes, and our Opus harnesses rely on code written at build time and call the model only 14–18 times per game.
  • Development. Researchers refined those harnesses over runs on the public games; Tycho also selects among four policies on all 25. Ours receive two autonomous revisions from five games.
  • Model and effort. The Sol references use maximum effort; an earlier Retrodict run with Sol high scored 84.5.

Spend alone does not explain the gap: Retrodict reports about USD 26 per game, while two of our initial Sol-xhigh harnesses spent USD 31–38 per game for 44 RHAE. Our Astra scores mainly reflect the model, which the official leaderboard lists at 99.9 with a simple harness that preserves reasoning state.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their detailed reviews. New analyses are collected in a new Appendix H of the revised PDF.

Main-text changes:

  1. Abstract and conclusion: selection as a co-equal result, scope of the evidence
  2. Section 5: primary result first, robustness of the gap, ±5-point counts for score and cost
  3. Section 5 (gain against headroom), Section 7 (abstention, comparison with human-steered harnesses) and Section 8 (builder/player)

Main findings. The reviews discuss the paper mostly through one number, the +2.80 mean held-out gain. The paper claims more than that, and our replies refer to these findings, so we restate them:

  1. Autonomous construction yields harnesses spanning 0 to 97 held-out RHAE.
  2. Two revisions change mean held-out RHAE, our primary comparison, by +2.80 [−1.44, 7.35]. Development RHAE rises by 8.42, so the adaptation gap is 5.62 [1.78, 9.77].
  3. No revision retains its parent, although this was allowed. Selecting among existing snapshots by development score, a post hoc rule, yields +4.53 [0.81, 8.68] over H0.
  4. Lower cost does not indicate improvement: cost falls in 18 of 27 lineages, yet only 5 of these gain more than 5 held-out points, and 3 lose more than 5.

Scale of the study. All three reviews ask for more revisions, repeated evaluations or more lineages. Each would add information. The constraint is cost: every data point is a full trajectory in which a model builds a harness, plays 25 games through it and revises it. The study already comprises 2,025 game evaluations at a list-price cost of at least $18,129. From the recorded stage costs (Table 8) we estimate:

  • one further revision of all 27 lineages: about $5,200;
  • ten evaluations of every snapshot instead of one: about $145,000;
  • the 126 lineages needed to detect a 3-point gain with 80% power: about $85,000.

We spent the budget on breadth, nine conditions × three independent constructions, because breadth is what exposes finding 1 and shows that one update procedure helps some lineages and harms others. For the rebuttal we ran the targeted checks the reviewers proposed: 810 further game evaluations, about $5,000.

Why two revisions. We measure how reliably a fixed update procedure transfers across many independent starting points, not the best result a long search can reach. The two steps add +1.33 and +1.47 held-out points and +5.46 and +2.96 development points; we extrapolate neither collapse nor compounding. Longer studies suggest that horizon alone is not decisive: SEAGym loses 14.5 ID-test points over 39 of 40 completed updates while a 20-update schedule gains 9.1, and RRSI obtains its transfer gains over 20–40 rounds with explicit proposal and acceptance rules.

Contamination (LxE4, qWXY). It cannot be excluded, but it does not explain our results. Development and held-out games are equally public and score alike before revision (26.13 vs. 24.54). All comparisons are within-lineage with the model fixed, so its prior exposure is the same before and after revision. Several models score near zero through their own harnesses (Sonnet 0.09, Luna 1.45, Terra 4.16 initial held-out). A direct test needs non-public games.

New evidence:

  • Paired repeats of H0 and H2 for three prespecified lineages (W3Qd) test whether single-evaluation outcomes are noise. The large gain and the large regression both reproduce in all three fresh pairs (+30.8 and −16.6 on average). Only the change of a few points stays unresolved, which is why we now lead with ±5-point counts.
  • A crossed builder/player evaluation (qWXY) separates building a harness from playing through it. A Sol-built harness barely helps Terra or Luna, so their gameplay limits their scores. Sonnet's harnesses stay near zero even when Sol plays them, so for Sonnet the limit is the harness it builds.
  • Selection rule (W3Qd, LxE4): when it was specified, dated from our version history, and whether its gain survives omitting each development game in turn.
  • Recovery (W3Qd): sensitivity analyses and worst-case bounds for interrupted evaluations.
  • Further checks: power analysis, exclusion of near-floor conditions, a rule-based check of the LLM-coded labels, the scale of the development signal, model versions and run dates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.