- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This paper studies whether frontier models can autonomously build and improve their own harness used on ARC-AGI-3. The authors have each of 9 combinations of models and reasoning strengths independently build 3 harnesses, revise each harness twice based on feedback from 5 development games, and evaluate all versions on 20 held-out games. The results show that initial harness performance varies widely, that the held-out average improves by only 2.80 points after two rounds of revision, and that gains on the development set clearly overestimate gains on held-out. The authors also conduct a coding analysis of revision behavior and compare with systems designed under human leadership.
Critical strengths
The experimental protocol is relatively tightly controlled, held-out games take no part in building or revision, every version of each lineage is evaluated, and reasoning cost is also fully recorded.
Critical weaknesses
The experimental scope and the conclusions of the paper are both insufficient to answer the question posed by the title. All experiments are run only on 25 public games from a single benchmark, each harness is revised only twice, development feedback comes only from 5 games, and the authors also acknowledge that these games may already have appeared in the models' training data. Under this setting, the core results still cannot be distinguished from noise, with a held-out average gain of 2.80 points after two rounds of revision, a confidence interval that includes zero, and a standard deviation that is already 3.99 for repeated evaluations of the same harness, while each snapshot is evaluated only once. The main finding that the development set overestimates held-out performance has already been reported in existing work such as SEAGym and HarnessOpt-Bench, and the paper also proposes no improvement method. Therefore, this work is more like a descriptive observation under a specific protocol, and the contribution is not enough to support a title as broad as whether an LLM can improve its own harness.
Questions for authors
- The average gain from two rounds of revision falls within the range of variation from repeated evaluation of a single harness, and overestimation of held-out by the development set has already been reported, so what new conclusion about the question posed by the title can this study actually draw under a protocol that fixes two rounds of revision and uses only 5 development games?
- How do the results distinguish weak ability to improve the harness from weak ability to execute in the games for conditions such as Terra, Luna, and Sonnet that always score near zero, when in each condition the model that builds the harness and the model that plays the games in the harness are the same?
3 comments
Thanks the authors for their improvement, which solved some of my problems about the soundness. I decided to increase my score a bit. However, I still think the contribution of this work is too incremental and too small to be accepted by ICLR. Thus I will maintain my score.
Thank you for reading our response and for raising the soundness score.
We disagree on one point: no single paper will settle whether models can improve their own harness. Longer runs, more benchmarks and repeated evaluations need more time and money than one paper can usually cover (see general response). The picture will come from several controlled studies that each add a piece (your main criticism is that we overclaim, but the restriction to ARC-AGI 3 is right in the title). This paper is one such piece, and the study took several months of work. To give an idea of the scale: the models built 147 harness candidates, and we evaluated the 81 frozen versions in 2,025 games, with about 512,000 game actions, 133,000 model responses, 23,000 USD cost if we include the further rebuttal experiments and 1,278 hours of gameplay. We do not consider this "too small", and we believe harness improvement is one of the most important questions in the current debate around AI self-improvement.
Thank you for the review.
Scope. “Can LLMs improve their own harness?” is a research question that will take many studies to answer. This paper tests one part of it: whether models that build their own harness can improve it through two revisions using feedback from five games. We repeat this across nine model–effort conditions, with three independent constructions each. The subtitle, “A Controlled Study on ARC-AGI-3,” and the Limitations section state that scope. We do not claim to settle what longer runs or other benchmarks would show. The general response explains why we chose breadth over more revisions, and why public games do not undermine a within-lineage comparison.
Noise. The review sets the 2.80-point mean gain against the 3.99-point SD of repeated evaluations. These are different quantities:
If every snapshot varied as the measured one does, evaluation noise would account for under a quarter of that spread's variance. Most of it reflects real differences between lineages: the same procedure helps some harnesses substantially and harms others.
We tested this directly for the rebuttal. Before collecting new results we selected a large gain, a large loss and a near-zero case, then evaluated each initial and revised harness three times on all held-out games (360 evaluations). The large improvement and the large regression both reproduced in all three fresh pairs (+30.8 and −16.6 on average); the near-zero case changed direction between runs (Table 26). The average gain is small because substantial gains and losses offset each other, not because nothing happens. Repeating every evaluation ten times would have cost about USD 145,000, so we spent the budget on independent constructions, which expose differences that repeating one harness cannot show.
The development–held-out gap is 5.62 [1.78, 9.77] in the main analysis. After excluding game comparisons affected by execution recovery, the estimate remains positive (4.21) but its interval includes zero. We report that sensitivity explicitly.
Prior work. The studies differ in what they start from and what they score:
The gap also differs: HarnessOpt-Bench compares the best validation score during search with the selected candidate's test score. We compare development and held-out gains from the same revision.
No method. This is an empirical study. Its contribution is evidence about what models can build, when their revisions help or harm performance, and whether development gains transfer to other games. Proposing a new optimization method is a different contribution.
Q1. Four conclusions under this protocol:
Q2 (builder vs. player). The original design could not separate the two, so we ran a crossed evaluation that swaps only the runtime model in frozen snapshots; generated code is byte-identical (Table 25, three lineages per arm, mean held-out RHAE):
With one evaluation per cell, these comparisons are descriptive; the reverse cells (e.g. a Terra-built harness played by Sol) were not run.