- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper focuses on AI systems where an agent is asked to iteratively change and update some code, by keeping the change if a validation score goes up; in particular, the paper verticalizes on how to decide if having the score going up means a real improvement. Indeed, the paper found that, in one pipeline, the effect monitored is smaller than noise. In addition, the paper studies 18 deployed agent systems, finding that, for 16 of them, the decision was taken after a single run. The paper finally recommends the usage of Welch t-test to validate score improvements.
Critical strengths
- The paper addresses the problem of how to detect if an improvement in one trial is indeed due to a real effect or if it can be explained by noise only. This is a critical topic that is sometimes neglected in experiments. Indeed, in the paper, the authors show one experiment with 480 runs where the effect that is being measured is smaller than the noise.
- The paper finds, in 16 over 18 deployed agents, that the decision after a run is taken only based on a single trial repetition, thus not accounting for noise. This is real evidence that statistical analysis is not performed in these systems.
Critical weaknesses
- The paper recommends using a Welch t-test, which is an already well-known solution to the addressed problem. Given so, the main contribution is the analysis performed with the single 480 run experiment, and the investigation of the 18 agents; the 480 run experiment is based on a single pipeline and a single model family [Limitations, line 465]. A more broad analysis, i.e., multiple model families and multiple pipelines would lead to more robust findings.
- The paper is sometimes hard to read: “Everything above holds the hardware fixed. A loop that concludes “below threshold” has made a claim about a machine, and we can now say how much of one.” [Section 6 - line 356]; “Localising it: a card that does not perturb cannot move” [Section 6 - line 388]; “That a bit-exact arm cannot flip is forced, not observed; the evidence is on the perturbing side.” [Section 6 - line 400]. These passages make the findings hard to follow for readers.
Questions for authors
- (Question) Do you have any insight/evidence that the findings in the main 480 run experiment generalize to other model families and to more pipelines?
Response to Reviewer ym3X
Reply to Reviewer ym3X. Thank you for the review.
Welch being well known: we agree, and we do not present it as new. The findings are about the setting. On measured noise at a short, known horizon, the anytime-valid rules we tried do not improve on Welch, and a rule handed a noise level from too few runs can lose its guarantee with no visible sign. The revision also adds a result that does not depend on Welch: when we replay 100-decision loops, no per-decision rule makes a loop's reported gain trustworthy.
Generalization: we repeated the training experiment on a second family, Gemma-2 (2B student, 9B teacher), with 30 runs for each of the same 16 conditions. The floor is present and wider. The median full-precision standard deviation is 0.22 points against 0.12 for Llama, and after GPTQ it is 0.52 against 0.55, so its size depends on the model.
We also ran a second pipeline, the community anchor loop (karpathy/autoresearch, unchanged code), 60 times on three A40s. On one card, identical runs differ by up to 0.0004 bits per byte at the 95th percentile. Therefore, its keep-if-lower rule keeps a do-nothing edit half the time, while its instructions name 0.001 as a gain worth weighing. Across cards the floor is larger. The budget is wall-clock time, so a faster card fits more steps, and one card landed 0.002 lower with the code unchanged. We then ran it 60 more times on four H100s, the hardware the loop targets (new appendix). The same card with the same code landed 0.0037 lower when it had the machine to itself than when three other runs shared it.
In other words, the floor's size depends on the pipeline and the machine, and we do not claim that the numbers carry over. The repeated-training experiment now covers two model families on our pipeline. Section 6 covers three models from two families on frozen weights, which is a different quantity. What we claim transfers is the method: re-run the unchanged pipeline before trusting a comparison.
Readability: you are right about those passages. We rewrote them in plain terms, along with the surrounding text in Section 6.