acceptodds
Under review as a conference paper at ICLR 2027

Measure First: The Noise Floor of an Autonomous Research Loop Dominates Its Accept Rule

Abstract

Autonomous research agents change their code, retrain, evaluate, and keep the change if the score went up. Recent work proposes better statistical tests for this keep-or-discard decision. We argue that the first problem is that nobody measures how noisy the score is, and we measure it in four ways. First, we re-run one production LLM distillation-and-quantization pipeline, unchanged, thirty times in each of sixteen settings (480 runs): run-to-run spread varies more than 13-fold across settings, and the improvement the pipeline's own authors were chasing cannot be distinguished from this noise (its 95% upper bound is 0.54 of it). Second, we read the source code of 26 systems: sixteen of eighteen deployed agent loops keep a change after a single run, and none lets a noise estimate reach the decision. Third, we replay eleven decision rules on the measured runs. The anytime-valid e-process, written against a noisy baseline as is natural, is not valid (false-accept 0.038, 0.148 and 0.278 at 8, 32 and 128 runs); corrected, its best variant reaches 0.048 power at half a standard deviation, where a five-run Welch t-test reaches 0.170 and needs no external noise estimate. Taking that estimate from three runs breaks the corrected rule in all sixteen settings. Fourth, we evaluate the same frozen weights 3,627 times on seven NVIDIA GPU models. Scores move in whole questions; 13 of 39 model, task and GPU combinations return the identical score every time while others drift by up to thirty questions; two GPUs that are each perfectly repeatable still disagree by twenty; and running inputs one at a time instead of batched removed the drift in every case we tested. We recommend two things: before trusting any comparison, re-run the unchanged pipeline several times on the hardware the loop will use, to learn how much the score moves on its own; and decide with a test that estimates the noise from the runs themselves, such as a Welch t-test, rather than one handed a noise level from outside.

Primary area:
datasets and benchmarks
Keywords:
evaluation, reproducibility, autonomous research, agents, measurement, anytime-valid inference, nondeterminism

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ym3XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper focuses on AI systems where an agent is asked to iteratively change and update some code, by keeping the change if a validation score goes up; in particular, the paper verticalizes on how to decide if having the score going up means a real improvement. Indeed, the paper found that, in one pipeline, the effect monitored is smaller than noise. In addition, the paper studies 18 deployed agent systems, finding that, for 16 of them, the decision was taken after a single run. The paper finally recommends the usage of Welch t-test to validate score improvements.

Critical strengths
  1. The paper addresses the problem of how to detect if an improvement in one trial is indeed due to a real effect or if it can be explained by noise only. This is a critical topic that is sometimes neglected in experiments. Indeed, in the paper, the authors show one experiment with 480 runs where the effect that is being measured is smaller than the noise.
  2. The paper finds, in 16 over 18 deployed agents, that the decision after a run is taken only based on a single trial repetition, thus not accounting for noise. This is real evidence that statistical analysis is not performed in these systems.
Critical weaknesses
  1. The paper recommends using a Welch t-test, which is an already well-known solution to the addressed problem. Given so, the main contribution is the analysis performed with the single 480 run experiment, and the investigation of the 18 agents; the 480 run experiment is based on a single pipeline and a single model family [Limitations, line 465]. A more broad analysis, i.e., multiple model families and multiple pipelines would lead to more robust findings.
  2. The paper is sometimes hard to read: “Everything above holds the hardware fixed. A loop that concludes “below threshold” has made a claim about a machine, and we can now say how much of one.” [Section 6 - line 356]; “Localising it: a card that does not perturb cannot move” [Section 6 - line 388]; “That a bit-exact arm cannot flip is forced, not observed; the evidence is on the perturbing side.” [Section 6 - line 400]. These passages make the findings hard to follow for readers.
Questions for authors
  1. (Question) Do you have any insight/evidence that the findings in the main 480 run experiment generalize to other model families and to more pipelines?
1 comment
AuthorsOpenReview ↗

Response to Reviewer ym3X

Reply to Reviewer ym3X. Thank you for the review.

Welch being well known: we agree, and we do not present it as new. The findings are about the setting. On measured noise at a short, known horizon, the anytime-valid rules we tried do not improve on Welch, and a rule handed a noise level from too few runs can lose its guarantee with no visible sign. The revision also adds a result that does not depend on Welch: when we replay 100-decision loops, no per-decision rule makes a loop's reported gain trustworthy.

Generalization: we repeated the training experiment on a second family, Gemma-2 (2B student, 9B teacher), with 30 runs for each of the same 16 conditions. The floor is present and wider. The median full-precision standard deviation is 0.22 points against 0.12 for Llama, and after GPTQ it is 0.52 against 0.55, so its size depends on the model.

We also ran a second pipeline, the community anchor loop (karpathy/autoresearch, unchanged code), 60 times on three A40s. On one card, identical runs differ by up to 0.0004 bits per byte at the 95th percentile. Therefore, its keep-if-lower rule keeps a do-nothing edit half the time, while its instructions name 0.001 as a gain worth weighing. Across cards the floor is larger. The budget is wall-clock time, so a faster card fits more steps, and one card landed 0.002 lower with the code unchanged. We then ran it 60 more times on four H100s, the hardware the loop targets (new appendix). The same card with the same code landed 0.0037 lower when it had the machine to itself than when three other runs shared it.

In other words, the floor's size depends on the pipeline and the machine, and we do not claim that the numbers carry over. The repeated-training experiment now covers two model families on our pipeline. Section 6 covers three models from two families on frozen weights, which is a different quantity. What we claim transfers is the method: re-run the unchanged pipeline before trusting a comparison.

Readability: you are right about those passages. We rewrote them in plain terms, along with the surrounding text in Section 6.

Reviewer_knFbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how much experimental noise affects the keep-or-discard decisions made by autonomous research systems. It combines 480 repeated runs and 3627 evaluations of frozen models across seven GPU models. The main findings are that run-to-run variability differs substantially across settings, uncertainty checks are often absent or not enforced. The paper recommends measuring variability before comparing changes and using a test that accounts for uncertainty in the variance estimate.

Critical strengths

It is interesting that two GPUs can each give perfectly consistent results but still disagree with each other. The conclusion that reproducibility on the same GPU does not guarantee reproducibility across different GPUs. The experiments on batch size are also useful, since they show a simple way to reduce this variability. The results show that estimating noise from too few runs can lead to unreliable decisions, even when a statistical test appears to perform well.

Critical weaknesses
  1. I am not fully convinced by the recommendation to use a five-run Welch test. Most of the main comparisons assume that the two groups have the same variance, while real experiments may have very different variances. Also, the paper only compares against a particular type of sequential test. It would be helpful to include methods that can handle unknown variance. Without these comparisons, I think the conclusion that Welch is preferable is a bit too strong.

  2. Some claims in the survey could be more carefully stated. For example, the authors claim that no system uses a noise estimate in its acceptance decision. However, Table 7 uses this through a prompt.

Questions for authors
  1. How reliable is the five-run Welch test when the two groups have different variances?

  2. Have you tested its false-acceptance rate when the true means are equal?

  3. How do you distinguish between rules that are not enforced in code and rules that agents do not actually follow?

1 comment
AuthorsOpenReview ↗

Response to Reviewer knFb

Reply to Reviewer knFb. Thank you for the review. Several of your points are correct, and the revision reflects them.

Welch with unequal variances and equal means: you are right that our main replay shifts one pool by a constant, so it never exercised this case. We now test it directly. We centre two real pools on the same mean and draw five runs from each, so the null is exact while the shapes and SD ratios are real. Over the 16 same-task Adam/Muon comparisons (SD ratio 0.53 to 1.89), false-accept averages 0.047 with a worst case of 0.073, and 7 of 16 are above 0.05 after Bonferroni. Over all 240 ordered pairs of pools (ratio up to 3.7) it averages 0.049, worst 0.095, with 91 above. Gaussian data at the largest same-task ratio gives 0.048, so we believe the excess comes from the shape of the pools rather than from the unequal variances. That is, Welch holds on average, but a single comparison at five runs can overshoot by about half its nominal level, and ten runs only bring the worst same-task pair down to 0.068. Therefore, the revised paper calls Welch approximately level rather than exact.

Real comparisons: we also score actual Adam/Muon pairs where the full pools resolve the sign, which is six of 16 directed comparisons. Welch keeps the better candidate 0.382 of the time (0.394 when the truth comes from a held-out half), the corrected e-process 0.188, AdaStop 0.322 and its one-sided variant 0.450. Best-of-1 keeps a genuinely worse candidate 20.0% of the time, against 0.2% for Welch.

Other sequential tests: the replay includes AdaStop, which needs no noise scale. Its one-sided variant reaches 0.238 at half a standard deviation, against 0.250 for Welch at k = 8 on about the same budget. However, it re-draws the incumbent, and one-sided it is no more exact than Welch (worst 0.076 on the unequal-variance null). We also added a scale-free t-test martingale (Wang and Ramdas). On eight pairs its best setting reaches 0.050 at false-accept 0.010, against 0.250 for Welch on the same 16 runs. Betting on the bounded metric (Waudby-Smith and Ramdas) never reached its threshold in eight pairs. These tests buy validity at every horizon, and that validity is what costs the power. Therefore, our recommendation is now scoped to a loop that knows its horizon.

The survey claim: you are right. One system's instruction file tells the agent to keep a change only if the gain exceeds a measured noise floor, so a noise estimate can reach the decision through the prompt. The revised text says that none of the systems enforces a noise estimate in code, and it describes that case.

Code versus behaviour: we do not distinguish these. We read the rules from source and did not measure whether agents follow prompt-only rules. Sections 4 and 8 now say so, and measuring adherence remains open.

Reviewer_v56pOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that the dominant problem in autonomous research loops (agents that change code, retrain, evaluate, and keep a change if the score improved) is not the statistical accept rule but the unmeasured noise floor of the score itself. It makes this case through four complementary measurements.

Critical strengths

Important, under-appreciated thesis, argued where it counts. The claim that "the missing measurement, not the missing test, is what binds" is a genuinely useful reframing of a fast-growing literature on accept rules for self-improving agents.

The source-code survey is a model of rigor.

Critical weaknesses

Generalization rests on a single pipeline and one model family at depth. The authors acknowledge this ("what transfers is the method rather than the number"), but it is the central limitation.

It is primarily an empirical study and how to address the problem should be further expanded.

Questions for authors

For the non-normal pools where the Stein-style t still exceeds α (worst 0.070 at k=3), did you try a bootstrap-calibrated or permutation threshold, and does it recover level without further power loss?

1 comment
AuthorsOpenReview ↗

Response to Reviewer v56p

Reply to Reviewer v56p. Thank you for the review.

Bootstrap and permutation thresholds: we tried both on the new runs over all 16 conditions, with sigma estimated from three runs and 8 candidate runs against 5 incumbent runs. A bootstrap-t threshold does not recover level, since three runs give too few distinct resamples; the worst condition reaches 0.096 (mean 0.038, power 0.157 at half a standard deviation). A permutation test of the means does recover it: mean 0.035, worst 0.048, no condition above 0.05, and power 0.223 against 0.129 for the t_(k-1) threshold under the same setup. It gains power because it drops the external noise estimate, and we believe this supports the recommendation. Its limit is that it assumes exchangeable arms. With unequal variances and equal means, a five-run permutation test averages 0.051 with a worst case of 0.071, which is level with Welch.

Generalization: we agree that this is the central limitation. We have now repeated the training experiment on a second family, a Gemma-2 2B student distilled from a 9B Gemma-2 teacher through the same corrected pipeline, with 30 runs for each of the same 16 task and optimizer conditions (480 runs). The floor is there, and it is wider. The median full-precision standard deviation across conditions is 0.22 points (Llama 0.12), with a range of 0.07 to 0.45 (Llama 0.09 to 0.32). After GPTQ the two families are nearly the same, a median of 0.52 points against 0.55. The optimizer effect on the quantization gap is 0.20 of its own noise in both, so testing it would need about 300 runs per arm either way. Note that the Gemma runs differ in three ways, which the new appendix states: the teacher is 9B, the training text is the Llama-generated set, and scores come from lm-eval's HF backend rather than vLLM. Therefore, we compare spreads, not score levels.

We also ran a second pipeline, the community anchor loop, 60 times unchanged on three A40s. Its floor is small on one card: 0.0004 bits per byte between identical runs at the 95th percentile, against the 0.001 gains its instructions name. Across cards it is larger, and a faster card landed 0.002 lower because its wall-clock budget fit more steps. On four H100s, the hardware the loop targets, 60 more runs show the same: one card scored 0.0037 better when it was alone on the machine than while three other runs shared it, which is 3.7 times the 0.001 gain. In other words, the size of the floor depends on the pipeline and the machine. We claim that the method transfers, not the numbers.

How to address the problem: Section 7 now gives five recommendations with run counts. Welch at half a standard deviation has power 0.18, 0.30, 0.48 and 0.62 at 5, 10, 20 and 30 runs per arm, and the effect chased here would need about 290 runs per arm for 80% power. A loop chasing effects that small needs a budget most loops do not have. We also replayed whole loops of 100 decisions. Even with Welch, 53% of loops accept falsely at least once under a true null. Therefore, we now recommend reporting a loop's gain only from runs taken after it stops.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the revision

Thank you for the reviews. We have uploaded a revision, and this note lists what changed so that you do not need to read the whole diff.

  1. New runs behind Sections 3-5. As configured, the pipeline's LoRA path omitted the next-token shift in its cross-entropy, both in training and in the validation loss that the search minimizes. All 480 submitted runs were affected. Therefore, we restored the shift and re-ran all 16 conditions at n = 30 under the same protocol: 480 new runs, all completed. Section 6 and the survey do not use these runs. On the new runs:
  • Run-to-run spread across settings is 3.7-fold (SD 0.086 to 0.318 points), where the submission reported 13.2-fold with the widest condition set aside. The collapsed MMLU condition is gone.
  • The effect that the pipeline's authors were chasing is still consistent with noise: rho = 0.20, permutation p = 0.48, signed mean +0.02 (SE 0.10), and a conservative 95% upper bound of 0.51 on the mean absolute effect (was 0.54).
  • The validation-loss proxy does not run backwards. However, it does not track the reported metric either (r = +0.075), so we removed the earlier reading.
  • Welch at five runs has 1.8 times the power of the best valid e-process we tried (0.180 against 0.099 at half a standard deviation), down from 3.5 times. Part of that drop comes from a new and stronger comparison, a fixed-lambda martingale on the information clock.

What held: the e-process written against a noisy baseline is still not anytime-valid (false-accept 0.038, 0.149 and 0.280 at 8, 32 and 128 runs).

  1. Corrections to earlier statements. A three-run noise estimate breaks the corrected gate in 11 of 16 conditions (mean 0.064), where the submission said all 16; tied calibration draws are now handled. Seeding is now tested in four conditions instead of one, and it is a partial cure: the variance ratio is 0.61 (95% interval 0.32-1.36), and only one condition's interval excludes 1. The survey wording is corrected as described in our reply to knFb.

  2. New results. These are Welch under equal means and unequal variances, real Adam/Muon comparisons with unequal variances, permutation and bootstrap thresholds, scale-free sequential tests, and a replay of 100-decision loops. The loop replay changes our conclusion. That is, no per-decision rule here makes a loop's reported gain trustworthy, and on true gain alone best-of-1 can beat Welch in our assumed streams while overstating its gain about 4.6-fold.

  3. The recommendations now give run counts, and Section 6 is rewritten in plainer prose.

Details are in our replies to each review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.