acceptodds
Under review as a conference paper at ICLR 2027

The Noise Floor of LLM Program Discovery

Abstract

Systems such as FunSearch, AlphaEvolve and ShinkaEvolve put a language model inside a search loop and report gains over plain sampling, usually from one or a few runs. We ask whether those gains are real at the budget most practitioners have, a few hundred LLM calls per problem. Holding the model, the tasks, the call budget and the final selection rule fixed, we compare best-of-N sampling with two island-evolution methods, reflect-then-sample and an EXP3 router on fifteen verified tasks, with 10 to 20 independent seeds per method and task. The result is a noise floor. At 100 calls no method beats sampling by more than 1.03 seed standard deviations on any task, and reflection and adaptive allocation never win; only FunSearch-style evolution has a small, consistent edge. At 300 calls that edge grows large on some instances and turns negative on others, and a pre-registered ladder of covering designs shows that distance from the published record does not predict which. What does move the floor is the model: with a weak 7B model whose first-shot programs are often broken, evolution repairs them and beats sampling clearly, and still ties where the model plateaus. The seed count matters as much as the method. With three seeds per method, the norm in this literature, our own data would have missed four of five effects the full analysis detects and, in one method pair in nine, named a winner it does not support. We therefore report every tie with its minimum detectable effect, every win with its split-half replication rate, and every prediction as written before the data came in, including the ones that failed.

Primary area:
datasets and benchmarks
Keywords:
LLM program search, evolutionary search, best-of-N sampling, FunSearch, test-time compute, noise floor, statistical reliability, random seeds, pre-registration, combinatorial constructions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bBTrOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically examined whether the claimed improvements by the "LLM program discovery" search loop systems (such as FunSearch, AlphaEvolve, ShinkaEvolve, Mind Evolution, etc.) truly exist under actual budgets (approximately several hundred calls), or if they are merely seed noise. The author conducted 15 tasks with deterministic validators (11 combination coverage design instances + MAX-3-SAT + planting groups), using DeepSeek V4-Flash as the main model, and systematically compared 6 methods including the best-of-N, FunSearch, Evolve, two reflective variants, and Hedge.

Critical strengths

The system compared six methods: best-of-N, FunSearch, Evolve, two variants of reflection, and Hedge. Each method was run with 20 independent seeds (N = 100 calls) for each task, and 10 seeds were run at N = 300. Additionally, cross-model verification was conducted on four tasks using Qwen2.5-7B and gpt-4.1-mini.

Critical weaknesses
  1. Among the 13 main tasks, 11 are from the La Jolla coverage design library. The reward is the number of discrete blocks, the structural height being isomorphic, and there is a known platform; the software/theorems/scientific discovery tasks with a stronger "program discovery" flavor are completely absent. There is a gap between the title's "LLM Program Discovery" and the empirical evidence.
  2. The main conclusion "100 calls without a method exceeding best-of-N 1.03 seed-SD" is entirely from DeepSeek V4-Flash; the other two models only appeared on 4 tasks and 10 seeds. The most interesting "model moving the floor" narrative in the entire text is precisely based on this very thin cross-model data (for example, on the same task C (20,6,3), FunSearch won 4.9 blocks and Evolve lost 4.0 blocks, which itself looks like noise).
  3. Hedge degraded to a repeated copy of the best-of-N due to the closure of compute feedback, and the authors themselves admitted that it "operates with 80% efficacy"; Reflect₃₂ only ran on 7 tasks; FunSearch/Evolve is a lightweight implementation transplanted by the authors rather than the original version optimized in the paper. On this asymmetric grid, "33 out of 135 pairs are distinguishable", which makes it easy to confuse "our prototype is not better than the published system" with "the published system is not better than sampling".
Questions for authors
  1. Please re-run the tests on tasks with more continuous rewards and more noisy evaluations (such as floating-point optimization, software engineering tasks with noisy validators). In such tasks, do you expect the noise floor to be higher or lower than the coverage design? If the floor itself varies with the task form, where is the boundary of the generalizable range of the "1.03 seed-SD" number?
  2. If you transfer the entire 15 tasks × 20 seed grids to gpt-4.1-mini (the saturated model described in your report) and Qwen2.5-7B (the model that often fails in the initial trial program), do you expect the main conclusion "There is no method that exceeds the best-of-N 1.03 seed-SD" to be retained, expanded, or overturned? Specifically, what evidence would make you recognize that the noise floor is a model-related quantity rather than an empirical constant that can be referenced across models?
  3. Among the 135 method comparisons reported in your report, how many pairs were conducted between two implementations that have reached the published system optimization level? If someone uses the complete configuration of the original FunSearch/AlphaEvolve (archive selection, cluster clustering, temperature scheduling) to re-run the same grid in the future, do you expect what proportion of the effects currently classified as "noise" will become reproducible significant advantages? In other words, "Evolution is not better than sampling in 100 calls" is a conclusion about the method category or about your current implementation?
3 comments

Thank you for the review. The general response describes the new experiments and defines the seed-SD, and the revised PDF marks all changes in blue.

W1. Mostly covering designs, and no software, theorem-proving or discovery tasks. Seven of the fifteen tasks are covering designs, three of them in the nine-task main grid, and the other eight are Sidon sets, LABS, Ramsey colourings, circle packing, two codes, MAX-3-SAT and planted clique (Table 1). These are the construction problems on which FunSearch, AlphaEvolve and ShinkaEvolve report results, and in each the model writes a program. We agree the other domains are absent, and the abstract and Limitations now say so.

W2. One model, and thin cross-model evidence. We agree the cross-model evidence is thin, as §6 says. The example cited combines two results from the main model. On C(20,6,3) at 300 calls both methods beat sampling, FunSearch by 4.9 blocks and Evolve by 3.4, and the 4.0-block loss is FunSearch on C(28,6,3) (§4.8). The strongest cross-model result is not marginal, since Qwen's win on C(28,6,3) reappears in 98% of random seed splits (§4.9).

W3. Hedge, Reflect32, the implementations and the grid. The 80% figure is the statistical power behind the minimum detectable effect (§3), not a description of Hedge, whose duplicate arm is disclosed in §3 and §6. Reflect32 ran on all nine main-grid tasks with twenty seeds, so the 135 pairs come from a symmetric grid, and it was Reflect16 that ran on seven tasks at 300 calls. FunSearch is a line-by-line port of the released logic with three disclosed deviations, and Evolve is a simpler second method added so the result does not rest on one implementation. Your concern about the port is fair, and we tested it (Q3).

Q1. Continuous or noisy rewards, and how far does 1.03 generalise? LABS and circle packing already have continuous scores and behave like the discrete tasks (between-to-within ratios 0.32 and 0.37), and the high-variance planted clique ties on every comparison. Our verifiers are exact, so noisy validators lie outside the suite. The 1.03 figure is an observation for one model at 100 calls, not a constant, and the new experiments show it moving, to 1.15 on five tasks without FunSearch's resets and to about two seed-SD at 1000 calls on two tasks. What we expect to transfer is the measurement, a gap in seed-SD units with its minimum detectable effect.

Q2. Would the conclusion survive on the full grid with the other models? The paper already treats the floor as depending on the model, which is its third result (§4.9). The 1.03 bound fails for Qwen, where evolution wins by 1.0 to 1.6 seed-SD on two of four tasks, and is exceeded once for gpt-4.1-mini. On a full grid we would expect it to hold where a model's sampling plateaus within 100 calls and fail elsewhere, as recorded before the runs in Appendix D. We did not run the full grid, and Limitations says so.

Q3. Category or implementation? It is a conclusion about the two implementations we ran, at 100 and 300 calls, and the revision says so. It does not cover AlphaEvolve, ShinkaEvolve or production budgets. Our FunSearch keeps the released clustering, cluster selection and temperature schedule. Your question led us to test its one free deviation, the island reset schedule. With resets off, the four advantages over sampling remain detectable with no detected difference between settings, and the Ramsey loss becomes a gain of 6.8 [+1.6, +11.7] (§4.11). One of the five results we retested therefore depended on our implementation, which is why the revision limits its conclusions to the methods as run.

Thanks for the response. After reading through the response, I find that my previous concerns remain unaddressed. Taking into account the comments from the other reviewers, I have decided to maintain my original score.

Thank you for reading our response. We understand that two of your concerns are not resolved by this revision, the absence of software and theorem-proving tasks and the limited cross-model evidence. The revised paper states both as limitations and scopes its claims to verified construction and solver tasks at a few hundred calls. On your question of whether the conclusion concerns the method category or our implementation, we tested the one free choice in our FunSearch port, and it changed one of the five outcomes we retested (§4.11), which is why the revision limits its conclusions to the methods as run. If there is a specific concern beyond these that you see as unaddressed, we would be glad to respond to it before the discussion closes.

Reviewer_ZsLcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is an interesting and timely paper. Under specific budgets, it compares best-of-N sampling with evolution, reflection, and adaptive allocation for LLM-generated programs. The authors report that across verified construction and solver tasks, method differences at 100 calls are generally small relative to run-to-run variation. Interestingly, evolution sometimes helps at 300 calls, but can also hurt. Tests with two additional models suggest that situations when sampling has not yet reached a model’s task-specific performance plateau, are the situations where evolution helps the most.

Critical strengths

I believe the paper addresses an important evaluation problem; that is, whether reported gains from LLM search loops in published papers hold up across independent runs, rather than appearing in a few "lucky" seeds. This becomes more and more relevant, with LLM-driven search being used for many practical discovery tasks.

Overall, there are three items that make the study more informative than only comparisons based on a small number of best runs, which are the 1) controlled comparisons, 2) common final selection rule, and 3) pre-registered predictions.

Importantly, the cross-model experiments offer a plausible explanation for some mixed results in prior work: when the model’s initial samples are poor, evolutionary feedback may help, but add little once sampling reaches a plateau.

Critical weaknesses

In my opinion, the conclusions are sometimes a bit broader than the evidence. One model, a narrow set of mostly combinatorial construction tasks, and two budgets are used for the main grid . The additional models are tested on only four tasks each. This might not yet support a general claim about when LLM evolution or adaptive allocation works. Does cross-model comparison really isolate capability from other differences between models and providers?

I think the “Noise floor” also needs careful interpretation. The paper defines it through seed-to-seed standard deviation and reports detectable effects (which is helpful). But I believe an observed difference below a minimum detectable effect is not necessarily evidence that methods are equivalent? The paper acknowledges this for the most parts, yet I believe it could still be improved overall. The abstract also notes that FunSearch has a “small, consistent edge”. I'm wondering if the mixed per-task results really support this?

As far as I understand it, some method comparisons have limited scope. Hedge’s compute arm becomes a duplicate of sampling when compute is disabled. this means that its results might not establish whether adaptive allocation generally helps. The FunSearch implementation also has declared deviations from the released method; the paper should try to make more clear how those choices might affect the comparison.

Questions for authors
  • Are the methods compared at equal cost? Maybe I missed it, but the budget is measured in API requests, but methods may differ in the number of samples per request and in token usage. Could you report tokens and cost by method, and check whether the main results hold under matched-token or matched-cost budgets?

  • It wasn't so clear to me how sensitive are the results to failed runs and invalid champions? Could you show an analysis that counts failures as outcomes, such as assigning them the lowest score, rather than excluding them?

  • Less of a question but please specify how reliable are the final test scores. How many instances are in each test partition, and are scores averaged across enough independent instances to support the comparisons? Are the same test instances reused across runs?

2 comments

Thank you for the review. The general response describes the new experiments and defines the seed-SD, and the revised PDF marks all changes in blue.

W1. Conclusions broader than the evidence, and does the cross-model test isolate capability? We agree and have narrowed the claims. The abstract now scopes them to verified construction and solver tasks at a few hundred calls, and the new 1000-call run (§4.10) shows the conclusion changing with the budget, with evolution ahead by 1.7 to 2.4 seed-SD on two of four tasks. The cross-model test does not isolate capability, because changing the model also changes provider and family, and the revision says so in Limitations and no longer claims otherwise in the introduction. Within each model the pattern is that evolution wins where that model's own samples are poor, since Qwen wins by 1.0 to 1.6 seed-SD on its two non-plateaued tasks and gpt-4.1-mini only on C(28,6,3) (Table 3). This is consistent with model strength mattering but does not establish it.

W2. Non-significance is not equivalence, and is FunSearch's edge "consistent"? Agreed on both. The paper tests non-inferiority directly in §4.1, and sampling is non-inferior within one seed-SD in 39 of 45 main-grid comparisons. §3 now defines a tie as a pair not distinguishable at the stated minimum detectable effect, and we call sampling a match only where non-inferiority holds. "Consistent" overstated four wins of about one seed-SD, one loss and a pooled effect of +0.17 [−0.01, +0.36], so it is removed and the edge is reported task by task. The one loss turned out to come from our implementation (W3).

W3. Hedge's duplicate arm and the FunSearch deviations. Agreed on Hedge, whose ties say nothing general about adaptive allocation, and the revision refers only to the EXP3 router we ran. Of the three FunSearch deviations (Appendix B), two give it the same inputs and selection rule as every other method, and the third, resetting its islands every N/4 requests where the released code uses a four-hour clock, was a free choice that we have now tested. With resets off (§4.11, Table 10), its four advantages over sampling remain detectable with no detected difference between settings, while its one loss, on Ramsey, reverses to a gain of 6.8 [+1.6, +11.7]. The largest gap at 100 calls becomes 1.15 seed-SD on these five tasks, against 1.03 across all fifteen, and the other ten tasks were not rerun.

Q1. Equal cost? Each generated program is one charged request, so FunSearch's four samples per prompt count as four, and all methods share the model and output cap. Token use does differ, and Table 11 now gives the cost per request relative to sampling. FunSearch averages 1.02 times, Evolve 1.33 and reflection about 1.15, so equal calls favours Evolve and reflection, whose ties are therefore conservative, while on the larger covering designs FunSearch costs 1.2 to 1.9 times sampling. A matched-cost comparison will follow in the final version.

Q2. Failed runs as outcomes? We recomputed every comparison with each failed run set to the worst score on its task. The seven failed DeepSeek runs (0.4%) change no comparison against sampling at 100 or 1000 calls, and at 300 calls they make one more loss for evolution distinguishable, Evolve on circle packing. The three failed Qwen cells are all sampling cells, so the evolutionary wins only grow. Appendix B reports this.

Q3. Test-set size and reuse? Each task's test set is a constant of the task, shared by every run and method and never seen during search. The sets are small. Codes and covering designs are scored on the single object they ask for, so their score is exact, and the other tasks use one or two fixed instances (listed in Appendix B). Scores are therefore not averages over many instances, and we do not claim generalisation to other instances. The comparisons measure which method finds the better program on the same fixed test set, so their uncertainty comes from run-to-run variation alone, which the seed-SD captures.

Thank you for the clarifications. I will keep my positive evaluation.

Reviewer_ATGQOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper analyzes the extent to which differences in search methods and variability across independent runs affect discovery performance in algorithm discovery. The authors investigate whether there are performance differences between methods and to what extent random variation across runs affects the results.

  • The experiments show that, with around 100 LLM calls, performance differences among search methods are small on many tasks. Meanwhile, performance varies substantially across runs, indicating that the best results obtained from a small number of runs are insufficient to properly assess the relative performance of different methods. The authors also show that the effectiveness of evolutionary search depends on how well the LLM can perform on each task through simple sampling.

Critical strengths
  1. The analysis is interesting and provides useful insights for algorithm discovery. The observation that the best results can vary substantially across independent runs, even with the same method, is also consistent with my personal experience.
Critical weaknesses
  1. The experimental scale is small. The authors conduct experiments with budgets of 100 and 300 LLM calls and argue that, at around 100 calls, performance differences among search methods are small and the methods do not demonstrate a substantial advantage over simple sampling. I understand that the study focuses on comparisons under small budgets, but there seems to be a gap between these settings and typical practical settings. Is it possible that the results would change at a larger scale? For example, if the budget were increased to 1000 or more calls, could performance differences between methods become substantially larger on many tasks?

  2. Search costs may not be equal across methods or runs, meaning that the experiments may not provide a performance comparison under equal costs. In the paper, the authors fix the number of LLM calls, use the same model, set the maximum output length for program generation to 3000 tokens, and count improvement and reflection requests toward the budget. However, even with the same output limit, actual output lengths differ. The inputs also differ depending on whether a fixed reference implementation or a parent program is provided. Therefore, these settings alone do not guarantee equal total costs. In addition to comparisons with an equal number of calls, I would like the authors to verify whether the conclusions still hold when costs are equalized.

Questions for authors

(Some of my questions are already mentioned in the Weaknesses section)

Q1. In Appendix B (L642), reflection is limited to at most five bullet points and 150 words, with a maximum output length of 500 tokens. What is the rationale for choosing these constraints? (Could these constraints be limiting the performance of reflection?)

1 comment

Thank you for the review. The general response describes the new experiments and defines the seed-SD, and the revised PDF marks all changes in blue.

W1. Would the results change at 1000 or more calls? On some tasks, yes, and we ran the experiment. At 1000 calls with ten seeds on four tasks (§4.10, Table 4), FunSearch and Evolve beat sampling by 1.7 to 2.4 seed-SD on MAX-3-SAT and on C(20,6,3), Evolve also gains on C(28,6,3), all methods tie on A3(13,8), and no method loses to sampling. The gaps stay about the same size while the noise shrinks, so effects that sat at the edge of the noise at 100 calls stand clear of it at 1000. We now present the result as one about budgets of a few hundred calls, which current work does study, since ShinkaEvolve reports its circle-packing record from about 150 samples and BaSE studies 8 to 512 calls. Four chosen tasks do not locate a general boundary, and the revision says so.

W2. Are the methods compared at equal cost? Equal calls is not equal cost, and the revision adds the cost per request of every method, computed from the provider's token counts (Table 11). At 100 calls a FunSearch request costs 1.02 times a sampling request on average and 0.84 times on the nine main-grid tasks, so equal calls does not favour it there. Evolve costs 1.33 times and reflection about 1.15 times, so their ties with sampling are conservative. On the larger covering designs FunSearch costs 1.2 to 1.9 times sampling, so part of its advantage there is paid for in tokens. The final version will add a comparison at matched cost from the per-request logs.

Q1. Why the five-bullet, 150-word limit on the reflection, and does it limit performance? The limit is part of the method, which conditions later samples on a short distilled insight in place of example programs, and the 500-token cap never binds. We did not test other lengths, so the effect of the cap is untested and our results do not show that it is harmless. Two observations bear on it. The method beat sampling on cap sets and binary codes in an earlier protocol (Appendix C), and where it fails here it falls below sampling, by 15 blocks on C(24,6,3), which suggests misdirection more than too little information. Limitations now states that the cap was not ablated.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Reply to all

We thank the reviewers. We ran two new experiments and uploaded a revised PDF in which every change is printed in blue. Throughout, a seed is one independent run of a whole search and the seed-SD is the standard deviation of a method's final score across seeds. We express effect sizes in seed-SD units and report confidence intervals and minimum detectable effects to quantify uncertainty. Square brackets are 95% bootstrap intervals.

Experiment 1, a budget of 1000 calls (§4.10, Table 4). We reran sampling (best-of-N), FunSearch and Evolve at 1000 calls with ten seeds on C(20,6,3), C(28,6,3), A3(13,8) and MAX-3-SAT. Evolution now clearly beats sampling on two of the four tasks, by 4.95 to 5.85 clauses on MAX-3-SAT and 2.2 to 2.6 blocks on C(20,6,3), or 1.7 to 2.4 seed-SD. Evolve also saves 2.4 blocks [+0.4, +4.6] on C(28,6,3), all three methods tie on A3(13,8), and no method loses to sampling. The gaps barely change with the budget while the noise shrinks, since FunSearch's lead on MAX-3-SAT is 5.3 clauses at 100 calls and 5.0 at 1000 while the seed-SD falls from 5.7 to 2.4. The paper's claim is therefore one about budgets of a few hundred calls, and the revision says so, adding that four chosen tasks do not locate a general boundary.

Experiment 2, FunSearch without island resets (§4.11, Table 10). FunSearch keeps ten populations of programs, called islands, and periodically restarts the weaker half from the stronger ones. The released code does this every four hours, whereas our port did it every N/4 requests, which is three resets in a 100-call run. We reran FunSearch with resets off, at 100 calls with twenty seeds, on the five tasks where it had beaten or lost to sampling. Its four advantages remain detectable, with no detected difference between the settings, and its one loss, on the Ramsey task, reverses from 9.3 points below sampling to 6.8 above [+1.6, +11.7]. That loss was an artefact of our reset schedule. The largest gap observed at 100 calls becomes 1.15 seed-SD on these five tasks, against 1.03 across all fifteen. The other ten tasks were not rerun.

Other changes. The abstract names the methods we ran, scopes the claims to verified construction and solver tasks, reports both bounds and drops "consistent". The introduction no longer says the cross-model test isolates model strength, and Limitations adds the points raised in the reviews. Appendix B now reports the effect of counting failed runs as outcomes and lists every test set, Appendix D marks the two new runs as not pre-registered, and Appendix E adds the cost per request of each method (Table 11). A comparison at matched cost will follow in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.