- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper systematically examined whether the claimed improvements by the "LLM program discovery" search loop systems (such as FunSearch, AlphaEvolve, ShinkaEvolve, Mind Evolution, etc.) truly exist under actual budgets (approximately several hundred calls), or if they are merely seed noise. The author conducted 15 tasks with deterministic validators (11 combination coverage design instances + MAX-3-SAT + planting groups), using DeepSeek V4-Flash as the main model, and systematically compared 6 methods including the best-of-N, FunSearch, Evolve, two reflective variants, and Hedge.
Critical strengths
The system compared six methods: best-of-N, FunSearch, Evolve, two variants of reflection, and Hedge. Each method was run with 20 independent seeds (N = 100 calls) for each task, and 10 seeds were run at N = 300. Additionally, cross-model verification was conducted on four tasks using Qwen2.5-7B and gpt-4.1-mini.
Critical weaknesses
- Among the 13 main tasks, 11 are from the La Jolla coverage design library. The reward is the number of discrete blocks, the structural height being isomorphic, and there is a known platform; the software/theorems/scientific discovery tasks with a stronger "program discovery" flavor are completely absent. There is a gap between the title's "LLM Program Discovery" and the empirical evidence.
- The main conclusion "100 calls without a method exceeding best-of-N 1.03 seed-SD" is entirely from DeepSeek V4-Flash; the other two models only appeared on 4 tasks and 10 seeds. The most interesting "model moving the floor" narrative in the entire text is precisely based on this very thin cross-model data (for example, on the same task C (20,6,3), FunSearch won 4.9 blocks and Evolve lost 4.0 blocks, which itself looks like noise).
- Hedge degraded to a repeated copy of the best-of-N due to the closure of compute feedback, and the authors themselves admitted that it "operates with 80% efficacy"; Reflect₃₂ only ran on 7 tasks; FunSearch/Evolve is a lightweight implementation transplanted by the authors rather than the original version optimized in the paper. On this asymmetric grid, "33 out of 135 pairs are distinguishable", which makes it easy to confuse "our prototype is not better than the published system" with "the published system is not better than sampling".
Questions for authors
- Please re-run the tests on tasks with more continuous rewards and more noisy evaluations (such as floating-point optimization, software engineering tasks with noisy validators). In such tasks, do you expect the noise floor to be higher or lower than the coverage design? If the floor itself varies with the task form, where is the boundary of the generalizable range of the "1.03 seed-SD" number?
- If you transfer the entire 15 tasks × 20 seed grids to gpt-4.1-mini (the saturated model described in your report) and Qwen2.5-7B (the model that often fails in the initial trial program), do you expect the main conclusion "There is no method that exceeds the best-of-N 1.03 seed-SD" to be retained, expanded, or overturned? Specifically, what evidence would make you recognize that the noise floor is a model-related quantity rather than an empirical constant that can be referenced across models?
- Among the 135 method comparisons reported in your report, how many pairs were conducted between two implementations that have reached the published system optimization level? If someone uses the complete configuration of the original FunSearch/AlphaEvolve (archive selection, cluster clustering, temperature scheduling) to re-run the same grid in the future, do you expect what proportion of the effects currently classified as "noise" will become reproducible significant advantages? In other words, "Evolution is not better than sampling in 100 calls" is a conclusion about the method category or about your current implementation?
3 comments
Thanks for the response. After reading through the response, I find that my previous concerns remain unaddressed. Taking into account the comments from the other reviewers, I have decided to maintain my original score.
Thank you for reading our response. We understand that two of your concerns are not resolved by this revision, the absence of software and theorem-proving tasks and the limited cross-model evidence. The revised paper states both as limitations and scopes its claims to verified construction and solver tasks at a few hundred calls. On your question of whether the conclusion concerns the method category or our implementation, we tested the one free choice in our FunSearch port, and it changed one of the five outcomes we retested (§4.11), which is why the revision limits its conclusions to the methods as run. If there is a specific concern beyond these that you see as unaddressed, we would be glad to respond to it before the discussion closes.
bBTr
Thank you for the review. The general response describes the new experiments and defines the seed-SD, and the revised PDF marks all changes in blue.
W1. Mostly covering designs, and no software, theorem-proving or discovery tasks. Seven of the fifteen tasks are covering designs, three of them in the nine-task main grid, and the other eight are Sidon sets, LABS, Ramsey colourings, circle packing, two codes, MAX-3-SAT and planted clique (Table 1). These are the construction problems on which FunSearch, AlphaEvolve and ShinkaEvolve report results, and in each the model writes a program. We agree the other domains are absent, and the abstract and Limitations now say so.
W2. One model, and thin cross-model evidence. We agree the cross-model evidence is thin, as §6 says. The example cited combines two results from the main model. On C(20,6,3) at 300 calls both methods beat sampling, FunSearch by 4.9 blocks and Evolve by 3.4, and the 4.0-block loss is FunSearch on C(28,6,3) (§4.8). The strongest cross-model result is not marginal, since Qwen's win on C(28,6,3) reappears in 98% of random seed splits (§4.9).
W3. Hedge, Reflect32, the implementations and the grid. The 80% figure is the statistical power behind the minimum detectable effect (§3), not a description of Hedge, whose duplicate arm is disclosed in §3 and §6. Reflect32 ran on all nine main-grid tasks with twenty seeds, so the 135 pairs come from a symmetric grid, and it was Reflect16 that ran on seven tasks at 300 calls. FunSearch is a line-by-line port of the released logic with three disclosed deviations, and Evolve is a simpler second method added so the result does not rest on one implementation. Your concern about the port is fair, and we tested it (Q3).
Q1. Continuous or noisy rewards, and how far does 1.03 generalise? LABS and circle packing already have continuous scores and behave like the discrete tasks (between-to-within ratios 0.32 and 0.37), and the high-variance planted clique ties on every comparison. Our verifiers are exact, so noisy validators lie outside the suite. The 1.03 figure is an observation for one model at 100 calls, not a constant, and the new experiments show it moving, to 1.15 on five tasks without FunSearch's resets and to about two seed-SD at 1000 calls on two tasks. What we expect to transfer is the measurement, a gap in seed-SD units with its minimum detectable effect.
Q2. Would the conclusion survive on the full grid with the other models? The paper already treats the floor as depending on the model, which is its third result (§4.9). The 1.03 bound fails for Qwen, where evolution wins by 1.0 to 1.6 seed-SD on two of four tasks, and is exceeded once for gpt-4.1-mini. On a full grid we would expect it to hold where a model's sampling plateaus within 100 calls and fail elsewhere, as recorded before the runs in Appendix D. We did not run the full grid, and Limitations says so.
Q3. Category or implementation? It is a conclusion about the two implementations we ran, at 100 and 300 calls, and the revision says so. It does not cover AlphaEvolve, ShinkaEvolve or production budgets. Our FunSearch keeps the released clustering, cluster selection and temperature schedule. Your question led us to test its one free deviation, the island reset schedule. With resets off, the four advantages over sampling remain detectable with no detected difference between settings, and the Ramsey loss becomes a gain of 6.8 [+1.6, +11.7] (§4.11). One of the five results we retested therefore depended on our implementation, which is why the revision limits its conclusions to the methods as run.