- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper compares 30 search configurations across 4 LLMs and 3 math tasks, using over 3.1 million candidate generations. It reports that:
- No fixed configuration shows a statistically reliable overall advantage over a simple baseline.
- Complex OpenEvolve-style configurations often perform worse than simpler methods.
- Early scores help identify promising runs. Simulations suggest that stopping weak runs and continuing strong runs improves results under equal generation budgets.
The main message is that search settings should depend on the model and problem. The study treats the LLM as a candidate proposer; it does not evaluate an interactive agent within each proposal.
Critical strengths
The fact that single-turn llm proposer auto-discovery has no optimal harness reveals a fundamental instability in this auto-discovery approach. This makes a strong case for shifting toward mechanisms that encourage automated agents to carry out the discovery process, instead of leveraging fixed workflows.
Critical weaknesses
The title is misleading. The authors define automated discovery as a family of workflows following OpenEvolve style or TTT-Discover style, where each proposal is generated by a single LLM run. Many more frontier auto-discovery works have used agentic proposers, including CORAL, Co-Scientist, etc..
Questions for authors
- It is pretty counter-intuitive to see that OpenEvolve perform worse than baseline sequential BoN as OpenEvolve is quite a popular method in auto-discovery. Does OpenEvolve generate weaker ideas, invalid code, or proposals that exceed execution limits? Does extra context reduce proposal quality? A failure analysis would make the result much more useful.
- Is it possible to also include analysis including agentic proposers?
Thank you for your review and feedback on our paper. We address your comments below.
W1
CORAL and Co-Scientist change how candidate ideas are produced. Our study focuses on the search components used to select and develop the candidates. We take apart the popular OpenEvolve-style and TTT-Discover-style harnesses and recombine their components to compare the strength of different combinations while keeping proposal generation fixed. With these comparisons, we are able to test if the search components themselves work consistently across model/task pairs. No single combination of harness components performed the best throughout all the pairs we tested. These results show how the tested combinations of search components perform across our model/task pairs, which can help guide future discovery systems, including ones with agentic proposers.
Q1
We audited three progressively configured OpenEvolve variants on Heilbronn with Qwen2.5-3B: one island without inspiration sampling, one island with the regular prompt, and four islands with the regular prompt. Each variant has five archived runs of 1,600 iterations.
Q2
The addition of an agentic proposer would call for experiments that require a lot of compute. For this paper, we will keep the proposal mechanism constant so that we can focus on the search heuristics.