- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EvoDuet integrates web retrieval into evaluator-driven evolutionary program search. It enables optimizers to proactively acquire external knowledge using a combination of knowledge-gap gating, iterative query refinement, search memory, and utility-driven evidence selection based on predicted downstream evaluator scores. The core research direction is important, and the proposed system demonstrates compelling empirical potential.
Critical strengths
- Clean writing and easy to understand.
- Selecting evidence based on predicted downstream task utility rather than raw semantic relevance is a well-motivated and promising design principle.
- Evaluation: The empirical evaluation spans 21 main optimization tasks across multiple models and transparently reports negative results. The method achieves substantial performance improvements on notable tasks such as Swap Reduction.
Critical weaknesses
- Overclaim: The theoretical claim of bilevel optimization is overstated. The actual inner loop relies on a frozen LLM acting as a self-predicted heuristic surrogate without formal optimization guarantees. Labeling the query loop as "co-evolution" is strong. The mechanism operates primarily as iterative query refinement or active retrieval.
- Compute not matched: EvoDuet adds up to RJ+R+2 LLM calls and up to three web searches per RETRIEVE iteration, but baselines are matched only on candidate count. Cost is reported only for Table 1a and Denoising. A baseline with matched tokens or LLM calls (e.g., more samples, self-reflection) is needed, especially since parallel generation alone helps OpenEvolve a lot (Appendix D.2).
- The key comparison against DeepEvolve is not convincing enough.
- Combining open public benchmarks with unrestricted web search blurs the line between genuine "scientific discovery" and literature-assisted solution reconstruction.
Questions for authors
- Do you have stronger compute-matched baselines, e.g., OpenEvolve with more sampling/self-reflection? It would also be useful to report score vs. tokens / LLM calls / evaluator calls.
- Do you have ablations for the utility scorer, e.g., comparing it with semantic relevance ranking, standard reranking, or random document selection?
- Do you have any evaluation on tasks where the target solution cannot be directly found or reconstructed from the public web, even on a small subset?
4 comments
Response by Authors (2/2)
Q2. Do you have ablations for the utility scorer, e.g., comparing it with semantic relevance ranking, standard reranking, or random document selection?
A. Thank you for this suggestion. We additionally evaluated random selection, semantic relevance ranking, and standard reranking with BGE on Denoising, Sums/Diffs, and Swap Reduction using GPT-5.6-Luna.
| Document selection | Denoising (PBMC/Tabula) ↑ | Sums/Diffs (c-value) ↑ | Swap (Q20 SWAPs) ↓ |
|---|---|---|---|
| Random selection | 0.6744 ± 0.0402 | 1.081573 ± 0.006919 | 18,611 ± 569 |
| Semantic relevance | 0.6276 ± 0.0293 | 1.111068 ± 0.015881 | 19,113 ± 957 |
| BGE reranker | 0.6901 ± 0.0262 | 1.079084 ± 0.010439 | 19,841 ± 226 |
| EvoDuet | 0.7019 ± 0.0201 | 1.087436 ± 0.010649 | 17,894 ± 1,579 |
As shown in the table, EvoDuet’s hypothetical evidence scoring achieves better mean performance than the alternative selection methods, except on Sums/Diffs, where semantic relevance ranking performs best.
Q3. Do you have any evaluation on tasks where the target solution cannot be directly found or reconstructed from the public web, even on a small subset?
A. We conduct an additional experiment on ConStellaration [3], a dataset and machine learning optimization benchmark for quasi-isodynamic (QI) stellarator plasma boundary shapes. No public solution is currently available. For robustness, we run the experiment three times.
| Method | ConStellaration score ↑ |
|---|---|
| OpenEvolve | 0.432046 |
| EvoDuet (n=1) | 0.435753 |
| EvoDuet (n=8) | 0.435825 |
| EvoDuet (n=16) | 0.436631 |
As shown in the table, EvoDuet outperforms OpenEvolve and exhibits method transfer. This involves exploring principles in the scientific literature and adapting them into executable code, including Fourier-mode regularization and evaluator-guided coefficient tuning. These results suggest that co-evolution between web search and solution optimization is helpful even on tasks where no public solution exists.
References:
[1] Dumouchelle, Justin, et al. "Neur2bilo: Neural bilevel optimization." Advances in Neural Information Processing Systems 37 (2024): 86688-86719.
[2] Chew, Ruth Wan Theng, Quoc Phong Nguyen, and Bryan Kian Hsiang Low. "BILBO: BILevel Bayesian Optimization." arXiv preprint arXiv:2502.02121 (2025).
[3] Cadena, Santiago, et al. "ConStellaration: A dataset of QI-like stellarator plasma boundaries and optimization benchmarks." Advances in Neural Information Processing Systems 38 (2026).
Thank you for the additional experiments. They address parts of my concerns.
Considering the contribution is primarily integrating established components, such integration can be valuable, but it would need stronger evidence than the current ablations and the modest gain over DeepEvolve provide. For example, denoising accounts for ~half of the reported average gain over OpenEvolve. But token-matched OpenEvolve reaches 0.7114 on denoising, better than the average of EvoDuet and achieves ~86% NDG. Given that standard OpenEvolve scored 0 NDG in the 21-task comparison, and the budget-matched comparisons cover only 2 tasks. I remain unsure how much of EvoDuet’s reported advantage persists under matched budgets. I am therefore maintaining my rating of 2.
Response by Authors
Thank you for clarifying your concern. We agree that two tasks are insufficient to establish a general advantage under matched token budgets. We will extend the token-matched evaluation to the remaining 19 tasks to provide broader evidence, when we acquire sufficient cost (e.g., API cost) for the experiment.
Response by Authors (1/2)
W1. Overclaim: The theoretical claim of bilevel optimization is overstated. The actual inner loop relies on a frozen LLM acting as a self-predicted heuristic surrogate without formal optimization guarantees. Labeling the query loop as "co-evolution" is strong. The mechanism operates primarily as iterative query refinement or active retrieval.
A. Thank you for highlighting this issue. We agree that describing our method as “bilevel optimization” may overstate its theoretical scope, since the inner loop relies on an LLM-predicted surrogate objective without formal optimization guarantees. We used this formulation to emphasize the mutual influence between the search and solution loops, drawing on prior work on surrogate-assisted bilevel optimization [1,2]. However, we agree that the term “bilevel” should be used more carefully. We will therefore revise the manuscript to use “co-evolution” instead, referring specifically to the interaction between query refinement and solution optimization.
(Q1 & W2). Stronger baselines under matched token budget
A. We agree that matching candidate counts alone does not account for retrieval overhead. We added OpenEvolve baselines with more sampling and self-reflection, using the same recorded LLM token caps as EvoDuet n=1 (GPT-5.6-Luna).
The table reports higher scores for EvoDuet than for OpenEvolve with more sampling or self-reflection, alongside similar token usage and fewer LLM calls.
W3. The key comparison against DeepEvolve is not convincing enough.
A. We expanded the DeepEvolve comparison to all 21 tasks using GPT-5.6-Luna. For robustness, we run three times.
In the Table, EvoDuet has the highest overall point estimate (+3.14 NDG points over DeepEvolve), with larger gains in Denoising and Swap. DeepEvolve remains competitive in mathematics and scaling laws. We will add the full results to the revised paper.
W4. Combining open public benchmarks with unrestricted web search blurs the line between genuine "scientific discovery" and literature-assisted solution reconstruction.
A. Thank you for raising this concern. The table below reports SOTA matches and improvements by observed behavior.
Although some SOTA results involve public artifact reuse or target-informed reconstruction, method transfer also appears in eight runs that exceed SOTA. Appendix I provides concrete examples of how models apply principles from the literature to revise their code, supporting the value of web search for literature-assisted optimization. However, we agree that such optimization should be distinguished from scientific discovery and will clarify this point in the revised manuscript.