- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces SeaEvo, a way to make LLM-guided evolutionary search remember the “idea” behind each program, not just the code and its score. It then uses these strategy descriptions to retrieve useful past attempts and guide the search away from directions that seem stuck.
Critical strengths
-
The main idea is simple and useful: instead of passing full programs around, SeaEvo keeps short strategy descriptions that are easier for the LLM to reuse. This also makes the search more interpretable and reduces token cost.
-
The method is modular and can be added to existing evolutionary search systems. The paper also reports some modest improvements in best scores, so the direction has some promise
Critical weaknesses
- The empirical evidence is weak. The main results use only three runs over 9 general tasks, so the reported gains are not statistically convincing.
- Many improvements are small, close to the standard deviation, or look like near-ties. This makes it hard to know whether SeaEvo is genuinely better or just benefiting from noise.
- I also have significant concerns about some of the obtained results. Indeed, the Prism task in Figure 3 lists the Average as 43.6+-29.8, and the best result as 78. This differs substantially from other entries in the table and, as confirmed by authors in the Appendix, is not statistically significant. Therefore, this entry is misleading and should not have been added to the table.
- The model choice is not well justified. The paper mainly uses two models, Gemini-3-Flash and MiMo-V2-Pro, but it is unclear why these are enough to support general claims.
Questions for authors
Could the authors clarify whether SeaEvo has been compared with newer frontier models or high-thinking configurations?
2 comments
[W4 & Q1]: Model selection rationale and evaluation with a stronger frontier model
To clarify our model selection and address whether SeaEvo remains effective with stronger frontier models, we explain the role of our primary mutation models and highlight an additional evaluation with Claude Opus 4.8 already included in Appendix D.3:
- Evaluation with a stronger frontier model: Claude Opus 4.8 (Appendix D.3, Table 14). Gemini-3-Flash and MiMo-V2-Pro were selected for the main experiments to provide broad multi-task and multi-backbone coverage within our experimental budget. We additionally evaluate Claude Opus 4.8 as a third mutation model on TXN, using identical initialization, evaluator, and 100-generation search budgets, with three independent runs. The results demonstrate improvements across both evolutionary backbones:
-
ShinkaEvolve: improves from to with SeaEvo (individual run scores: 4065.04, 4255.32, 3937.01 vs. 4255.32, 4219.41, 4115.23).
-
OpenEvolve: improves from to with SeaEvo (individual run scores: 3861.00, 4098.36, 3816.79 vs. 4237.29, 4237.29, 4132.23).
- Scope across model tiers and reasoning configurations. With Claude Opus 4.8, SeaEvo improves the mean score across both evolutionary backbones and reduces the observed run-to-run standard deviation by more than half in these three-run comparisons.
We will move the Claude Opus 4.8 results from Appendix D.3 into a more prominent position in the main text, providing additional evidence that persistent strategy-state organization remains effective when paired with a stronger frontier mutation model.
We thank the reviewer again for highlighting these important limitations. We will revise the manuscript to present a more careful account of statistical uncertainty, evaluator reliability, and model coverage, while making the evidence for SeaEvo's strategy-level contribution more explicit.
Thank you for recognizing the simplicity, interpretability, and practical value of SeaEvo's strategy-level representation. Below we address your Weaknesses and Questions.
We agree that with three runs per configuration on the main benchmarks, several task-level margins are small relative to run-to-run variance, and that descriptive mean gains must be clearly distinguished from statistically significant improvements. We therefore clarify the statistical scope of our results and highlight complementary controlled analyses that provide additional evidence for SeaEvo's mechanism:
Explicit qualification of aggregate mean comparisons. The original 23/24 summary (and 19/20 after excluding Prism and EPLB, as discussed below) counts configurations with a higher observed sample mean rather than statistically established per-task wins. We will explicitly clarify this distinction in the abstract and main text, and avoid presenting near-ties as decisive improvements.
Matched 100-state retrieval control (Table 12). To examine whether semantic retrieval provides a genuine mutation-level benefit beyond stochastic variation across complete evolutionary runs, Table 12 conducts a controlled comparison across 100 matched search states on Circle Packing, sampled across three independent trajectories. Holding the parent program, SLN navigation guidance, model configuration, reference count (2), and inference call count (1) fixed, SER's semantic retrieval outperforms uniformly random strategy retrieval with 56 wins, 34 losses, and 10 ties (p=0.026, sign test). This provides a paired, single-step counterfactual analysis of retrieval effectiveness under matched inference budgets, complementary to the full-run evaluations.
Variance reduction in four-run ablations (Table 3) and association-disruption controls (Table 4). In Table 3, our audit confirmed that seven configurations aggregate n=4 runs and SER+SLN aggregates n=3 runs; the corresponding raw run-level results are provided in our response to Reviewer 7cVZ. Full SeaEvo achieves 2.6297±0.0077 across four runs, with individual run-level best scores of 2.6343, 2.6186, 2.6305, and 2.6353. Its sample standard deviation is approximately an order of magnitude smaller than those of all non-empty ablated variants (0.0724–0.0855), each of which has at least one run whose best-so-far score falls between 2.44 and 2.51. Moreover, Table 4 shows that disrupting program–strategy associations (SER-Permute) or strategy–fitness associations (SLN-Shuffle) on TXN reduces the mean score from 3886.1 to 3340.1 and 3538.5, respectively.
While we carefully qualify the statistical scope of the three-run benchmark averages, the matched retrieval comparison (Table 12), the substantially lower run-to-run variance in the Full configuration (Table 3), and the association-disruption controls (Table 4) provide complementary evidence supporting the mechanism-level contribution of structured strategy reuse beyond descriptive mean comparisons across a limited number of independent runs.
We fully agree with your critique of the Prism result (43.6±29.8, Best 78): the variance is exceedingly large and, as noted in Appendix D.8, the difference is not statistically significant.
[1] Ao Qu et al., CORAL: Towards Autonomous Multi-Agent Evolution for Open-Ended Discovery, arXiv:2604.01658, 2026.