- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose SpecEvo, a multi-LLM evolutionary framework that coordinates lightweight and frontier models by continually consolidating search evidence. The framework assigns three roles. Lightweight models act as parallel Speculators, which generate and refine candidate programs, and as an Advisor, which periodically distills accumulated successes, failures, and saturated directions into revisable guidance for later exploration. A frontier-model Navigator inspects the evolving population at sparse checkpoints and intervenes through synthesis, surgical refinement, or reframing, chosen according to a stagnation signal. Candidates are maintained in a MAP-Elites-style archive whose niches combine code-structure features with embeddings of the strategy descriptions. The method is evaluated on 176 tasks spanning mathematical discovery, systems optimization, combinatorial optimization (CO-Bench), and scientific equation discovery (LSR-Synth). The authors report performance comparable to or better than frontier-model baselines at up to 80% lower API cost.
Critical strengths
-
Practical and reproducible design. The framework is built from simple, well-understood components (a MAP-Elites-style archive, rank-based parent sampling, and periodic LLM-generated guidance). The paper provides complete prompts, algorithms, and hyperparameters, which should make it straightforward to reimplement. The Advisor in particular appears effective: it reduces execution errors to about 5–8%, compared with 13–25% for LEVI (Figure 9).
-
Broad evaluation. The method is tested on 176 tasks across four domains: mathematical discovery, systems optimization, combinatorial optimization (CO-Bench), and scientific equation discovery (LSR-Synth). Comparisons are run under three budget types (wall-clock time, API cost, and evaluation count) and with multiple frontier and lightweight backbones.
Critical weaknesses
imited comparison with multi-agent orchestration approaches. The baselines are either single-model evolutionary methods or model-allocation strategies (LEVI, RelayEvolve, and the controlled All-cheap, Fixed-switch, Random, and Bandit variants). All of these decide which model handles a call. None represents multi-agent orchestration, where agents with distinct roles communicate or share memory. CORAL (Qu et al., 2026), for example, is cited in Appendix B but not evaluated. Without such a comparison, it is unclear whether the gains come from the specific Speculator/Advisor/Navigator design or would also appear with a more flexible multi-agent system at similar cost. I would encourage the authors to include at least one such baseline under matched API-cost budgets.
Questions for authors
- How does SpecEvo perform in longer-horizon settings, such as much longer runs or tasks with expensive evaluations? Do the fixed stagnation horizons and the single rolling Advisor note still work there?
- How do SpecEvo’s results on the AlphaEvolve math tasks compare with the best published values?
3 comments
Response to reviewer (2)
In summary
We thank you for your time and constructive feedback. We hope our responses can sufficiently address your concern and improve your ratings. Thank you for your consideration.
Official Comment by Authors
Dear Reviewers,
We sincerely appreciate your time, thoughtful feedback, and valuable suggestions throughout the review process.
We have now posted our rebuttal and updated manuscript to address your comments and concerns. As the discussion period draws to a close, we kindly invite you to review our responses and revisions. We would be pleased to provide any further clarifications and look forward to hearing your thoughts.
Best regards,
The Authors
Response to reviewer (1)
Dear reviewer zzTq,
We deeply thank you for your time and efforts in providing constructive reviews for our paper. We are glad that you find our paper clear and easy to follow. We would like to address your concerns below
Thank you for your feedback. Following your feedback, we’ve added CORAL (Qu et al., 2026), with one and two agents that collaborate through persistent shared memory under the matched API budget ($10 per run).
SpecEvo has the highest mean score and the best single run on both tasks.. The difference lies in how the two frameworks allocate inference resources. CORAL relies on long-running coding agents that repeatedly reason, inspect experimental results, exchange knowledge through shared memory, and refine solutions, leading to substantial LLM inference and context-processing overhead. In contrast, SpecEvo prioritizes inexpensive candidate generation with only occasional frontier-model interventions. Consequently, CORAL exhausts the $10 budget in approximately one hour, whereas SpecEvo sustains exploration for 15–16 hours on average.
Matched wall-clock time (three hours). Because the matched-cost runs differ so much in duration, we also compare under the three-hour budget used in CORAL's evaluation:
SpecEvo achieves the highest mean score on both tasks, with 93.2%–95.1% lower API expenditure than two-agent CORAL. CORAL achieves the best individual result on rectangle packing, so the main finding is competitive solution quality at substantially lower cost.
Thank you for this question. We added an extended-search experiment on Polyominoes Packing from Frontier-CS, a challenging task to study long-running search. Due to resource constraint, we terminate each run after 100 consecutive evaluations without improvement or two hours without improvement, whichever occurs first. This allows search to continue while it remains productive.
SpecEvo reaches 2,551 evaluations, compared with 81–201 for the baselines, achieving scores 8.7% above CORAL (1 agent) and 14.3% above OpenEvolve at comparable or lower API cost. This sustained progress is achieved without changing the stagnation horizons or expanding the single rolling Advisor note. The horizons track evaluations since the last improvement, allowing the Navigator to respond to local stagnation throughout a longer run (18 Reframe with 72.2% lead to new best within next 20 evals). Meanwhile, approximately 50 scheduled Advisor updates refresh the note using recent progress and persistent failure evidence, with candidate solutions retained in the archive. These results support the adequacy of both mechanisms on this extended-search task.
Thank you for this suggestion. Table 12 now compares our best results with human references, AlphaEvolve, and the strongest published values. SpecEvo's reported Circle Packing best exceeds the quoted human reference of 2.634 and is close to the published AI results. Rectangle Packing remains below AlphaEvolve, with a displayed gap of 0.002. Heilbronn Triangle also remains below the reported AlphaEvolve value. The convex and MinMax results are in the same numerical range.
We do not claim new mathematical records from these rounded values. Because models, budgets, and the selection of best runs differ, this table contextualizes solution quality; the controlled experiments provide the evidence for the performance–cost comparisons.