acceptodds
Under review as a conference paper at ICLR 2027

SpecEvo: Speculative Evolution with Large Language Models for Cost-Efficient Scientific Discovery

Abstract

LLM-driven evolutionary search is a promising paradigm for scientific discovery, but existing methods remain costly and slow, often applying expensive reasoning uniformly while underusing evidence from failed, redundant, and saturated trials. We introduce Speculative Evolution (SpecEvo), a multi-LLM framework that coordinates lightweight and frontier models through continual consolidation of search evidence. Lightweight models power both parallel Speculators, which generate and refine candidate programs, and an Advisor, which distills accumulated successes, failures, and saturated directions into persistent guidance for subsequent exploration. At sparse checkpoints, a frontier-model Navigator inspects the evolving population and intervenes through synthesis, surgical refinement, or reframing according to search stagnation. This speculate-then-consolidate workflow delegates routine exploration and evidence consolidation to inexpensive models while reserving frontier reasoning for strategic interventions whose benefits extend across many subsequent low-cost evaluations. Across 176 tasks spanning four scientific and algorithmic discovery domains, SpecEvo matches or outperforms frontier-model baselines while reducing API cost by up to 80%, achieves up to 1.5× greater evaluation efficiency than cost-efficient baselines on equation discovery, and improves over the strongest CO-Bench baseline by 3.8%. These results support trajectory-conditioned model coordination as an effective approach to cost-efficient scientific discovery.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agent, scientific discovery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zzTqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose SpecEvo, a multi-LLM evolutionary framework that coordinates lightweight and frontier models by continually consolidating search evidence. The framework assigns three roles. Lightweight models act as parallel Speculators, which generate and refine candidate programs, and as an Advisor, which periodically distills accumulated successes, failures, and saturated directions into revisable guidance for later exploration. A frontier-model Navigator inspects the evolving population at sparse checkpoints and intervenes through synthesis, surgical refinement, or reframing, chosen according to a stagnation signal. Candidates are maintained in a MAP-Elites-style archive whose niches combine code-structure features with embeddings of the strategy descriptions. The method is evaluated on 176 tasks spanning mathematical discovery, systems optimization, combinatorial optimization (CO-Bench), and scientific equation discovery (LSR-Synth). The authors report performance comparable to or better than frontier-model baselines at up to 80% lower API cost.

Critical strengths
  • Practical and reproducible design. The framework is built from simple, well-understood components (a MAP-Elites-style archive, rank-based parent sampling, and periodic LLM-generated guidance). The paper provides complete prompts, algorithms, and hyperparameters, which should make it straightforward to reimplement. The Advisor in particular appears effective: it reduces execution errors to about 5–8%, compared with 13–25% for LEVI (Figure 9).

  • Broad evaluation. The method is tested on 176 tasks across four domains: mathematical discovery, systems optimization, combinatorial optimization (CO-Bench), and scientific equation discovery (LSR-Synth). Comparisons are run under three budget types (wall-clock time, API cost, and evaluation count) and with multiple frontier and lightweight backbones.

Critical weaknesses

imited comparison with multi-agent orchestration approaches. The baselines are either single-model evolutionary methods or model-allocation strategies (LEVI, RelayEvolve, and the controlled All-cheap, Fixed-switch, Random, and Bandit variants). All of these decide which model handles a call. None represents multi-agent orchestration, where agents with distinct roles communicate or share memory. CORAL (Qu et al., 2026), for example, is cited in Appendix B but not evaluated. Without such a comparison, it is unclear whether the gains come from the specific Speculator/Advisor/Navigator design or would also appear with a more flexible multi-agent system at similar cost. I would encourage the authors to include at least one such baseline under matched API-cost budgets.

Questions for authors
  • How does SpecEvo perform in longer-horizon settings, such as much longer runs or tasks with expensive evaluations? Do the fixed stagnation horizons and the single rolling Advisor note still work there?
  • How do SpecEvo’s results on the AlphaEvolve math tasks compare with the best published values?
3 comments
AuthorsOpenReview ↗

Response to reviewer (1)

Dear reviewer zzTq,

We deeply thank you for your time and efforts in providing constructive reviews for our paper. We are glad that you find our paper clear and easy to follow. We would like to address your concerns below

Comparison with multi-agent orchestration

Thank you for your feedback. Following your feedback, we’ve added CORAL (Qu et al., 2026), with one and two agents that collaborate through persistent shared memory under the matched API budget ($10 per run).

TaskMethodScore ↑Best ↑Evaluations
Circle PackingCORAL (1 agent)2.5549 ± 0.02552.590556
CORAL (2 agents)2.5409 ± 0.01152.552348
SpecEvo (full)2.6283 ± 0.00582.63641387
Circle Packing RectCORAL (1 agent)2.3033 ± 0.02062.332467
CORAL (2 agents)2.3105 ± 0.01452.326755
SpecEvo (full)2.3595 ± 0.00412.36422024

SpecEvo has the highest mean score and the best single run on both tasks.. The difference lies in how the two frameworks allocate inference resources. CORAL relies on long-running coding agents that repeatedly reason, inspect experimental results, exchange knowledge through shared memory, and refine solutions, leading to substantial LLM inference and context-processing overhead. In contrast, SpecEvo prioritizes inexpensive candidate generation with only occasional frontier-model interventions. Consequently, CORAL exhausts the $10 budget in approximately one hour, whereas SpecEvo sustains exploration for 15–16 hours on average.

Matched wall-clock time (three hours). Because the matched-cost runs differ so much in duration, we also compare under the three-hour budget used in CORAL's evaluation:

TaskMethodScore ↑Best ↑EvaluationsAPI cost
Circle PackingCORAL (1 agent)2.4927 ± 0.00102.493433$6.85
CORAL (2 agents)2.6200 ± 0.00012.6201100$33.82
SpecEvo (full)2.6323 ± 0.00422.6361277$1.66
Circle Packing RectCORAL (1 agent)2.3551 ± 0.00832.364287$23.96
CORAL (2 agents)2.3484 ± 0.00592.355281$30.20
SpecEvo (full)2.3584 ± 0.00032.3586417$2.04

SpecEvo achieves the highest mean score on both tasks, with 93.2%–95.1% lower API expenditure than two-agent CORAL. CORAL achieves the best individual result on rectangle packing, so the main finding is competitive solution quality at substantially lower cost.

Longer-horizon search, expensive evaluations, fixed stagnation horizons, and the rolling Advisor note

Thank you for this question. We added an extended-search experiment on Polyominoes Packing from Frontier-CS, a challenging task to study long-running search. Due to resource constraint, we terminate each run after 100 consecutive evaluations without improvement or two hours without improvement, whichever occurs first. This allows search to continue while it remains productive.

MethodScore ↑EvaluationsAPI cost (USD) ↓
OpenEvolve63.1426838.29
GEPA60.26388511.00
AdaEvolve62.3081819.75
EvoX60.286110613.29
CORAL (1 agent)66.357310131.64
CORAL (2 agents)65.549920160.18
SpecEvo72.14152,55110.42

SpecEvo reaches 2,551 evaluations, compared with 81–201 for the baselines, achieving scores 8.7% above CORAL (1 agent) and 14.3% above OpenEvolve at comparable or lower API cost. This sustained progress is achieved without changing the stagnation horizons or expanding the single rolling Advisor note. The horizons track evaluations since the last improvement, allowing the Navigator to respond to local stagnation throughout a longer run (18 Reframe with 72.2% lead to new best within next 20 evals). Meanwhile, approximately 50 scheduled Advisor updates refresh the note using recent progress and persistent failure evidence, with candidate solutions retained in the archive. These results support the adequacy of both mechanisms on this extended-search task.

Compare with the best published values?

Thank you for this suggestion. Table 12 now compares our best results with human references, AlphaEvolve, and the strongest published values. SpecEvo's reported Circle Packing best exceeds the quoted human reference of 2.634 and is close to the published AI results. Rectangle Packing remains below AlphaEvolve, with a displayed gap of 0.002. Heilbronn Triangle also remains below the reported AlphaEvolve value. The convex and MinMax results are in the same numerical range.

We do not claim new mathematical records from these rounded values. Because models, budgets, and the selection of best runs differ, this table contextualizes solution quality; the controlled experiments provide the evidence for the performance–cost comparisons.

AuthorsOpenReview ↗

Response to reviewer (2)

In summary

We thank you for your time and constructive feedback. We hope our responses can sufficiently address your concern and improve your ratings. Thank you for your consideration.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers,

We sincerely appreciate your time, thoughtful feedback, and valuable suggestions throughout the review process.

We have now posted our rebuttal and updated manuscript to address your comments and concerns. As the discussion period draws to a close, we kindly invite you to review our responses and revisions. We would be pleased to provide any further clarifications and look forward to hearing your thoughts.

Best regards,
The Authors

Reviewer_aUorOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper describes SpecEvo, a framework that decreases the API cost of LLM-driven evolutionary program search. Lightweight models (Qwen3-30B) operate as Speculators and Advisor. The Speculators make new programs from a MAP-Elites archive. The Advisor writes a short guidance note from the search results and errors. A frontier model (GPT-5) operates as the Navigator at sparse checkpoints. It shows how to combine the small and large LLMs together to search.

Critical strengths
  1. Cost efficiency. SpecEvo gives equal or better results with much less API cost. With a budget of 500 calls, GPT-5 makes only approximately 2.4% of the calls.

  2. Clear division of work. Lightweight models do most of the work. The expensive frontier model operates only at important points. This design is simple and easy to use.

Critical weaknesses
  1. Limited novelty. SpecEvo mainly combines methods from earlier work. Earlier work already uses each part of SpecEvo:
  • MAP-Elites archive: OpenEvolve, LEVI
  • Changes to the search when it does not improve: AdaEvolve
  • Guidance from the search history: ReEvo, AdaEvolve
  • Different roles for lightweight and frontier models: LEVI, RelayEvolve
  1. The effect of frontier model(GPT-5) is not clear. The paper does not separate the effect of GPT-5 from the effect of the other parts. Only 15.5% of the new best scores are from GPT-5 programs or their direct child programs (Figure 7). In this study, some Navigator variants are worse than no Navigator(Table 3).
Questions for authors
  1. Can the authors give statistics about the Navigator calls? For example:
  • the number of Navigator calls in each run;
  • the percentage of valid Navigator programs;
  • for each mode, the percentage of calls that give a new best score within the next few evaluations;
  • the mean score increase for each mode.
  1. Can the authors replace GPT-5 with the lightweight model in the Navigator, with the same prompts? This test will show if the better results are from GPT-5 or from the prompts of the three modes.
4 comments
AuthorsOpenReview ↗

Response to reviewer (1)

Dear reviewer aUor

We deeply thank you for your time and efforts in providing constructive reviews for our paper. We are glad that you find our paper clear and easy to follow. We would like to address your concerns below

W1. Limited novelty

Thank you for your feedback. We agree that each ingredient you list has precedent, and Appendix B discusses these works. Our contribution is a coordination policy that continually uses search evidence to revise lightweight-model guidance and select the scope of sparse frontier interventions. The key question is therefore not only how many calls go to each model, but how accumulated progress, saturation, and failure information should shape subsequent search.

Several of these components already appear in SOTA baselines such as EvoX, AdaEvolve, and OpenEvolve. However, these frameworks primarily rely on a single LLM, often requiring a frontier model to achieve strong performance (as illustrated in Figure 1, where replacing it with smaller models substantially degrades performance). This makes it difficult to disentangle the benefits of the search framework from those of the underlying LLM's capabilities, while also creating a cost barrier to reproducing the results. Closely related baselines such as RelayEvolve and LEVI primarily focus on model allocation without explicitly leveraging these evolving search dynamics. Our distinction is not whether search feedback is used, but which evidence is consolidated and how it shapes subsequent lightweight guidance and frontier interventions. This enables SpecEvo to achieve competitive performance against SOTA baselines across multiple budget settings at low API cost.

Our additional controls test this distinction: random and fixed Navigator modes test state-dependent intervention selection; an error-only Advisor tests guidance beyond error prevention (Extended ablation study on Table 20); and component transfers to LEVI test whether individual additions recover the full framework's gains (Table 21). These experiments reveal two key benefits of SpecEvo:

  1. How frontier calls are used matters more than how many are made. Under 500 calls, SpecEvo uses fewer frontier calls than LEVI yet achieves a higher mean score on all six tasks (Table 1). It maintains the highest mean on all six tasks when every method receives the same 10% frontier-call share (Table 13). Within SpecEvo, randomly selecting the Navigator mode reduces the mean score on all six tasks in the extended ablation, while the full policy outperforms the best fixed mode on every task (Table 20).

  2. Lightweight search is hindered by recurring errors and repeated exploration of saturated directions, but consolidating this evidence is inexpensive. The Advisor maintains execution error rates of approximately 5–8%, compared with 13–25% for LEVI (Figure 9). An error-only Advisor recovers 36–62% of the full Advisor's benefit on the five tasks where removing the Advisor causes a performance drop greater than 1%.

W2. The effect of GPT-5 is unclear, and some Navigator variants are worse than no Navigator

We agree that an inappropriate frontier intervention can be worse than no intervention, and our extended ablation confirms it (Table 20): random mode selection falls below the no-Navigator variant on four of six tasks, and individual fixed modes do so on some tasks (e.g., Surgical-only on Heilbronn Triangle, −12.47% vs. −9.42% without the Navigator). Our claim is therefore not that any Navigator call is beneficial, but that its intervention scope should depend on the search state.

The fixed- and random-mode ablations test this distinction: the state-dependent policy has the highest mean on all six tasks, above random selection and above every fixed mode. A fixed-dollar budget-reallocation experiment tests whether the Navigator's budget would be better spent on additional lightweight exploration: under $3 per run, full SpecEvo scores 2.49% and 0.27% higher than a no-Navigator variant that spends the same budget on more Speculator calls with the Advisor retained.

ConfigurationCircle Packing ↑Circle Packing Rectangle ↑
SpecEvo (full)2.6207 ± 0.00012.3586 ± 0.0001
w/o Navigator2.5570 ± 0.00102.3522 ± 0.0011
w/o Navigator + Advisor every 10 evaluations2.4836 ± 0.00532.2506 ± 0.0001
w/o Navigator + GPT-5 Advisor2.4766 ± 0.00102.3481 ± 0.0019
w/o Navigator + GPT-5 Advisor + 10% GPT-5 Speculator calls2.6179 ± 0.00012.2587 ± 0.0001

The statistic on Figure 7 (15.5%) counts new-best events, not their improvement magnitude or the counterfactual effect of removing the Navigator, and it excludes initialization (Appendix E.4). It also credits a new best to the frontier model only when the program comes from the Navigator or has a Navigator-generated direct parent; later descendants are not counted. We therefore interpret it together with the intervention statistics and the budget-reallocation results.

AuthorsOpenReview ↗

Response to reviewer (2)

We address the questions in ‘Questions for Authors’ below:

Q1. Navigator call statistics

Thank you for this suggestion. We now report detailed statistics over 18 SpecEvo runs with a GPT-5 Navigator, covering invocation frequency, program validity, subsequent improvements, and gains for each mode.

Navigator modeCalls: total (mean per run)Valid programsNew best within 5 evalsNew best within 10 evalsMean gain: all callsMean gain: new best found
Synthesis71 (3.9)94.4%38.0%42.3%+0.23%+0.55%
Surgical16 (0.9)93.8%18.8%25.0%+0.28%+1.12%
Reframe43 (2.4)79.1%14.0%16.3%+0.76%+4.66%
All130 (7.2)89.2%27.7%31.5%+0.41%+1.30%

The breakdown reveals distinct patterns across the three modes:

  • Synthesis is the most frequently used mode and has the highest validity and subsequent new-best rates. Improvements follow 42.3% of calls within 10 evaluations, with relatively small conditional gains (+0.55%).
  • Surgical is used least often. A new best follows 25.0% of calls within 10 evaluations, with a larger conditional gain of +1.12%.
  • Reframe is invoked under severe stagnation to propose a new strategy family. Its validity and subsequent new-best rates are lower, but it has the largest gains: +0.76% across all calls and +4.66% when a new best is found. This pattern is consistent with less frequent but larger improvements following broader changes in search strategy.

Q2. Can the authors replace GPT-5 with the lightweight model in the Navigator, with the same prompts?

Thank you for this suggestion. We conduct the requested replacement using Qwen3-30B with identical prompts for all three Navigator modes, and also compare GPT-5 at low and medium reasoning effort.

TaskNavigatorScoreEvalsTotal costValid programsNew best within 5 evals
Circle packingGPT-5, low2.6188 ± 0.0009377$2.02100%67%
GPT-5, medium2.6321 ± 0.0016398$1.9490%45%
Qwen3-30B2.5283 ± 0.0217166$0.93100%0%
Circle packing rectangleGPT-5, low2.1993 ± 0.0090202$1.1383%35%
GPT-5, medium2.3165 ± 0.0001269$1.8671%14%
Qwen3-30B2.2653 ± 0.0056415$1.487%0%

The lightweight Navigator does not recover GPT-5 medium’s performance with the same prompts: its scores are lower by 0.1038 on circle packing and 0.0512 on rectangle packing. The intervention statistics reveal two distinct limitations. On circle packing, Qwen3-30B produces valid programs, but none of its calls is followed by a new best within five evaluations. On rectangle packing, 14 of its 15 programs are invalid, with failures including overlapping circles, perimeter violations, runtime errors, and missing code. Its final score remains lower despite completing 415 evaluations versus 269 for GPT-5 medium.

These observations are consistent with the intended division of work in SpecEvo. The Navigator must propose useful interventions when the current search struggles to improve, including changes of strategy under severe stagnation. Such interventions require both a promising algorithmic idea and a feasible implementation that subsequent cheap-model exploration can build upon. The three-mode prompts specify the intervention to attempt; model capability affects whether that intervention can be realized effectively. The results therefore support selectively allocating a stronger model to this role.

In summary

We thank you for your time and constructive feedback. We hope our responses can sufficiently address your concern and improve your ratings. Thank you for your consideration.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewers,

We sincerely appreciate your time, thoughtful feedback, and valuable suggestions throughout the review process.

We have now posted our rebuttal and updated manuscript to address your comments and concerns. As the discussion period draws to a close, we kindly invite you to review our responses and revisions. We would be pleased to provide any further clarifications and look forward to hearing your thoughts.

Best regards,
The Authors

Thanks for the response and additional experiments. This address my concerns. I would keep my supportive score.

Reviewer_LSt7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

Summary

SpecEvo makes evolutionary search cheaper by combining lightweight exploration, reusable feedback, and occasional stronger-model interventions. The efficiency gains are useful, but I’m less convinced about the novelty beyond a well-designed inference-time search framework.

Critical strengths
  • Addresses a practical problem of reducing evolutionary search cost without sacrificing solution quality.
  • Broad evaluation across datasets from different domains, with multiple budget settings and relevant baselines.
Critical weaknesses
  • The contribution feels incremental over existing work (eg [1] , [2]). What is the key new insight beyond combining these components and getting better search efficiency?

  • The cost claim raised in paper is in my opinion inflated. The 50–80% savings reported in cost by authors are against frontier-only baselines, which is expected by design. Against the closer competitor baselines (like RelayEvovle [1] , LEVI [2] or AdaEvolve) , costs are about equal and quality gaps are small based on the current reported results in the paper. I would like to haer authors' thoughts on this point.

  • The speculative-decoding connection feels mostly motivational, there is no analogous verification or distribution-preservation mechanism. I would frame this more directly as a cost-efficient inference-time search framework. Being an inference framework is fine, the key question in my opinion is whether the added complexity provides enough benefit over simpler alternatives which I dont think is studied well in the current version of paper.

  • Section 4.1 says SpecEvo is best in all 8 LSR-Synth settings, but the appendix says 6 of 8, with LLM-SR better on Biology. Which one is correct?

  • In Table 1, several allocation controls use roughly 50% frontier calls, versus 2.4% for SpecEvo. These results are useful, but a control matching the frontier-call share, initialization, and search components would better isolate the benefit of the proposed coordination policy.

  • Table 3 tests component ablations on only two mathematical tasks and reports relative drops without uncertainty. Some changes are below 1%, making their importance hard to judge.

  • Also I wonder how much does the Advisor gain from strategic guidance versus simply reminding models about feasibility and coding requirements? An error-only Advisor might help to separate these effects. Would adding the Advisor and adaptive intervention modes to earlier works (eg [2]) recover the observed improvements in terms of efficiency?

[1] Relay, Don't Route: Adaptive Population Handoff for Cost-Efficient LLM-Driven Evolution, 2026

[2] LEVI: Stronger Search Architectures Can Substitute for Larger LLMs in Evolutionary Search, 2026

Questions for authors

questions below and check weaknesses section:

  • I wonder is the Navigator worth it in the design of your framework? What happens if its dollar budget goes to more cheap calls plus the Advisor instead?

  • I'm concerned that the wall-clock comparison is not fair, given that some baselines run sequentially against 4 parallel workers? Can you clarify this? Maybe better to provide cost comparisons in terms of tokens instead of time and samples ?

5 comments
AuthorsOpenReview ↗

Response to reviewer (1)

Dear reviewer LSt7,

We deeply thank you for your time and efforts in providing constructive reviews for our paper. We would like to address your concerns below, and our updated changes in the paper are in blue.


W1. What is the key insight beyond combining existing components?

Thank you for your question. We agree that MAP-Elites archives, reflection, and heterogeneous model use are not individually new. Our contribution is a coordination policy that continually uses search evidence to revise lightweight-model guidance and select the scope of sparse frontier interventions. The key question is therefore not only how many calls go to each model, but how accumulated progress, saturation, and failure information should shape subsequent search.

As discussed in Appendix B, several of these components already appear in SOTA baselines such as EvoX, AdaEvolve, and OpenEvolve. However, these frameworks primarily rely on a single LLM, often requiring a frontier model to achieve strong performance (as illustrated in Figure 1, where replacing it with smaller models substantially degrades performance). This makes it difficult to disentangle the benefits of the search framework from those of the underlying LLM's capabilities, while also creating a cost barrier to reproducing the results. Closely related baselines such as RelayEvolve and LEVI primarily focus on model allocation without explicitly leveraging these evolving search dynamics. Our distinction is not whether search feedback is used, but which evidence is consolidated and how it shapes subsequent lightweight guidance and frontier interventions. This enables SpecEvo to achieve competitive performance against SOTA baselines across multiple budget settings at low API cost.

Specifically, our additional controls test this distinction: random and fixed Navigator modes evaluate state-dependent intervention selection; an error-only Advisor tests guidance beyond error prevention; and component transfers to LEVI examine whether individual additions can recover the full framework's gains (W5–W7). These experiments reveal two key benefits of SpecEvo:

  1. How frontier calls are used matters more than how many are made. Under 500 calls, SpecEvo uses fewer frontier calls than LEVI yet achieves a higher mean score on all six tasks (Table 1). It maintains the highest mean on all six tasks when every method receives the same 10% frontier-call share (Table 13). Within SpecEvo, randomly selecting the Navigator mode reduces the mean score on all six tasks in the extended ablation, while the full policy outperforms the best fixed mode on every task (Table 20).

  2. Lightweight search is hindered by recurring errors and repeated exploration of saturated directions, but consolidating this evidence is inexpensive. The Advisor maintains execution error rates of approximately 5–8%, compared with 13–25% for LEVI (Figure 9). An error-only Advisor recovers 36–62% of the full Advisor's benefit on the five tasks where removing the Advisor causes a performance drop greater than 1%.


W2. Scope of the cost claim and gains over close competitors

Thank you for your feedback. We would like to respectfully clarify that our reported 50–80% reduction in API costs is measured only relative to frontier-heavy configurations, not against all baselines. Our main finding is that SpecEvo matches or surpasses state-of-the-art baselines at substantially lower cost, while delivering better solution quality than cost-efficient alternatives at comparable API expenditure.

The paired analyses (Appendices G.3–G.4) confirm that these improvements extend beyond selected examples. On CO-Bench, SpecEvo achieves 21 wins, 7 ties, and 8 losses against LEVI, with a mean normalized-score improvement of +0.0332 (bootstrap 95% CI: [+0.0070, +0.0668]). On LSR-Synth, under 500 evaluations, it achieves lower ID NMSE on 98/129 problems against LEVI and 101/129 against RelayEvolve. On Transaction, SpecEvo achieves a 10.5% higher mean score than LEVI at nearly identical API cost.

Importantly, API costs differ across baselines (Table 1), with SpecEvo's expenditure primarily comparable to LEVI and All-cheap. To further control for model allocation, Table 13 introduces a matched setting with 500 total calls and a fixed 10% frontier-call share across methods. SpecEvo retains the strongest performance across all six tasks, while competing methods exhibit substantial performance degradation.

We therefore emphasize two complementary findings: substantial API savings relative to frontier-heavy approaches, and superior solution quality and search efficiency relative to cost-efficient competitors.

AuthorsOpenReview ↗

Response to reviewer (2)

W3. Speculative-decoding analogy and added complexity.

Thank you for your feedback. We agree that SpecEvo is a cost-efficient inference-time evolutionary search framework. The speculative-decoding connection motivates asymmetric computation; it does not imply distribution preservation. Our evaluator checks validity and fitness, rather than implementing distribution-correcting verification. We’ll clarify this boundary in Section 3.1 and the introduction.

The added complexity is evaluated against simpler alternatives through the expanded policy ablations in W6 and the fixed-dollar budget-reallocation experiment in Q1


W4. Eight versus six LSR-Synth settings

Thank you for identifying the ambiguous wording. SpecEvo achieves the lowest NMSE in all eight settings among the evolutionary baselines in Table 2. When the specialized equation-discovery methods in Table 19 are included, it achieves the lowest NMSE in six of eight settings; LLM-SR performs better on Biology ID and OOD.


W5. Matching frontier-call share, initialization, and search components

Thank you for your feedback. We agree that Table 1 alone cannot isolate the coordination policy. Tables 13–15 therefore add three complementary controls. Table 13 matches all methods at 500 model calls and 10% frontier-call share. Table 14 uses the same algorithmic seeds, with 500 calls excluding initialization. Table 15 gives all baselines SpecEvo’s population-management and search mechanisms, including embeddings, niche-based archive management, periodic refitting, and candidate selection. We summarise the results below:

ControlMethodCircle Packing ↑Signal Processing ↑
Frontier share (Table 13)RelayEvolve2.483 ± 0.0210.563 ± 0.011
Fixed-switch2.521 ± 0.0130.623 ± 0.068
Random2.511 ± 0.0120.632 ± 0.041
Initialization (Table 14)RelayEvolve2.491 ± 0.0240.574 ± 0.014
Fixed-switch2.526 ± 0.0160.621 ± 0.032
Random2.508 ± 0.0150.642 ± 0.029
Population management (Table 15)RelayEvolve2.503 ± 0.0210.653 ± 0.021
Fixed-switch2.557 ± 0.0320.659 ± 0.031
Random2.518 ± 0.0180.634 ± 0.025
All three controlsSpecEvo2.615 ± 0.0210.669 ± 0.013

SpecEvo achieves the highest mean score on all six tasks in each comparison, although some margins are small relative to run-to-run variation. These results support a contribution from its coordination design beyond frontier-call availability, initialization, or population management alone.

AuthorsOpenReview ↗

Response to reviewer (3)

W6. Broader ablation coverage

Thank you for your feedback. We agree that relative performance drops on only two mathematical tasks were insufficient to assess the generality and magnitude of individual component effects. Table 20 (Appendix G.5) therefore extends the ablation study to six tasks across three domains: Mathematics, Systems, and Combinatorial Optimization. The extended table reports absolute mean scores, standard deviations across three runs, and relative changes.

We summarize some representative results below:

ConfigurationLLM-SQLTransactionAssortment
Full SpecEvo0.7327 ± 0.00304219.60 ± 0.010.9745 ± 0.1020
Random Navigator mode0.7177 ± 0.00284115.38 ± 42.100.7342 ± 0.0389
Without Navigator0.7230 ± 0.00014087.76 ± 58.800.8229 ± 0.0411
Without Advisor0.6991 ± 0.00073984.61 ± 57.560.3190 ± 0.0003
Fixed Advisor guidance0.7041 ± 0.00224020.43 ± 47.360.4632 ± 0.0151
LEVI + Adaptive intervention modes0.7117 ± 0.00154075.71 ± 38.420.7067 ± 0.0518
LEVI + Advisor0.7158 ± 0.00094098.92 ± 35.160.6653 ± 0.0427

First, the additional tasks reveal substantial effects beyond the small performance drops observed in the original table. Removing the Advisor reduces the mean score by 5.6% on Transaction and 67.3% on Assortment. Fixed guidance also results in a substantial performance gap: although it retains an advice note, it reduces the Assortment mean score by more than half. These findings support the importance of continually updating guidance rather than simply providing static advice.

Second, the Navigator comparisons examine how frontier interventions are selected. The random-mode variant retains the Navigator and its available intervention types but removes state-dependent selection. Its mean performance is lower across all six tasks. Thus, merely providing multiple intervention types does not recover the performance of the full configuration.

The fixed-mode comparisons further strengthen this finding. The strongest fixed mode varies across tasks: Synthesis on Heilbronn Triangle, Surgical on LLM-SQL, and Reframe on Assortment. Even when compared against the strongest fixed mode for each task, the full configuration achieves a higher mean score on all six tasks. On Assortment, its mean score is 24.7% higher than that of the strongest fixed mode, Reframe. Together, the random- and fixed-mode comparisons support the benefit of adapting intervention scope to search progress.

We agree that isolated sub-1% performance drops do not constitute decisive evidence. The extended analysis instead emphasizes task-dependent effect sizes, reported variability, and consistent performance patterns across policy comparisons.


W7. Strategic guidance versus error prevention, and transfer to LEVI

Thank you for your feedback. We had already evaluated fixed guidance (e.g., predefined coding requirements) in Table 3 and observed performance degradation. To further address your concern, we introduce an error-only Advisor in Table 20, which excludes strategic guidance on productive mechanisms and saturated search directions.

This variant achieves lower mean performance than the full Advisor across all six tasks. Relative performance drops include 1.73% on LLM-SQL, 2.18% on Transaction, and 36.05% on Assortment. These results support the contribution of progress- and saturation-aware guidance beyond merely preventing invalid programs.

We also evaluated the transfer of individual SpecEvo components to LEVI. Under a budget of 500 evaluations on LSR-Synth (Table 21), LEVI + Navigator outperforms LEVI in six of eight settings, while LEVI + Advisor does so in five of eight. However, neither variant matches SpecEvo in any of the eight settings. Similarly, neither variant achieves the performance of the full SpecEvo configuration on the six tasks in the extended ablation study (see W6).

Separate target-matching experiments further demonstrate that an evaluation-efficiency gap remains:

ConfigurationEvaluations to match SpecEvo@500Additional evaluations
SpecEvo500 (reference)—
LEVI75250.4%
LEVI + Navigator68537.0%
LEVI + Advisor73046.0%

These results demonstrate that the individual components provide transferable benefits to LEVI, but neither addition alone recovers the full framework's performance or evaluation efficiency.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.