acceptodds
Under review as a conference paper at ICLR 2027

Evolving with Feasibility: Turning Failures into Feedback for LLM-Driven Algorithm Evolution

Abstract

Large language models (LLMs) are increasingly used to automate algorithm design by evolving executable solver programs. However, generated offspring solvers may fail to execute, exceed runtime limits, or violate problem constraints, preventing their algorithmic changes from being meaningfully evaluated. Such failures are particularly costly under limited candidate budgets: they waste generation opportunities, can terminate useful search trajectories, and may recur across generations. We propose RevivEvo, a feasibility-guided framework that treats feasibility not only as an evaluation outcome, but as feedback for evolution. RevivEvo uses parent-specific search and failure experience to prevent known risks during generation, type-aware bounded repair to recover infeasible offspring, and failure and repair memory to reuse experience across generations. Across 14 tasks from CO-Bench and HeuriGym with a budget of 20 candidates, RevivEvo increases the final candidate feasibility rate from 35.60% to 92.38% over the strongest feasibility baseline and improves the macro-averaged Test Avg. Score from 0.665 to 0.832 over the strongest performance baseline. Moreover, 52.4% of final selected solvers either result from repair or descend from a repaired candidate, showing that recovery can benefit subsequent evolution beyond immediate candidate repair. These results demonstrate that incorporating feasibility feedback throughout evolution improves both the reliability and effectiveness of LLM-driven algorithm design.

Primary area:
optimization
Keywords:
Large Language Models, Automatic Algorithm Design, Combinatorial Optimization, Evolutionary Search, Feasibility-Aware Evolution, Program Repair

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_j4m9OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses offspring infeasibility in LLM-driven algorithm evolution. Built on MCTS-AHD, RevivEvo integrates Parent-Specific Offspring Generation, Type-Aware Bounded Repair, and Failure and Repair Memory. Experiments on 14 tasks from CO-Bench and HeuriGym show improvements in Test Avg. Score and the final candidate feasibility rate. Component ablations, analysis of parent-specific generation, and lineage analysis further examine the roles of repair and experience reuse.

Critical strengths
  1. The paper is well motivated and addresses offspring infeasibility, a practical bottleneck in LLM-driven algorithm evolution.

  2. The experiments across 14 tasks support the method’s effectiveness in improving candidate feasibility and final solver quality, while the ablation studies provide evidence for the contributions of its main components.

Critical weaknesses

Limited evaluation across candidate budgets. The main experiments use a 20-candidate budget, including initialization. Although Figure 6(a) extends selected baselines to 80 candidates, it covers only seven tasks with single runs and does not evaluate RevivEvo at larger budgets. Larger-budget comparisons across multiple random seeds would clarify whether its advantage persists.

Questions for authors
  1. Table 3 supports the usefulness of both memory components. What evidence specifically supports separating Lineage-Specific Failure Memory and Retrieved Repair-Skill Memory, rather than using a single memory for both generation and repair with the same historical information and similar input lengths?
  2. For Figure 6(b), how are input, output, and reasoning tokens counted, and are all LLM calls used to compare programs and update memories included?
  3. Could the authors report, for each method, the proportion of test instances on which the final selected solver is feasible, the proportion of final selected solvers that are strictly feasible on the test set, and how often no strictly feasible candidate is found on the development set, requiring selection of the candidate with the highest partial score?
  4. Unclear sensitivity to the LLM backbone. RevivEvo is evaluated only with DeepSeek V4 Flash. The direct LLM synthesis baselines do not assess its sensitivity to the LLM backbone. Evaluation with additional backbones would clarify whether the reported gains depend on the chosen model.
4 comments

1. Response to W1

W1: Limited evaluation across candidate budgets. The main experiments use a 20-candidate budget, including initialization. Although Figure 6(a) extends selected baselines to 80 candidates, it covers only seven tasks with single runs and does not evaluate RevivEvo at larger budgets. Larger-budget comparisons across multiple random seeds would clarify whether its advantage persists.

Thank you for this suggestion. We have so far added a single-run evaluation of RevivEvo with a target budget of 80 candidates on the same seven tasks: aircraft landing, capacitated warehouse location, open-shop scheduling, periodic vehicle routing, operator scheduling, crew pairing, and pedigree. We compare against the 80-candidate results of EoH, ReEvo, and RefineEvo reported in Figure 6(a). Following Appendix A.3, the final program is selected using development performance and evaluated separately on the test set; the reported Test Avg. Score gives equal weight to the seven tasks. Due to the computational cost, repeated-seed experiments at this larger budget have not yet been completed; we plan to add them in follow-up experiments.

MethodTest Avg. Score ↑Candidate Feasible Rate ↑
EoH0.763436.25%
ReEvo0.810852.14%
RefineEvo0.806446.79%
RevivEvo0.897193.21%

RevivEvo achieves a Test Avg. Score of 0.8971 versus 0.8108 for ReEvo, the strongest listed baseline, a gain of 0.0863. Candidate feasibility is also higher (93.21% versus 52.14%). The advantage is also observed in this single-run larger-budget evaluation. These results are reported in Appendix B.2.


2. Response to Q1

Q1: Table 3 supports the usefulness of both memory components. What evidence specifically supports separating Lineage-Specific Failure Memory and Retrieved Repair-Skill Memory, rather than using a single memory for both generation and repair with the same historical information and similar input lengths?

Thank you for this question. We maintain separate memories because they capture experience from different stages of the process, are accessed at different stages, and retain different types of information. Failure memory is populated with errors revealed when an offspring fails evaluation and guides subsequent generation along its lineage. In contrast, repair-skill memory is built from completed repair attempts and their validation outcomes, and is retrieved when repairing a later offspring with a similar failure. The two memories also differ in content: failure memory focuses on what failed and what to avoid, whereas repair-skill memory focuses on how the failure was addressed and whether the repair succeeded, including corrective actions and validation outcomes (Section 2.4 and Appendix C.3).

For example, an e-graph extraction offspring introduced an SCC-based selection mechanism that produced cyclic dependencies in the extracted subgraph, causing it to fail the acyclicity check during development evaluation. Repair restored level-based acyclic selection to resolve this constraint violation. This episode provided two forms of experience: failure memory recorded a warning against cycle-blind selection, which informed a later proposal to incorporate explicit cycle checks; repair-skill memory recorded the corrective procedure and its successful validation on six development instances for retrieval during later repairs.


3. Response to Q2

Q2: For Figure 6(b), how are input, output, and reasoning tokens counted, and are all LLM calls used to compare programs and update memories included?

For Figure 6(b), we count input, output, and reasoning tokens from all LLM calls, including candidate generation, program comparison and selection, feedback analysis, failure diagnosis, code repair, and memory updates. All these calls are included in the total token count.

4. Response to Q3

Q3: Could the authors report, for each method, the proportion of test instances on which the final selected solver is feasible, the proportion of final selected solvers that are strictly feasible on the test set, and how often no strictly feasible candidate is found on the development set, requiring selection of the candidate with the highest partial score?

Thank you for this suggestion. We computed the following statistics from the main-experiment records. The three columns correspond directly to your questions:

  • Test-instance feasibility: for each task–seed run, we compute "the proportion of test instances on which the final selected solver is feasible", then average these proportions equally across the 14 tasks and three seeds.
  • Strict test feasibility: "the proportion of final selected solvers that are strictly feasible on the test set," requiring each solver to pass every test instance.
  • No feasible dev candidate: "how often no strictly feasible candidate is found on the development set." This is the proportion of runs that end without finding any candidate passing all development instances; lower is better. Such runs require selection by the highest partial score.

The latter two columns report counts and percentages over 42 runs per method (14 tasks × 3 random seeds).

MethodTest-instance feasibility ↑Strict test feasibility ↑No feasible dev candidate ↓
FunSearch86.59%21/42 (50.00%)15/42 (35.71%)
EoH84.02%21/42 (50.00%)16/42 (38.10%)
ReEvo85.33%19/42 (45.24%)10/42 (23.81%)
HSEvo82.91%20/42 (47.62%)17/42 (40.48%)
MCTS-AHD65.26%12/42 (28.57%)24/42 (57.14%)
HiFo-Prompt80.13%20/42 (47.62%)17/42 (40.48%)
RefineEvo81.24%17/42 (40.48%)14/42 (33.33%)
A2DEPT81.18%18/42 (42.86%)20/42 (47.62%)
RevivEvo99.11%34/42 (80.95%)0/42 (0.00%)

RevivEvo achieves 99.11% test-instance feasibility, compared with at most 86.59% among the listed baselines. In addition, 34 of its 42 selected solvers (80.95%) pass every test instance, compared with at most 21 of 42 (50.00%) for the baselines. All 42 RevivEvo runs find a strictly feasible development candidate.


5. Response to Q4: Sensitivity to the LLM Backbone

Q4: Unclear sensitivity to the LLM backbone. RevivEvo is evaluated only with DeepSeek V4 Flash. The direct LLM synthesis baselines do not assess its sensitivity to the LLM backbone. Evaluation with additional backbones would clarify whether the reported gains depend on the chosen model.

Thank you for this suggestion. We added experiments using Qwen 3.7 Flash and Doubao Seed 2.1 Turbo. Each backbone was evaluated independently on all 14 tasks using seed 2024 only, without repeated runs. The table reports the macro-averaged Test Avg. Score, as reported in Appendix B.3:

LLM configurationTest Avg. Score ↑
DeepSeek V4 Flash0.8238
Qwen 3.7 Flash0.8024
Doubao Seed 2.1 Turbo0.8503

RevivEvo achieves macro-averaged scores of 0.8238 with DeepSeek V4 Flash, 0.8024 with Qwen 3.7 Flash, and 0.8503 with Doubao Seed 2.1 Turbo. These results show that RevivEvo maintains strong performance across all three tested LLM backbones, supporting the effectiveness of our pipeline beyond DeepSeek V4 Flash.


We again thank the reviewer for the careful evaluation and constructive feedback. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.

Dear Reviewer j4m9,

Thank you again for your time and consideration. As the ICLR discussion period is approaching its conclusion, we would greatly appreciate any feedback on whether our rebuttal has addressed your concerns.

If any concerns remain, we would be happy to provide further clarification. If our responses have adequately addressed your concerns, we would sincerely appreciate your reconsideration of the score.

Thank you again for your time.

Best regards,
The Authors

Reviewer_j4m9OpenReview ↗

Good reply

Thank you for the additional experiments and clarifications. These have addressed most of my concerns, and I will maintain my score

Reviewer_UqYbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A novel approach towards evolving feasible problem solving algorithms in automated algorithm evolution using LLMs. To improve feasibility of the generated algorithms, authors use failure evidence during the search to better guide the evolutionary operators. The results improve feasibility significantly across a comparison with existing approaches on benchmarking tasks from CO-Bench and HeuriGym.

Critical strengths
  • Novel approach
  • Good results outperforming existing methods
Critical weaknesses
  • The paper is not always clear, using math formalism where it is not helpful to improve understandability
  • The benchmarks are maybe what everybody is using, but what is the real-world practical relevance?
Questions for authors

150: what do you mean by "instance"? Likewise, what is an Evaluation set? 158: you are mentioning "benchmark" - I see what your are proposing, but it is not obvious for non-EA experts. Eqns (3), (4): What is A? What Kind of mapping is defined here (Domain, codomain)? I understand what you want to say, but it Needs to be defined in more Detail. 269: What do you mean by "receives"? I generates 20 programs, iteratively? Or the Population size is 20?

I miss a proper state of the art description - i.e., how has this been handled in other Systems for automated algorithm evolution? In section 4 you mention related work but do not clarify how it handles constraints.

Appendix: eqn (8) has too many undefined variables - it is not possible to understand what you are doing here.

3 comments

1. Response to W1 / Q1

W1: The paper is not always clear, using math formalism where it is not helpful to improve understandability.

Q1: 150: what do you mean by "instance"? Likewise, what is an Evaluation set? 158: you are mentioning "benchmark" - I see what you are proposing, but it is not obvious for non-EA experts. Eqns (3), (4): What is A? What kind of mapping is defined here (Domain, codomain)? I understand what you want to say, but it needs to be defined in more Detail. 269: What do you mean by "receives"? I generates 20 programs, iteratively? Or the Population size is 20?

Thank you for identifying these ambiguities. We now explain the concepts and mappings in words alongside the equations:

  • “What do you mean by ‘instance’?” / “What is an Evaluation set?” / “benchmark.” An instance is one concrete task input, such as a set of cities and their pairwise distances in TSP. An evaluation set is a collection of instances used to assess a program's feasibility and solution quality. A benchmark provides the instances, constraint verifiers, and scoring rules. These definitions are now explicit in §2.1.
  • “What is A? What kind of mapping is defined here (Domain, codomain)?” A\mathcal A denotes a stochastic LLM generation procedure with a role-specific prompt. The Proposal Agent maps parent evidence, a progress cue, and the task description to textual revision proposals; the Code Agent maps parent source code, the proposal, and the task description to complete offspring source code. These input domains and output codomains are now defined beside Equations (3)–(4), §2.2.
  • “What do you mean by ‘receives’?” / “Or the Population size is 20?” Each run generates up to 20 candidates over initialization and subsequent evolution, including unsuccessful generations. This is a cumulative generation budget, not the population size, as clarified in §3.1.

We also checked the rest of the manuscript and revised analogous ambiguities to ensure consistent definitions and explanations throughout.


2. Response to W2

W2: The benchmarks are maybe what everybody is using, but what is the real-world practical relevance?

Thank you for this question. The practical relevance is to develop solver programs for constrained decisions in logistics and computing, where both solution quality and feasibility matter. For example, CO-Bench's warehouse-location task models facility selection and customer allocation under capacity constraints, with facility and allocation costs as the objective. Its periodic-routing task models multi-day delivery with customer visit-pattern and vehicle-capacity requirements. HeuriGym's intra-op parallelism task models execution-strategy selection to reduce computational cost under memory limits. Appendix A.1 details these tasks and their objectives.

RevivEvo addresses a practical obstacle in automating the development of such solvers: generated programs may fail to execute, exceed runtime limits, or produce solutions that violate these requirements. A low-cost route or allocation is not usable if it violates the task constraints. By preventing and repairing these failures, RevivEvo increases the yield of feasible candidates and improves final solver quality under limited search budgets. Our experiments evaluate this benefit on reproducible benchmark instances of these application-relevant problems.


3. Response to Q2

Q2: I miss a proper state of the art description - i.e., how has this been handled in other Systems for automated algorithm evolution? In section 4 you mention related work but do not clarify how it handles constraints.

Thank you for pointing out this omission. Existing systems handle constraints through fixed solver components or execution-based filtering and penalties. ReEvo's capacitated vehicle-routing implementation uses fixed solver components; FunSearch evaluates and filters generated functions during evolution; and A2DEPT completes missing helper functions and penalizes failed or infeasible candidates.

During evolution, modifying a complete solver can introduce new task-constraint violations. Fixed components cannot guarantee that modified offspring remain feasible, while filtering and dependency repair do not directly diagnose and correct these violations. RevivEvo distinguishes code errors, timeouts, and constraint violations to guide offspring diagnosis and bounded repair, then strictly re-evaluates the result and reuses the experience in subsequent evolution. We have added this comparison to §4, “Feasibility-Aware Algorithm Synthesis.”

4. Response to Q3

Q3: Appendix: eqn (8) has too many undefined variables - it is not possible to understand what you are doing here.

Thank you for highlighting the missing definitions. Equation (8) selects the next child during tree traversal by balancing value and exploration. Here, v,wv,w are the current node and its child; tt is the search step; Nv,NwN_v,N_w are visit counts; HwH_w is the child's selection value, distinct from program quality QQ; hmin⁡,hmax⁡h_{\min},h_{\max} are running reward bounds; and nt,Bn_t,B are the consumed and total candidate budgets.

Search selects the child maximizing Ut(w)U_t(w). The first term favors higher values, while the second encourages less-visited branches with a weight that decreases as the budget is consumed. When the normalization bounds coincide, the least-visited child is selected instead. Appendix C.1 now defines these quantities and explains value initialization and backpropagation. We also checked the rest of the manuscript and revised analogous ambiguities to ensure consistent definitions and explanations throughout.


We again thank the reviewer for the careful evaluation and constructive feedback. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.

Dear Reviewer UqYb,

Thank you again for your time and consideration. As the ICLR discussion period is approaching its conclusion, we would greatly appreciate any feedback on whether our rebuttal has addressed your concerns.

If any concerns remain, we would be happy to provide further clarification. If our responses have adequately addressed your concerns, we would sincerely appreciate your reconsideration of the score.

Thank you again for your time.

Best regards,
The Authors

Reviewer_HgE5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RevivEvo, a feasibility-guided algorithm evolution framework combining parent-specific generation, type-aware repair, and failure/repair memory. Experiments on CO-Bench and HeuriGym demonstrate improvements in solver feasibility and performance.

Critical strengths
  • The paper is well written and clearly structured.

  • The paper addresses a practical and important problem in LLM-driven algorithm evolution. The idea of recovering infeasible candidates is intuitive and meaningful.

  • The experiments cover diverse optimization tasks and show promising improvements. The ablation studies and case studies provide useful insights into the proposed method.

Critical weaknesses
  1. Repair attempts do not count toward the candidate budget, introducing additional LLM calls and evaluations. Figure 6 does not fully address this issue. Comparisons under matched total compute budgets would be more convincing.

  2. The paper lacks a full w/o Repair ablation and comparisons against simple self-repair baselines. More systematic ablations of the proposal, repair, and memory components would help clarify their contributions.

  3. Some repairs may revert to the parent's original algorithm rather than preserve new mechanisms. The reported lineage contribution does not directly establish that repair enables better algorithm discovery.

  4. The evaluated tasks have relatively fixed interfaces and constraints. It remains unclear whether the approach generalizes to more heterogeneous settings, such as kernel optimization or mathematical discovery.

Questions for authors
  • How does RevivEvo compare with MCTS-AHD + generic self-repair under matched compute budgets?

  • Can the authors provide more complete component ablations and analyze how often repair preserves the proposed algorithmic changes?

I would be open to raising my score if the authors provide more comprehensive ablations and additional experiments demonstrating generalization to more diverse algorithm-discovery settings.

4 comments

1. Response to W1 / Q1

W1: Repair attempts do not count toward the candidate budget, introducing additional LLM calls and evaluations. Figure 6 does not fully address this issue. Comparisons under matched total compute budgets would be more convincing.

Q1: How does RevivEvo compare with MCTS-AHD + generic self-repair under matched compute budgets?

Thank you for highlighting this gap. We added comparisons under two separate per-run resource budgets: 400K total LLM tokens or 30 development-evaluator calls. The token budget counts input and output tokens across all search-related LLM calls, including proposal generation, code generation, diagnosis, repair, and memory updates. One evaluator call evaluates a candidate program on the entire development set DdevD_{\mathrm{dev}}; both initial evaluations and post-repair re-evaluations count toward the 30-call budget. Under each budget, we select the program based on development performance and evaluate it on the test set. To limit computational cost, we selected seven of the 14 tasks, focusing on more challenging tasks across different problem types: aircraft landing, capacitated warehouse location, open-shop scheduling, periodic vehicle routing, operator scheduling, crew pairing, and pedigree. The table reports Test Avg. Score, with equal weight assigned to each task.

To test whether ordinary self-repair is sufficient, we added MCTS-AHD + Generic Repair. It augments the CO-Bench implementation of MCTS-AHD and attempts to repair infeasible candidates using the task requirements, current code, and evaluation feedback, with up to five repair attempts per candidate, as in RevivEvo, but without our proposal planning, type-specific diagnosis, or failure/repair memories.

Method400K tokens30 evaluator calls
EoH0.77260.6922
ReEvo0.73410.7932
RefineEvo0.75960.7854
MCTS-AHD + Generic Repair0.82770.8255
RevivEvo0.89400.9051

RevivEvo scores 0.8940 versus 0.8277 for MCTS-AHD + Generic Repair under the token budget, and 0.9051 versus 0.8255 under the evaluator-call budget, yielding gains of 0.0663 and 0.0796. Thus, adding generic self-repair does not close the performance gap to the full RevivEvo framework under either tested budget. This supports the benefit of the full framework beyond ordinary self-repair under the tested resource constraints. The settings and results are included in Appendix B.1.


2. Response to W2 / Q2

W2: The paper lacks a full w/o Repair ablation and comparisons against simple self-repair baselines. More systematic ablations of the proposal, repair, and memory components would help clarify their contributions.

Thank you for this suggestion. We expanded the original within-component analysis to include Proposal, Repair, and Memory ablation results, covering 14 tasks with 20 candidates per task. The full-component ablation study, described in Appendix B.4, uses seed 2024 without repeated runs. We report the macro-averaged Test Avg. Score and final candidate feasibility after each variant's pipeline.

VariantTest Avg. Score ↑Feasible Rate ↑
Full0.823893.57%
w/o Proposal Agent0.727487.86%
w/o Repair0.696669.64%
w/o Memory0.763383.57%

Removing repair causes the largest observed degradation: Test Avg. Score decreases from 0.8238 to 0.6966, while candidate feasibility decreases from 93.57% to 69.64%, a drop of 23.93 percentage points. Removing the Proposal Agent or memory also lowers the score, to 0.7274 and 0.7633, respectively, and reduces feasibility to 87.86% and 83.57%. These results support the contribution of each component to solver quality and candidate feasibility within the full framework under this experimental setting. The results have been added to Appendix B.4, “Component Ablation Study.”

3. Response to W3 / Q2

W3: Some repairs may revert to the parent's original algorithm rather than preserve new mechanisms. The reported lineage contribution does not directly establish that repair enables better algorithm discovery.

Q2: Can the authors provide more complete component ablations and analyze how often repair preserves the proposed algorithmic changes?

Thank you for highlighting this distinction. Our goal is for repair to recover new algorithmic mechanisms, rather than restore feasibility by reverting to the parent. We therefore audited 98 successfully repaired offspring with feasible parents for comparison from the main experiments, using GPT-6 Astra with medium reasoning effort. With fixed prompts and criteria, the audit compared the proposed change and the repaired program with the parent, judging mechanisms implemented in operative code rather than textual similarity or inactive code. We distinguished three outcomes:

  • Core retained: the repaired program preserves the core new mechanism proposed in the offspring.
  • Core replaced: the repaired program implements a different substantive mechanism from the proposed one, while remaining distinct from the parent's strategy.
  • Parent mechanism: the repaired program abandons the proposed new mechanism and reverts to the parent's strategy.
CategoryCountShare
Core retained5758.2%
Core replaced88.2%
Parent mechanism3333.7%

Overall, 65 of 98 successful repairs (66.3%) preserved the proposed new mechanism or implemented an alternative mechanism beyond the parent's, while 33 (33.7%) reverted to the parent. The first two categories are effective in recovering algorithmic changes; the third restores feasibility without recovering a new mechanism.

However, the parent-reverting repairs do not account for the repair contributions to the final selected solvers in Figure 3(b). We re-audited all 22 repair-associated final solvers (52.4% of 42) and found that the repairs underlying their direct or lineage-based contributions preserved or introduced new mechanisms, rather than reverting to the parent. Thus, all 22 remain associated with effective repairs under the above definition, and the 52.4% result in Figure 3(b) is unchanged. These findings support our conclusion that repair enables better algorithm discovery by recovering useful changes and preserving branches that lead to competitive final solvers. The audit is reported in Appendix B.5.

4. Response to W4

W4: The evaluated tasks have relatively fixed interfaces and constraints. It remains unclear whether the approach generalizes to more heterogeneous settings, such as kernel optimization or mathematical discovery.

Thank you for encouraging this extension. Our original evaluation was not limited to classical combinatorial optimization: it also covered biological design and compiler/hardware optimization, including protein sequence design, e-graph extraction, and intra-op parallelism. To further evaluate our pipeline in more heterogeneous algorithm-discovery settings, we added experiments in mathematical discovery. Specifically, LLM-SRBench [1] requires discovering mathematical expressions from observational data. We selected one biology task (BPG0) and one chemistry task (CRK1).

We compared RevivEvo with LLM-SR [2], an agentic method tailored to scientific equation discovery, using the same LLM, data splits, and numerical evaluator. Both methods received 20 evaluator calls per task, including repair re-evaluations where applicable. Search used training data only; the selected programs were then evaluated on in-distribution (ID) and out-of-distribution (OOD) test sets. We report normalized mean squared error (NMSE; lower is better).

MethodBPG0 ID NMSE ↓BPG0 OOD NMSE ↓CRK1 ID NMSE ↓CRK1 OOD NMSE ↓
LLM-SR4.066 × 10⁻⁶0.10045.127 × 10⁻⁸2.265 × 10⁻⁴
RevivEvo3.902 × 10⁻⁶0.11237.630 × 10⁻⁹2.904 × 10⁻⁵

Under the same 20-evaluator-call budget, RevivEvo achieves lower NMSE on three of the four task–split combinations. On CRK1 OOD, NMSE decreases from 2.265 × 10⁻⁴ with LLM-SR to 2.904 × 10⁻⁵ with RevivEvo; on BPG0 OOD, LLM-SR performs better (0.1004 versus 0.1123). These results provide initial evidence that our pipeline can generalize to more heterogeneous algorithm-discovery settings, including mathematical discovery, addressing your concern beyond the original task domains.

[1] Shojaee, P., Nguyen, N.-H., Meidani, K., Barati Farimani, A., Doan, K. D., and Reddy, C. K. LLM-SRBench: A New Benchmark for Scientific Equation Discovery with Large Language Models. ICML, 2025.

[2] Shojaee, P., Meidani, K., Gupta, S., Barati Farimani, A., and Reddy, C. LLM-SR: Scientific Equation Discovery via Programming with Large Language Models. ICLR, 2025.


We again thank the reviewer for their careful evaluation and constructive feedback. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.

Dear Reviewer HgE5,

Thank you again for your time and consideration. As the ICLR discussion period is approaching its conclusion, we would greatly appreciate any feedback on whether our rebuttal has addressed your concerns.

If any concerns remain, we would be happy to provide further clarification. If our responses have adequately addressed your concerns, we would sincerely appreciate your reconsideration of the score.

Thank you again for your time.

Best regards,
The Authors

Reviewer_2DPKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes RevivEvo to address execution errors, timeouts, and constraint violations in LLM-driven algorithm evolution. Specifically, it combines parent-specific revision planning, type-aware bounded repair, and the reuse of past failure and repair experiences. Across 14 tasks from CO-Bench and HeuriGym, RevivEvo improves both candidate feasibility and final solver performance compared with the baselines.

Critical strengths
  1. A practically important problem setting. As the paper points out, failures occur frequently during the search process, and addressing them is a practically important problem. The proposed intuition of preventing failures and reusing experiences from past failures is both natural and important, and could prove useful in real-world settings.
  2. Strong performance across diverse tasks. RevivEvo achieves the best performance on many tasks, with particularly notable gains on PVRP and PSD: 0.872 and 0.961, compared with the best baseline scores of 0.470 and 0.586, respectively. The high feasibility rate of final candidates reported in Table 2 further supports the system's practical potential.
Critical weaknesses
  1. Lack of performance comparisons under equal total computational budgets. The experiments do not establish whether RevivEvo retains its performance advantage when all methods receive the same total computational budget. Although Figure 6 analyzes search efficiency, Figure 6(a) uses candidate counts as a measure of budget, which does not fully account for the cost of RevivEvo's additional repair steps. Figure 6(b) reports tokens per feasible candidate rather than comparing final solution quality under the same total token budget.
  2. Limited reporting of statistical reliability and aggregation. Although the main experiments use three seeds, Table 1 provides only one value per method and task, without reporting variance or standard deviation across seeds. The aggregation method used to obtain each entry also needs clarification. Reporting the mean and variance across the three seeds would help establish the reliability of the performance differences. Furthermore, while Table 2 describes outcomes after each method's complete pipeline, the precise denominators, aggregation across tasks and seeds, and rules for assigning failure categories are insufficiently specified, limiting the interpretation of the large feasibility improvements.
  3. Underspecified controls in the component analysis. The paper provides ablations, lineage analysis, and case studies, but the operators, prompts, and evidence used in the controls shown in Figure 5 remain unclear. It is also unclear whether evidence-augmented operators receive the same parent information and memories, and how these variants differ beyond the separate proposal stage. These details matter for attributing the observed gains to the separation of revision planning from code implementation.
Questions for authors
  1. Please address the concerns raised in the weaknesses section.
  2. Could the authors clarify the rationale for selecting these 14 tasks? Since prior work such as ReEvo and RefineEvo evaluates on TSP, I would be interested in understanding whether there was a particular reason for not including TSP here. If available, results on TSP would also be helpful.
  3. How is each percentage in Table 2 calculated? Please clarify the counting unit and denominator, how initialization and repair attempts are treated, how results are aggregated across tasks and seeds, and how candidates with different failure types across instances are classified.
  4. Could the authors clarify how the variants in Figure 5 differ? In particular, it would be helpful to specify the operators, prompts, and parent information provided to each variant, and explain what distinguishes evidence-augmented operators from parent-specific generation.
6 comments

1. Response to W1 / Q1

W1: Lack of performance comparisons under equal total computational budgets. ... Figure 6(b) reports tokens per feasible candidate rather than comparing final solution quality under the same total token budget.

Thank you for highlighting this gap. We added comparisons under two separate per-run resource budgets: 400K total LLM tokens or 30 development-evaluator calls. The token budget counts input and output tokens across all search-related LLM calls, including proposal generation, code generation, diagnosis, repair, and memory updates. One evaluator call evaluates a candidate program on the entire development set DdevD_{\mathrm{dev}}; both initial evaluations and post-repair re-evaluations count toward the 30-call budget. Under each budget, we select the program based on development performance and evaluate it on the test set. To limit computational cost, we selected seven of the 14 tasks, focusing on more challenging tasks across different problem types: aircraft landing, capacitated warehouse location, open-shop scheduling, periodic vehicle routing, operator scheduling, crew pairing, and pedigree. The table reports Test Avg. Score, with equal weight assigned to each task.

Method400K tokens30 evaluator calls
EoH0.77260.6922
ReEvo0.73410.7932
RefineEvo0.75960.7854
RevivEvo0.89400.9051

RevivEvo achieves the highest observed score under both budgets: 0.8940 versus 0.7726 for EoH under the token budget, and 0.9051 versus 0.7932 for ReEvo under the evaluator-call budget, corresponding to gains of 0.1214 and 0.1119. RevivEvo retains its advantage under both resource controls: a fixed total LLM-token budget and a fixed number of development-evaluator calls. The settings and results are included in Appendix B.1.


2. Response to W2 / Q3

Q3: How is each percentage in Table 2 calculated? Please clarify the counting unit and denominator, how initialization and repair attempts are treated, how results are aggregated across tasks and seeds, and how candidates with different failure types across instances are classified.

Thank you for highlighting the unclear reporting. We clarify the requested details of Table 2 point by point below.

  1. Counting unit and denominator. The counting unit is one candidate program. The denominator is 840 candidates per method (14 tasks × 3 seeds × 20 candidates). Each percentage equals its category count / 840 × 100. A candidate is feasible only if it passes every development instance within the prescribed execution limits and satisfies all constraint checks.

  2. Initialization and repair. Initialization and unsuccessful generations count toward the 20-candidate budget. Repair does not add candidates: regardless of the number of repair attempts, each candidate is counted once by its final outcome.

  3. Aggregation across tasks and seeds. We pool all candidate outcomes across the 14 tasks and three seeds, then calculate the category percentages.

  4. Different failures across instances. When a candidate exhibits multiple failure types across development instances, we prioritize code error > constraint violation > timeout, addressing one failure type at each repair step. The same priority determines its terminal category in Table 2. This follows a correctness-first principle: we first ensure that the program executes correctly and produces valid solutions, then address runtime efficiency, which may depend on instance size or difficulty. Our audit found 2 candidates with code errors and timeouts, 3 with constraint violations and timeouts, and none with both code errors and constraint violations. Thus, in the observed mixed-failure cases, this rule prioritizes execution or solution-validity issues before timeouts.

3. Response to W2

W2: Although the main experiments use three seeds, Table 1 provides only one value per method and task, without reporting variance or standard deviation across seeds.

Thank you for highlighting the need to report variability. Below, we report mean ± sample SD across three seeds for every task and all nine agentic methods. The macro average gives equal weight to the 14 tasks; its SD is computed across the three seed-level macro averages.

CO-Bench

MethodALBP1DCWLCDDOSSSCPVRP
FunSearch0.6501 ± 0.13360.9339 ± 0.01410.6928 ± 0.02360.8536 ± 0.00740.7348 ± 0.16100.8857 ± 0.00550.3338 ± 0.0482
EoH0.7597 ± 0.08680.9300 ± 0.00750.7584 ± 0.06090.8356 ± 0.15930.6583 ± 0.22070.8335 ± 0.03290.4696 ± 0.1734
ReEvo0.8501 ± 0.03950.9226 ± 0.00020.8253 ± 0.08660.9203 ± 0.03930.9184 ± 0.01550.6784 ± 0.36670.3366 ± 0.4520
HSEvo0.6045 ± 0.35670.9248 ± 0.00410.7462 ± 0.02380.6505 ± 0.19480.7846 ± 0.25020.8818 ± 0.02680.4313 ± 0.1669
MCTS-AHD0.7567 ± 0.13280.9319 ± 0.01650.7056 ± 0.03680.8383 ± 0.18110.5294 ± 0.46990.7476 ± 0.03160.1561 ± 0.2703
HiFo-Prompt0.8297 ± 0.03450.9405 ± 0.02640.7189 ± 0.02300.7407 ± 0.21010.7170 ± 0.19860.8698 ± 0.00080.1786 ± 0.3094
RefineEvo0.8164 ± 0.01060.9500 ± 0.02820.7690 ± 0.02680.7617 ± 0.18320.8294 ± 0.11490.8688 ± 0.00190.1173 ± 0.2031
A2DEPT0.4280 ± 0.38030.9405 ± 0.01880.6647 ± 0.00000.4783 ± 0.16110.9052 ± 0.02410.8049 ± 0.05400.1878 ± 0.1046
RevivEvo0.8574 ± 0.05510.9444 ± 0.02200.8490 ± 0.06200.9348 ± 0.03440.9514 ± 0.01950.8773 ± 0.02310.8718 ± 0.0747

HeuriGym

MethodOSCPEGIOPPEDPSDTM
FunSearch0.9647 ± 0.01150.5006 ± 0.28920.8757 ± 0.01930.6204 ± 0.07740.6074 ± 0.01900.5271 ± 0.10720.1163 ± 0.1058
EoH0.9732 ± 0.00330.7685 ± 0.11810.6329 ± 0.34580.6468 ± 0.05370.4250 ± 0.11880.5858 ± 0.02920.0248 ± 0.0429
ReEvo0.7529 ± 0.37940.9027 ± 0.00190.2653 ± 0.09090.7503 ± 0.09040.4883 ± 0.14780.5519 ± 0.03210.1218 ± 0.1134
HSEvo0.9753 ± 0.00390.7760 ± 0.11050.2407 ± 0.16210.6117 ± 0.11990.3996 ± 0.07130.3657 ± 0.03910.0959 ± 0.0968
MCTS-AHD0.3231 ± 0.55970.2612 ± 0.29190.0924 ± 0.08010.5875 ± 0.16500.7078 ± 0.11740.4574 ± 0.19360.0000 ± 0.0000
HiFo-Prompt0.9825 ± 0.01120.6067 ± 0.26300.6870 ± 0.38940.5079 ± 0.24400.8283 ± 0.12270.5231 ± 0.10390.0683 ± 0.1183
RefineEvo0.9714 ± 0.01720.7661 ± 0.05400.3517 ± 0.31720.7206 ± 0.06230.7685 ± 0.18160.4033 ± 0.00000.1096 ± 0.0628
A2DEPT0.9185 ± 0.09050.6485 ± 0.34260.4356 ± 0.30630.6649 ± 0.03090.5269 ± 0.18220.5412 ± 0.14700.2583 ± 0.2964
RevivEvo0.9856 ± 0.01420.8596 ± 0.05120.7172 ± 0.10440.7701 ± 0.02130.8415 ± 0.09550.9605 ± 0.06840.2294 ± 0.0054

Overall performance across 14 tasks

MethodOverall Test Avg. Score (mean ± SD)
FunSearch0.6641 ± 0.0503
EoH0.6644 ± 0.0324
ReEvo0.6632 ± 0.0199
HSEvo0.6063 ± 0.0255
MCTS-AHD0.5068 ± 0.0238
HiFo-Prompt0.6571 ± 0.0348
RefineEvo0.6574 ± 0.0110
A2DEPT0.6002 ± 0.0491
RevivEvo0.8321 ± 0.0090

Task abbreviations follow Table 1: AL, aircraft landing; BP1D, one-dimensional bin packing; CWL, capacitated warehouse location; CDD, common due-date scheduling; OSS, open-shop scheduling; SC, set covering; PVRP, periodic vehicle routing; OS, operator scheduling; CP, crew pairing; EG, e-graph extraction; IOP, intra-op parallelism; PED, pedigree; PSD, protein sequence design; TM, technology mapping.

RevivEvo retains the highest macro-averaged score (0.8321 ± 0.0090), compared with 0.6644 ± 0.0324 for EoH, the strongest baseline by mean macro score. These task-level statistics make the variation across seeds explicit. We have also added the mean ± SD of the seed-level macro averages to Table 1.

4. Response to W3 / Q4

Q4: Could the authors clarify how the variants in Figure 5 differ? In particular, it would be helpful to specify the operators, prompts, and parent information provided to each variant, and explain what distinguishes evidence-augmented operators from parent-specific generation.

Thank you for highlighting this ambiguity. Parent-specific generation uses a separate Proposal Agent to propose a revision, which the Code Agent implements. Evidence-augmented operators instead generate offspring directly using MCTS-AHD's fixed operators (e2/m1/m2/s1), without a separate Proposal Agent. Both receive the same parent-evidence fields, constructed by the same rules: parent evaluation, code-structure summaries, search progress, and within-run experience. Thus, the distinction is proposal-guided generation versus fixed-operator generation, rather than the information provided. The fixed-operator variant uses the same operators without the additional parent-evidence package. All variants disable repair and use 20 candidates per run, evolving independently. Details of the original experimental setup are provided in Appendix A.3. We have revised the corresponding description in §3.4 of the manuscript to make these distinctions explicit.

Our Proposal prompt is provided in Appendix D.1, “Generation Prompts.” The controls use i1 for initialization and e2/m1/m2/s1 for evolution. Due to the response length limit, we refer the reviewer to Appendix E.1 of MCTS-AHD [1] for the original fixed-operator prompts adapted in our controls.

Figure 5 shows two benefits: evidence-augmented operators outperform fixed operators, supporting the value of parent evidence; parent-specific generation further improves solver quality and candidate feasibility over evidence-augmented operators, supporting the additional value of a separate proposal stage.


5. Response to Q2

Q2: Could the authors clarify the rationale for selecting these 14 tasks? Since prior work such as ReEvo and RefineEvo evaluates on TSP, I would be interested in understanding whether there was a particular reason for not including TSP here. If available, results on TSP would also be helpful.

Thank you for this suggestion. We first grouped all tasks in CO-Bench and HeuriGym by problem type based on their descriptions, then selected tasks with complex constraints and challenging solution requirements from each group. This yielded 14 tasks—seven from each benchmark—covering scheduling, resource allocation, covering/routing, graph extraction/mapping, and biological design (Appendix A.1). Within routing, we prioritized PVRP rather than TSP because its additional multi-day visit-pattern and vehicle-capacity constraints directly test offspring feasibility; TSP was therefore not separately included in the original task set.

Following your suggestion, we evaluated RevivEvo on the TSP task from CO-Bench using DeepSeek V4 Flash with a search budget of 30 candidates. To limit additional computational cost, we ran only RevivEvo and directly compared it with the published results for all agentic methods in the original CO-Bench TSP evaluation. Those baseline results used GPT-o3 with a budget of 64 candidates. The table reports TSP Avg. Score, where higher is better.

MethodTSP Avg. Score ↑
FunSearch0.9017
EoH0.7935
ReEvo0.8489
MCTS-AHD0.5961
RevivEvo0.9375

RevivEvo achieves a Avg. Score of 0.9375, exceeding the reported FunSearch score of 0.9017 by 0.0358 and ranking highest among the listed results. Backbone differences may contribute to this comparison, while RevivEvo uses fewer than half as many candidate slots (30 versus 64). The result provides additional evidence of its performance on TSP with a limited candidate budget.

[1] Zheng, Z., Xie, Z., Wang, Z., and Hooi, B. (2025). Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design. ICML.


We again thank the reviewer for their careful evaluation and constructive feedback. If our responses have addressed your concerns, we would greatly appreciate your reconsideration of the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.