- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes AlgoEvo, a unified agentic framework for automated algorithm discovery that replaces predefined evolutionary search pipelines with an adaptive, LLM-driven search process. The framework integrates three components: (1) an agentic search loop that dynamically selects code inspection, diagnosis, editing, and evaluation actions; (2) a design skill hub that abstracts paradigm-specific knowledge and enables a shared discovery engine to handle single-heuristic, multi-objective, and multi-component optimization; and (3) a hierarchical experience bank that organizes search trajectories into experience cards and task-level trees, with cross-task consolidation into reusable skills.
Experiments across six combinatorial optimization tasks demonstrate competitive or superior solution quality compared with specialized baselines, including EoH, ReEvo, FunSearch, MEoH, MOTIF, and E2OC. The paper reports substantial reductions in candidate evaluations and token consumption, alongside analyses of progressive optimization, cross-task knowledge transfer, and methodological skill reuse.
Critical strengths
-
Unified and flexible framework. AlgoEvo integrates agentic search, design skills, and hierarchical experience memory into a unified framework supporting three algorithm-design paradigms. The separation of paradigm-specific knowledge from search execution is a valuable architectural contribution.
-
Strong empirical performance and sample efficiency. Table 1 demonstrates competitive or superior performance across six benchmarks, with substantial reductions in candidate evaluations (as few as 7% of the baseline budget) and LLM token consumption.
-
Promising experience accumulation and transfer. Tables 2–4 provide preliminary evidence that accumulated experience improves successive optimization runs and supports cross-task transfer. Section 4.4 further demonstrates the flexibility of reusing methodological knowledge from existing algorithms.
Critical weaknesses
-
Insufficient isolation of core contributions. Existing ablations do not directly compare adaptive agentic search with a fixed search loop under identical tools and resources. Moreover, removing the experience bank or skill hub causes relatively small performance degradation (Appendix F.3), leaving their individual contributions unclear.
-
Incomplete computational efficiency evaluation. The reported evaluation savings exclude diagnostic operations, which may incur substantial computational overhead. Without end-to-end runtime or total execution cost comparisons, the practical efficiency gains remain uncertain.
-
Limited evidence for self-evolution. The progressive optimization experiments (Tables 2–3) reuse both previous best-performing code and accumulated experience, making their respective contributions difficult to distinguish. Cross-task transfer is evaluated on only two related task pairs (Table 4).
Questions for authors
-
Adaptive search: Can the authors compare AlgoEvo against a fixed search loop using the same LLM, design skills, diagnostic tools, and evaluation budget to isolate the benefits of autonomous action selection?
-
Computational efficiency: Could the authors report end-to-end runtime and total computational cost, including diagnostic operations, to substantiate the efficiency claims?
-
Experience accumulation: Can the authors separate the effects of code-only transfer, experience-only transfer, and their combination under matched budgets?
-
Cross-task evolution: In Table 4, does transfer involve retrieved experience cards, evolved design skills, or both? How do their individual contributions affect performance?
8 comments
Overview of the detailed content for Reviewer 94BB
Thank you for your review. The three points you raise, namely whether the core contributions are isolated, whether the efficiency evaluation is complete, and whether the evidence for self-evolution is sufficient, are exactly the questions we needed to answer head-on. Your remark that no direct comparison with a fixed search loop was provided led directly to the resource-matched control in the revision. The three responses follow these three points, with the table data included. If you think other controls are needed, we are glad to add them.
We believe this work has practical value for automated algorithm design, and we hope its positioning can be understood accurately.
If these clarifications resolve your concerns, we would be very grateful if you would reconsider the rating.
The three points are answered in the three responses below.
Detailed Response to Reviewer 94BB on W1 and Q1 (1/3): Isolating the Control and Knowledge Layers
W1: isolation of the core contributions. Insufficient isolation of core contributions. Existing ablations do not directly compare adaptive agentic search with a fixed search loop under identical tools and resources. Moreover, removing the experience bank or skill hub causes relatively small performance degradation (Appendix F.3), leaving their individual contributions unclear.
Q1: comparison against a fixed search loop. Adaptive search: Can the authors compare AlgoEvo against a fixed search loop using the same LLM, design skills, diagnostic tools, and evaluation budget to isolate the benefits of autonomous action selection?
Response
Thank you for your feedback. This comment addresses the isolation of core contributions across two key levels: comparing the autonomous control layer against a fixed loop, and separating the component-wise contributions of the knowledge layer in ablation studies. We have validated both aspects through newly added control experiments in the revision, detailed as follows:
1. Control-Layer Isolation: Adaptive Search vs. Fixed-Schedule Loop (Table 14) To isolate the performance gains brought by autonomous action selection, we established a rigorous iso-resource control experiment in Table 14:
- Alignment Conditions: The fixed-schedule controller and the adaptive search loop share the exact same backbones, skill files, experience bank, initial templates, evaluation entry points, 500-evaluation budget, and action set. The sole difference lies in the action scheduling mechanism: the fixed controller strictly alternates between "diagnostic probing" and "code editing" in a predetermined order and submits every edit, whereas the adaptive loop decides the next action from the runtime state.
- Experimental Results: Adaptive search achieves a cost of 8.999 on the held-out test set, outperforming the fixed schedule's 9.136. Concurrently, adaptive search consumes only 271 evaluations, 7,530s of wall-clock time, and 139.2M tokens, whereas the fixed schedule exhausts the full 500 evaluations, taking 10,759s and consuming 152.1M tokens.
Adaptive control outperforms the fixed loop across quality, evaluation count, and computational resources simultaneously. This rules out the explanation that "adaptiveness merely equates to early stopping," proving the efficacy of dynamically selecting actions according to search states.
Table 14 (Appendix F.5)
| Control | Train | Held-out | Evals | T_tot (s) | Tokens |
|---|---|---|---|---|---|
| Adaptive | 8.860 | 8.999 | 271 | 7,530 | 139.2M |
| Fixed schedule | 9.180 | 9.136 | 500 | 10,759 | 152.1M |
2. Knowledge-Layer Contribution Separation: Task Saturation and Unsaturated Ablations (Table 10) In Appendix F.3 of the original draft, the performance degradation upon removing the experience bank or skill hub was relatively small (e.g., 0.66% and 0.58% on CVRP-DR). This is because CVRP-DR fully converges within the 500-evaluation budget: final metrics for all five ablation variants cluster between 9.040 and 9.122 (a narrow range of 0.09), compressing inter-mechanism differences within the measurement precision of the convergence plateau.
To prevent saturation effects from masking mechanistic differences, we re-ran this ablation suite under identical starting points and budgets on the unsaturated Bi-TSP task (Table 10 and Figure 12):
- Full framework reaches 0.8566;
- w/o autonomous diagnosis drops to 0.8103 (−5.41%);
- w/o experience bank drops to 0.8214 (−4.11%);
- w/o design skill hub drops to 0.8352 (−2.50%);
- w/o contextual trigger injection drops to 0.8473 (−1.08%).
All four removals yield consistent negative degradations, with identical directional trends across all three random seeds.
Table 10 (Appendix F.3)
| Variant | CVRP-DR | Bi-TSP (Δ) |
|---|---|---|
| Complete | 9.040 | 0.8566 |
| w/o diagnosis | 9.122 (+0.91%) | 0.8103 (−5.41%) |
| w/o bank | 9.100 (+0.66%) | 0.8214 (−4.11%) |
| w/o hub | 9.092 (+0.58%) | 0.8352 (−2.50%) |
| w/o injection | 9.059 (+0.21%) | 0.8473 (−1.08%) |
These two sets of experiments clearly delineate the sources of our core contributions from the control and knowledge layers respectively: Table 14 quantifies the dual advantages in quality and budget conferred by autonomous action scheduling (reducing held-out cost by 0.137 and saving 229 evaluations); Table 10 reveals the contribution hierarchy among components on an unsaturated task, where removing autonomous diagnosis incurs the largest drop (5.41% > 4.11% > 2.50%).
This substantiates our methodological positioning that "core control resides in runtime state" and justifies presenting it alongside saturated tasks. We have added both controls (Table 14, Appendix F.5; Table 10, Appendix F.3) and marked them in red. We are grateful for these suggestions.
Response to Reviewer 94BB: Contribution Isolation, End-to-End Cost, Self-Evolution and Cross-Task Transfer
We sincerely thank the reviewer 94BB for the constructive and detailed assessment. Your comments on contribution isolation, efficiency accounting and self-evolution have directly improved the evidence we present, and we are grateful for the time and care behind them.
We added a fixed-schedule control sharing every resource with the adaptive loop (Table 14), an end-to-end cost study with diagnostic overhead (Table 15), a five-variant ablation on two tasks (Table 10, Fig. 12), and a third transfer pair (Table 4).
1. (W1, Q1) Isolation: Fixed Search Loop
The control layer is isolated in Table 14. The fixed-schedule control shares the backbone, design skills, experience bank, starting code, evaluation entry, budget of 500 and action set, and alternates diagnosis probes and edits; only the decision differs. The adaptive loop reaches 8.860 / 8.999 (train / held-out) with 271 evaluations, 7530 s and 139.2M tokens; the fixed schedule 9.180 / 9.136 with the full 500, 10759 s and 152.1M. The adaptive loop wins on quality and on cost at once: 0.137 better held-out, 229 fewer evaluations, 3,229 s less wall-clock and 12.9M fewer tokens.
The knowledge layer is isolated in Table 10. Five variants, one mechanism each, run on CVRP-DR and Bi-TSP; CVRP-DR converges within the allowance (variants within 0.09), while on the unsaturated Bi-TSP they separate: autonomous diagnosis −5.41%, experience bank −4.11%, design skill hub −2.50%, situation-triggered injection −1.08%, all three seeds in the same direction (Fig. 12).
2. (W2, Q2) End-to-End Cost
Table 15 splits one TSP design run into Teval and Tsearch, with diagnostic operations inside Tsearch rather than excluded. A complete design takes 5.78 min against 21.57–34.25 min, with 1.80 min of evaluation and 3.98 min of search, 2.9M tokens against 3.2–4.3M.
3. (W3) Self-Evolution Evidence
Tables 2–3 do not reuse the code of a previous round. Every seed and every round starts from the same template, and improved code is distilled into a card as a rule and retrieved as context. Warm starting moves TSP from 4.961 to 4.808 at 48→40 evaluations, and multi-round accumulation moves 4.8928 to 4.7995 (−1.91%) with Bi-FJSP HV rising from 0.1549 to 0.1862 (+20.21%).
Cross-task transfer now covers three pairs: CVRP→CVRP-DR 10.314→9.816 (−4.8%), TSP→Bi-TSP 336.5→352.2 (+4.7%) and FJSP 4-Ops→Bi-FJSP 0.911→0.914 (+0.33%). The percentage varies with problem difficulty and with the scale the evaluator reports on: a target near its achievable HV has little headroom, and a coarse objective moves in larger steps. The direction is what this comparison establishes, and it holds in all three pairs, including the one that crosses a paradigm boundary (FJSP 4-Ops → Bi-FJSP), a transfer that methods designed for a single problem class cannot attempt.
4. (Q3) Code Lives in Experience: No Code-Only Transfer
The design decision is not code versus experience; it is which rule applies here. The experience bank stores the situation, the modification as a rule, the reference implementation that realizes it and the outcome. At the next run the agent reads the bank and rewrites the program from the template, so code always travels inside the bank and never as a starting program. That is what separates AlgoEvo from continual-evolution systems such as E2OC, which carry the previous program forward as the new starting point. With a single channel, the natural test is to close it: the without-bank variant of Table 10 costs −4.11% on Bi-TSP and +0.66% on CVRP-DR, and Tables 2–3 give the controlled readings of the same channel.
5. (Q4) Only Experience Cards Transfer
Only the experience bank transfers; the skills are identical in both variants. The bank stores the situation, the modification, the rule and the outcome, so the two variants of Table 4 differ only in the bank they can retrieve from, while the target keeps the same template, skill files and budget.
The bank is tested at three levels: successive runs (Table 2, 4.961→4.808 at 48→40 evaluations; Table 3, 4.8928→4.7995 with HV 0.1549→0.1862), across problems (Table 4), and by removing it (Table 10, −4.11% on Bi-TSP and +0.66% on CVRP-DR). The skill is tested by removing the hub (Table 10, −2.50% and +0.58%) and by agents without it: general coding agents without a skill hub or bank reach only 9.03–10.50 (Table 13), and the weakest backbones produce no valid submission (Table 11). The control layer is isolated in Table 14. Skill consolidation needs at least μ experience trees and is not enabled here, so Table 4 measures the bank alone.
All revisions are marked in red, and we hope they address your concerns. We note with appreciation the time and care behind your comments. Should you have any further questions or suggestions, we would be glad to discuss them.