- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript introduces AdaEva, a drop-in adaptive partial-evaluation framework for LLM-driven automated algorithm design methods. Two instantiations are considered: AdaEva-S, based on successive halving, and AdaEva-R, based on statistical racing. The experiments suggest that adaptive evaluation generally gives a better breadth–depth trade-off than any single fixed evaluation depth.
Critical strengths
S1: The proposed AdaEva-S and AdaEva-R are modular and can be readily integrated into existing LLM-driven algorithm design frameworks.
S2: This manuscript provides the source code, improving the reproducibility.
Critical weaknesses
W1: The novelty is limited. The two core mechanisms underlying AdaEva-S and AdaEva-R are established techniques, namely successive halving and statistical racing, respectively. Therefore, the main technical contribution appears to lie primarily in adapting these existing mechanisms to the LLM4AD setting.
W2: The manuscript only compares AdaEva-S and AdaEva-R against Fixed-1, Fixed-25%, Fixed-50%, and full evaluation. However, it lacks comparisons with other strong or state-of-the-art adaptive allocation methods.
W3: The central advantage of partial evaluation is that it allows the search procedure to examine more candidate algorithms within a fixed number of candidate-instance evaluations. However, every additional candidate generally requires an additional LLM generation call. Consequently, matching only the number of candidate-instance evaluations does not imply matching total compute, wall-clock time, API cost, or token consumption.
Questions for authors
Please see Critical Weaknesses.
5 comments
Response to Reviewer qiv2 (Part 2/3)
W2: The manuscript only compares AdaEva-S and AdaEva-R against Fixed-1, Fixed-25%, Fixed-50%, and full evaluation. However, it lacks comparisons with other strong or state-of-the-art adaptive allocation methods.
We thank the reviewer for raising this concern. The concern about comparisons with existing adaptive evaluation methods is closely related to a point raised in another review. We provide the same clarification here for completeness:
We have added cascading evaluation, a multi-stage partial-evaluation mechanism used in prominent LLM-driven algorithm discovery systems such as AlphaEvolve and OpenEvolve, as an additional baseline. We chose cascading because it addresses the same problem as AdaEva (reducing candidate-evaluation costs through progressive evaluation) and is an established approach in LLM-driven program search.
Unlike AdaEva, which compares candidates using accumulated evaluation evidence, cascading advances candidates according to predefined absolute performance thresholds. Such thresholds can be challenging to calibrate and may require problem-specific knowledge. In contrast, AdaEva does not require absolute performance thresholds, and we use a single default configuration across all tested frameworks and benchmarks without setting-specific tuning. We evaluate cascading on TSP and BBOB, where meaningful thresholds can be established.
The new results show that:
- AdaEva-R achieves better held-out mean ranks in five of six framework-backbone settings, tying cascading in the remaining setting. AdaEva-R generally maintains stronger anytime performance across evaluation-budget checkpoints, rather than improving only at the final budget.
- Cascading becomes less selective as search progresses. On BBOB with Qwen, its first-stage acceptance rate increases from 59.9% to 91.5%, reducing its ability to filter candidates.
- These results provide direct evidence that the adaptive evaluation used in AdaEva offers practical advantages over existing threshold-based partial-evaluation strategies, without requiring a problem-specific threshold. We have added the new results to Section 4.3: "Comparison with Cascading Partial Evaluation" (lines 455-485), of the main text, with implementation details and additional analyses in Appendix C.
| Framework | Benchmark | LLM | Fixed-1 | Fixed-25% | Fixed-50% | Full | Cascade | AdaEva-S | AdaEva-R |
|---|---|---|---|---|---|---|---|---|---|
| EoH | TSP | Devstral | 5.067 | 3.333 | 3.467 | 7.000 | 3.467 | 3.667 | 2.000 |
| EoH | TSP | Qwen | 4.167 | 2.867 | 2.900 | 6.567 | 5.133 | 3.500 | 2.867 |
| HiFo-Prompt | TSP | Devstral | 6.000 | 3.567 | 3.333 | 7.000 | 3.100 | 2.133 | 2.867 |
| HiFo-Prompt | TSP | Qwen | 7.000 | 3.367 | 3.000 | 4.667 | 4.733 | 2.833 | 2.400 |
| LLaMEA | BBOB | Devstral | 6.867 | 4.600 | 5.067 | 4.267 | 2.467 | 2.267 | 2.467 |
| LLaMEA | BBOB | Qwen | 7.000 | 4.567 | 5.200 | 3.800 | 3.433 | 2.333 | 1.667 |
[1] Novikov, Alexander, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov et al. "Alphaevolve: A coding agent for scientific and algorithmic discovery." arXiv preprint arXiv:2506.13131 (2025).
[2] Sharma, Asankhaya. "OpenEvolve: an open-source evolutionary coding agent." GitHub repository, https://github.com/algorithmicsuperintelligence/openevolve (2025).
Response to Reviewer qiv2 (Part 3/3)
W3: The central advantage of partial evaluation is that it allows the search procedure to examine more candidate algorithms within a fixed number of candidate-instance evaluations. However, every additional candidate generally requires an additional LLM generation call. Consequently, matching only the number of candidate-instance evaluations does not imply matching total compute, wall-clock time, API cost, or token consumption.
We thank the reviewer for raising this concern. This concern about accounting for the computational costs of additional LLM generations is also raised in another review. Our response is therefore consistent with the clarification provided there, which we reproduce below:
Generation cost per call is determined by the host framework, not the evaluation strategy, since all strategies use the framework's prompts unchanged. Total generation cost therefore scales with the number of candidates evaluated, as reported in Figure 4 of the original submission (now Figure 8 in the revised manuscript): on TSP, AdaEva-S and AdaEva-R evaluate 3.3× and 2.9× as many candidates as full-pool evaluation under EoH, and 4.2× and 3.4× as many under HiFo-Prompt.
Regarding the total-runtime budget that includes LLM sampling and evaluation runtime, AdaEva-R achieves the lowest total CPU time in three of the four framework--backbone combinations (19,834 s and 48,405 s for EoH with Devstral and Qwen, respectively, and 19,659 s for HiFo-Prompt with Qwen). Full-pool evaluation is fastest for HiFo-Prompt with Devstral at 23,104 s, so the runtime advantage depends on the setting, details are provided in Figure 9 and Appendix E in the revised manuscript.
| Framework | LLM | Fixed-1 | Fixed-25% | Fixed-50% | Full | AdaEva-S | AdaEva-R |
|---|---|---|---|---|---|---|---|
| EoH | Devstral | 99209 | 81548 | 47746 | 24884 | 96953 | 19834 |
| Qwen | 96697 | 56074 | 49186 | 111842 | 52714 | 48405 | |
| HiFo-Prompt | Devstral | 129122 | 43070 | 24463 | 23104 | 45749 | 28164 |
| Qwen | 66298 | 37400 | 24895 | 20516 | 23787 | 19659 |
In terms of ranking performance under matched total runtime, at least one AdaEva variant achieves the best held-out mean rank in each of the four combinations. Averaged across the combinations, AdaEva-R achieves the lowest mean rank (2.208), followed by AdaEva-S (2.417), compared with 2.958 for the best fixed strategy.
| Framework | LLM | Fixed-1 | Fixed-25% | Fixed-50% | Full | AdaEva-S | AdaEva-R |
|---|---|---|---|---|---|---|---|
| EoH / TSP | Devstral | 4.000 | 3.833 | 4.000 | 6.000 | 2.167 | 1.000 |
| EoH / TSP | Qwen | 5.500 | 3.167 | 2.500 | 4.500 | 3.500 | 1.833 |
| HiFo-Prompt / TSP | Devstral | 6.000 | 4.000 | 2.000 | 3.333 | 1.667 | 4.000 |
| HiFo-Prompt / TSP | Qwen | 6.000 | 3.500 | 3.333 | 3.833 | 2.333 | 2.000 |
| Mean | 5.375 | 3.625 | 2.958 | 4.417 | 2.417 | 2.208 |
Response to Reviewer qiv2 (Part 1/3)
We thank the reviewer for the constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns regarding methodological novelty, comparisons with existing adaptive evaluation methods, and computational costs. We respond to each point below.
We thank the reviewer for raising this concern. The question of methodological novelty is closely related to a point raised in another review, and our response is the same in substance. For completeness, we restate our clarification here:
We agree that successive halving and iterated racing are established techniques, and we do not claim novelty in their elimination rules. These mechanisms do not inherently depend on LLM-generated candidates and could also be applied to conventional evolutionary algorithms.
Our contribution is to make evaluation depth (the number of problem instances on which a candidate algorithm is evaluated) an explicit, adaptive decision in LLM-driven algorithm design and demonstrate why this matters across different search frameworks and benchmarks. This exposes three findings that a system tied to a fixed evaluation policy cannot address:
The optimal fixed evaluation depth does not transfer across settings. Fixed-1 performs best for EoH/OBP but worst for LLaMEA/BBOB, whereas full evaluation exhibits the opposite pattern. This demonstrates that selecting an evaluation depth in advance is unreliable, while AdaEva adapts without setting-specific depth tuning.
Existing threshold-based partial evaluation can become less selective during search. Our new AlphaEvolve-style cascading experiments show that fixed thresholds lose discriminative power as candidate quality improves.
Adaptive evaluation can be decoupled from candidate generation. AdaEva operates as a drop-in evaluation layer, keeping the host framework’s generation, evolution, and per-instance evaluation procedures. This enables controlled comparisons and integration across different LLM4AD frameworks.
We therefore position AdaEva not as a new racing algorithm, but as a general framework supported by empirical insights into when and why adaptive allocation benefits LLM-driven algorithm design.