acceptodds
Under review as a conference paper at ICLR 2027

AdaEva: Accelerating LLM-Driven Algorithm Design with Adaptive Partial Evaluation

Abstract

Large Language Models (LLMs) are increasingly used for automated algorithm design. However the computational cost of evaluating the generated algorithms can be excessive. We consider the common LLM-driven automated algorithm design (LLM4AD) setting in which a candidate algorithm is evaluated by aggregating its performance over a shared set of training instances. This instance-wise structure raises a natural question: must every candidate be evaluated on the entire instance set before deciding whether it remains competitive? Taking inspiration from algorithm configuration, we introduce AdaEva, a drop-in adaptive partial-evaluation framework that progressively evaluates candidates on larger subsets of the same instance pool and eliminates unpromising candidates as evidence accumulates. Importantly, AdaEva leaves the underlying LLM4AD procedure and per-instance evaluator unchanged and requires no prior knowledge about instance difficulty. We instantiate this idea using successive halving (AdaEva-S) and statistical racing (AdaEva-R), and evaluate both mechanisms across three representative LLM4AD frameworks, multiple LLM backbones, and optimization domains spanning combinatorial and continuous black-box optimization. Under matched evaluation budgets, AdaEva more reliably balances evaluation effort across candidates than fixed partial-evaluation strategies, yielding strong search efficiency and anytime performance together with improved held-out generalization across the evaluated settings.

Primary area:
optimization
Keywords:
LLMs for algorithm design

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qiv2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces AdaEva, a drop-in adaptive partial-evaluation framework for LLM-driven automated algorithm design methods. Two instantiations are considered: AdaEva-S, based on successive halving, and AdaEva-R, based on statistical racing. The experiments suggest that adaptive evaluation generally gives a better breadth–depth trade-off than any single fixed evaluation depth.

Critical strengths

S1: The proposed AdaEva-S and AdaEva-R are modular and can be readily integrated into existing LLM-driven algorithm design frameworks.

S2: This manuscript provides the source code, improving the reproducibility.

Critical weaknesses

W1: The novelty is limited. The two core mechanisms underlying AdaEva-S and AdaEva-R are established techniques, namely successive halving and statistical racing, respectively. Therefore, the main technical contribution appears to lie primarily in adapting these existing mechanisms to the LLM4AD setting.

W2: The manuscript only compares AdaEva-S and AdaEva-R against Fixed-1, Fixed-25%, Fixed-50%, and full evaluation. However, it lacks comparisons with other strong or state-of-the-art adaptive allocation methods.

W3: The central advantage of partial evaluation is that it allows the search procedure to examine more candidate algorithms within a fixed number of candidate-instance evaluations. However, every additional candidate generally requires an additional LLM generation call. Consequently, matching only the number of candidate-instance evaluations does not imply matching total compute, wall-clock time, API cost, or token consumption.

Questions for authors

Please see Critical Weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer qiv2 (Part 1/3)

We thank the reviewer for the constructive feedback. We have revised the manuscript and conducted additional experiments to address the concerns regarding methodological novelty, comparisons with existing adaptive evaluation methods, and computational costs. We respond to each point below.

W1: Limited novelty

We thank the reviewer for raising this concern. The question of methodological novelty is closely related to a point raised in another review, and our response is the same in substance. For completeness, we restate our clarification here:

We agree that successive halving and iterated racing are established techniques, and we do not claim novelty in their elimination rules. These mechanisms do not inherently depend on LLM-generated candidates and could also be applied to conventional evolutionary algorithms.

Our contribution is to make evaluation depth (the number of problem instances on which a candidate algorithm is evaluated) an explicit, adaptive decision in LLM-driven algorithm design and demonstrate why this matters across different search frameworks and benchmarks. This exposes three findings that a system tied to a fixed evaluation policy cannot address:

  • The optimal fixed evaluation depth does not transfer across settings. Fixed-1 performs best for EoH/OBP but worst for LLaMEA/BBOB, whereas full evaluation exhibits the opposite pattern. This demonstrates that selecting an evaluation depth in advance is unreliable, while AdaEva adapts without setting-specific depth tuning.

  • Existing threshold-based partial evaluation can become less selective during search. Our new AlphaEvolve-style cascading experiments show that fixed thresholds lose discriminative power as candidate quality improves.

  • Adaptive evaluation can be decoupled from candidate generation. AdaEva operates as a drop-in evaluation layer, keeping the host framework’s generation, evolution, and per-instance evaluation procedures. This enables controlled comparisons and integration across different LLM4AD frameworks.

We therefore position AdaEva not as a new racing algorithm, but as a general framework supported by empirical insights into when and why adaptive allocation benefits LLM-driven algorithm design.

AuthorsOpenReview ↗

Response to Reviewer qiv2 (Part 2/3)

W2: The manuscript only compares AdaEva-S and AdaEva-R against Fixed-1, Fixed-25%, Fixed-50%, and full evaluation. However, it lacks comparisons with other strong or state-of-the-art adaptive allocation methods.

We thank the reviewer for raising this concern. The concern about comparisons with existing adaptive evaluation methods is closely related to a point raised in another review. We provide the same clarification here for completeness:

We have added cascading evaluation, a multi-stage partial-evaluation mechanism used in prominent LLM-driven algorithm discovery systems such as AlphaEvolve and OpenEvolve, as an additional baseline. We chose cascading because it addresses the same problem as AdaEva (reducing candidate-evaluation costs through progressive evaluation) and is an established approach in LLM-driven program search.

Unlike AdaEva, which compares candidates using accumulated evaluation evidence, cascading advances candidates according to predefined absolute performance thresholds. Such thresholds can be challenging to calibrate and may require problem-specific knowledge. In contrast, AdaEva does not require absolute performance thresholds, and we use a single default configuration across all tested frameworks and benchmarks without setting-specific tuning. We evaluate cascading on TSP and BBOB, where meaningful thresholds can be established.

The new results show that:

  • AdaEva-R achieves better held-out mean ranks in five of six framework-backbone settings, tying cascading in the remaining setting. AdaEva-R generally maintains stronger anytime performance across evaluation-budget checkpoints, rather than improving only at the final budget.
  • Cascading becomes less selective as search progresses. On BBOB with Qwen, its first-stage acceptance rate increases from 59.9% to 91.5%, reducing its ability to filter candidates.
  • These results provide direct evidence that the adaptive evaluation used in AdaEva offers practical advantages over existing threshold-based partial-evaluation strategies, without requiring a problem-specific threshold. We have added the new results to Section 4.3: "Comparison with Cascading Partial Evaluation" (lines 455-485), of the main text, with implementation details and additional analyses in Appendix C.
FrameworkBenchmarkLLMFixed-1Fixed-25%Fixed-50%FullCascadeAdaEva-SAdaEva-R
EoHTSPDevstral5.0673.3333.4677.0003.4673.6672.000
EoHTSPQwen4.1672.8672.9006.5675.1333.5002.867
HiFo-PromptTSPDevstral6.0003.5673.3337.0003.1002.1332.867
HiFo-PromptTSPQwen7.0003.3673.0004.6674.7332.8332.400
LLaMEABBOBDevstral6.8674.6005.0674.2672.4672.2672.467
LLaMEABBOBQwen7.0004.5675.2003.8003.4332.3331.667

[1] Novikov, Alexander, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov et al. "Alphaevolve: A coding agent for scientific and algorithmic discovery." arXiv preprint arXiv:2506.13131 (2025).

[2] Sharma, Asankhaya. "OpenEvolve: an open-source evolutionary coding agent." GitHub repository, https://github.com/algorithmicsuperintelligence/openevolve (2025).

AuthorsOpenReview ↗

Response to Reviewer qiv2 (Part 3/3)

W3: The central advantage of partial evaluation is that it allows the search procedure to examine more candidate algorithms within a fixed number of candidate-instance evaluations. However, every additional candidate generally requires an additional LLM generation call. Consequently, matching only the number of candidate-instance evaluations does not imply matching total compute, wall-clock time, API cost, or token consumption.

We thank the reviewer for raising this concern. This concern about accounting for the computational costs of additional LLM generations is also raised in another review. Our response is therefore consistent with the clarification provided there, which we reproduce below:

Generation cost per call is determined by the host framework, not the evaluation strategy, since all strategies use the framework's prompts unchanged. Total generation cost therefore scales with the number of candidates evaluated, as reported in Figure 4 of the original submission (now Figure 8 in the revised manuscript): on TSP, AdaEva-S and AdaEva-R evaluate 3.3× and 2.9× as many candidates as full-pool evaluation under EoH, and 4.2× and 3.4× as many under HiFo-Prompt.

Regarding the total-runtime budget that includes LLM sampling and evaluation runtime, AdaEva-R achieves the lowest total CPU time in three of the four framework--backbone combinations (19,834 s and 48,405 s for EoH with Devstral and Qwen, respectively, and 19,659 s for HiFo-Prompt with Qwen). Full-pool evaluation is fastest for HiFo-Prompt with Devstral at 23,104 s, so the runtime advantage depends on the setting, details are provided in Figure 9 and Appendix E in the revised manuscript.

FrameworkLLMFixed-1Fixed-25%Fixed-50%FullAdaEva-SAdaEva-R
EoHDevstral992098154847746248849695319834
Qwen9669756074491861118425271448405
HiFo-PromptDevstral1291224307024463231044574928164
Qwen662983740024895205162378719659

In terms of ranking performance under matched total runtime, at least one AdaEva variant achieves the best held-out mean rank in each of the four combinations. Averaged across the combinations, AdaEva-R achieves the lowest mean rank (2.208), followed by AdaEva-S (2.417), compared with 2.958 for the best fixed strategy.

FrameworkLLMFixed-1Fixed-25%Fixed-50%FullAdaEva-SAdaEva-R
EoH / TSPDevstral4.0003.8334.0006.0002.1671.000
EoH / TSPQwen5.5003.1672.5004.5003.5001.833
HiFo-Prompt / TSPDevstral6.0004.0002.0003.3331.6674.000
HiFo-Prompt / TSPQwen6.0003.5003.3333.8332.3332.000
Mean5.3753.6252.9584.4172.4172.208
Reviewer_TFpaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Observing that the evaluation cost is high in LLM-based AHD, the paper proposes AdaEva, a framework for progressively evaluating instances. By framing the partial evaluation problem as a resource allocation problem, the paper implements two standard techniques, successive halving and statistical racing, to decide the evaluation progress. Across three AHD frameworks and four tasks, AdaEva shows the best overall performance on the same budget compared with full and fixed partial baselines.

Critical strengths

The problem is well motivated and meaningful, and the findings help practitioners.

The paper transforms the problem of evaluation into a resource allocation problem, which is a sharp move.

The paper conducts comprehensive experiments across different frameworks, problem types, and backbones, supporting the results.

Critical weaknesses
  1. Limited methodological novelty. AdaEva applies successive halving and irace-style racing with their standard configurations, while no specific LLM4AD component is developed. The partial-evaluation-aware prompt is dropped, as it was found to hurt. Related systems such as AlphaEvolve, HeuriGym, and GEPA, as mentioned in the related work section, treat partial evaluation as an implementation choice within a broader contribution, and this paper doesn’t show that promoting it to the main contribution yields new methodological insight.

  2. Baselines do not include existing partial-evaluation schemes. The finding that adaptive allocation beats non-adaptive allocation is already well established in the algorithm configuration and hyperparameter optimization literature. The relevant question is whether racing or successive halving improves on the partial-evaluation schemes already used in LLM-driven search. The authors should compare with related systems to report whether AdaEva could match or outperform them with the benefit of keeping both candidate generation and the per-instance evaluator, rather than simply dropping them

  3. The choice of the backbones limits how far the conclusions generalize. The main experiments use Devstral and Qwen2.5-Coder, with Gemini-3.1-Flash-Lite on a subset of settings, all capped at 2,048 generated tokens. These are relatively small or older models compared with those commonly used in current LLM4AD work. AdaEva's main advantage is early elimination of bad candidates, allowing it to evaluate more candidates effectively. The model matters in two ways. First, the latest models are more powerful and may produce fewer bad candidates in the first generation, so that partial evaluation methods like AdaEva degenerate closer to full evaluation. Second, the inference cost of the latest model increases compared with older models. When matching the budget, even early elimination helps a lot; AdaEva evaluates fewer candidates, which might influence performance.

Questions for authors
  1. What aspect of AdaEva depends on the candidates being LLM-generated? How would AdaEva differ from applying irace-style racing or successive halving to the selection step of any evolutionary algorithm?
  2. For each strategy, how many LLM calls (or tokens) and how much wall-clock time does a run use? Do the conclusions hold when strategies are matched on total cost rather than evaluation budget?
  3. Does the advantage persist with a stronger current model? Does the fraction of candidates eliminated after the first evaluation block differ across the three backbones used?
6 comments
AuthorsOpenReview ↗

Response to Reviewer TFpa (Part 1/5)

We thank the reviewer for the constructive feedback and for recognizing the motivation, practical relevance, and experimental breadth of our work. We have revised the manuscript and conducted additional experiments to address the concerns. We respond to each point below.

W1 and Q1: Limited methodological novelty

We agree that successive halving and iterated racing are established techniques, and we do not claim novelty in their elimination rules. These mechanisms do not inherently depend on LLM-generated candidates and could also be applied to conventional evolutionary algorithms.

Our contribution is to make evaluation depth(the number of problem instances on which a candidate algorithm is evaluated) an explicit, adaptive decision in LLM-driven algorithm design and demonstrate why this matters across different search frameworks and benchmarks. This exposes three findings that a system tied to a fixed evaluation policy cannot address:

  • The optimal fixed evaluation depth does not transfer across settings. Fixed-1 performs best for EoH/OBP but worst for LLaMEA/BBOB, whereas full evaluation exhibits the opposite pattern. This demonstrates that selecting an evaluation depth in advance is unreliable, while AdaEva adapts without setting-specific depth tuning.

  • Existing threshold-based partial evaluation can become less selective during search. Our new AlphaEvolve-style cascading experiments show that fixed thresholds lose discriminative power as candidate quality improves.

  • Adaptive evaluation can be decoupled from candidate generation. AdaEva operates as a drop-in evaluation layer, keeping the host framework’s generation, evolution, and per-instance evaluation procedures. This enables controlled comparisons and integration across different LLM4AD frameworks.

We therefore position AdaEva not as a new racing algorithm, but as a general framework supported by empirical insights into when and why adaptive allocation benefits LLM-driven algorithm design.

AuthorsOpenReview ↗

Response to Reviewer TFpa (Part 2/5)

W2: Baselines do not include existing partial-evaluation schemes. The finding that adaptive allocation beats non-adaptive allocation is already well established in the algorithm configuration and hyperparameter optimization literature. The relevant question is whether racing or successive halving improves on the partial-evaluation schemes already used in LLM-driven search. The authors should compare with related systems to report whether AdaEva could match or outperform them with the benefit of keeping both candidate generation and the per-instance evaluator, rather than simply dropping them

We thank the reviewer for this suggestion. We have added cascading evaluation, a multi-stage partial-evaluation mechanism used in prominent LLM-driven algorithm discovery systems such as AlphaEvolve [1] and OpenEvolve [2], as an additional baseline. We chose cascading because it addresses the same problem as AdaEva (reducing candidate-evaluation costs through progressive evaluation) and is an established approach in LLM-driven program search.

Unlike AdaEva, which compares candidates using accumulated evaluation evidence, cascading advances candidates according to predefined absolute performance thresholds. Such thresholds can be challenging to calibrate and may require problem-specific knowledge. In contrast, AdaEva does not require absolute performance thresholds, and we use a single default configuration across all tested frameworks and benchmarks without setting-specific tuning. We evaluate cascading on TSP and BBOB, where meaningful thresholds can be established.

The new results show that:

  • AdaEva-R achieves better held-out mean ranks in five of six framework-backbone settings, tying cascading in the remaining setting. AdaEva-R generally maintains stronger anytime performance across evaluation-budget checkpoints, rather than improving only at the final budget.
  • Cascading becomes less selective as search progresses. On BBOB with Qwen, its first-stage acceptance rate increases from 59.9% to 91.5%, reducing its ability to filter candidates.
  • These results provide direct evidence that the adaptive evaluation used in AdaEva offers practical advantages over existing threshold-based partial-evaluation strategies, without requiring a problem-specific threshold. We have added the new results to Section 4.3: "Comparison with Cascading Partial Evaluation" (lines 455-485), of the main text, with implementation details and additional analyses in Appendix C.
FrameworkBenchmarkLLMFixed-1Fixed-25%Fixed-50%FullCascadeAdaEva-SAdaEva-R
EoHTSPDevstral5.0673.3333.4677.0003.4673.6672.000
EoHTSPQwen4.1672.8672.9006.5675.1333.5002.867
HiFo-PromptTSPDevstral6.0003.5673.3337.0003.1002.1332.867
HiFo-PromptTSPQwen7.0003.3673.0004.6674.7332.8332.400
LLaMEABBOBDevstral6.8674.6005.0674.2672.4672.2672.467
LLaMEABBOBQwen7.0004.5675.2003.8003.4332.3331.667

[1] Novikov, Alexander, Ngân Vũ, Marvin Eisenberger, Emilien Dupont, Po-Sen Huang, Adam Zsolt Wagner, Sergey Shirobokov et al. "Alphaevolve: A coding agent for scientific and algorithmic discovery." arXiv preprint arXiv:2506.13131 (2025).

[2] Sharma, Asankhaya. "OpenEvolve: an open-source evolutionary coding agent." GitHub repository, https://github.com/algorithmicsuperintelligence/openevolve (2025).

AuthorsOpenReview ↗

Response to Reviewer TFpa (Part 3/5)

W3: The choice of the backbones limits how far the conclusions generalize. The main experiments use Devstral and Qwen2.5-Coder, with Gemini-3.1-Flash-Lite on a subset of settings, all capped at 2,048 generated tokens. These are relatively small or older models compared with those commonly used in current LLM4AD work. AdaEva's main advantage is early elimination of bad candidates, allowing it to evaluate more candidates effectively. The model matters in two ways. First, the latest models are more powerful and may produce fewer bad candidates in the first generation, so that partial evaluation methods like AdaEva degenerate closer to full evaluation. Second, the inference cost of the latest model increases compared with older models. When matching the budget, even early elimination helps a lot; AdaEva evaluates fewer candidates, which might influence performance.

We thank the reviewer for highlighting the potential influence of LLM capability on adaptive evaluation. We agree that stronger generators may produce different distributions of candidate quality, potentially affecting both early elimination and the trade-off between generation and evaluation costs.

Our experiments cover three LLM backbones, including Gemini-3.1-Flash-Lite (in a non-reasoning configuration). While these provide evidence across different generators, we acknowledge that they do not establish generalisation to substantially stronger reasoning models. Importantly, the effectiveness of early elimination depends not only on the absolute quality of generated candidates, but also on how reliably their relative performance can be distinguished using partial evaluations. Stronger generators do not necessarily eliminate the opportunity for adaptive evaluation, although further experiments are needed to establish this.

Regarding the 2,048-token generation cap, average completion lengths are well below this limit for EoH and HiFo-Prompt, although it may be more consequential for LLaMEA/BBOB. We discuss backbone-dependent evaluation behaviour in Q3 and the impact of inference costs in Q2.

FrameworkBenchmarkLLMAvg. Prompt Tokens / CallAvg. Completion Tokens / Call
EoHFSSPDevstral1,692.2 ± 227.3461.7 ± 53.1
Qwen1,112.1 ± 74.7376.9 ± 29.0
OBPDevstral580.3 ± 118.8167.5 ± 27.7
Qwen459.2 ± 62.7172.1 ± 11.8
TSPDevstral1,412.2 ± 265.0390.1 ± 66.0
Qwen1,367.1 ± 129.9446.4 ± 44.9
HiFo-PromptFSSPDevstral1,603.4 ± 188.2394.5 ± 33.1
Qwen1,706.0 ± 168.4492.2 ± 59.2
OBPDevstral1,313.7 ± 137.3287.9 ± 52.9
Qwen1,148.6 ± 164.0297.3 ± 42.0
TSPDevstral1,668.4 ± 146.1375.6 ± 50.8
Qwen1,384.5 ± 95.0362.9 ± 41.6
LLaMEABBOBDevstral2,499.9 ± 291.21,404.4 ± 274.7
Qwen3,314.4 ± 799.71,569.8 ± 261.4

Q1: What aspect of AdaEva depends on the candidates being LLM-generated? How would AdaEva differ from applying irace-style racing or successive halving to the selection step of any evolutionary algorithm?

We thank the reviewer for raising the concern of novelty in our contributions. Three aspects that distinguish our setting from racing over an evolutionary algorithm's selection step are:

  • The candidate set is produced by an external generator the evaluation layer does not control and cannot resample from. Unlike irace [1] there is no model to update from the race's outcome.

  • The candidates are executable programs that frequently fail to compile, crash, or time out, so the contender set is heterogeneous in a way parameter configurations are not.

  • The host framework consumes evaluation results in its prompts and selection, so partial evaluation changes what the generator is conditioned on, which does not arise when racing is internal to the search procedure.

[1] López-Ibáñez, M., Dubois-Lacoste, J., Cáceres, L. P., Birattari, M., & Stützle, T. (2016). The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3, 43-58.

Reviewer_tiTDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AdaEva, an adaptive partial-evaluation framework for evolutionary LLM-driven automated algorithm design. AdaEva progressively evaluates candidates on subsets of a shared instance pool and eliminates unpromising candidates using successive halving (AdaEva-S) or statistical racing (AdaEva-R). The authors compare both variants with Fixed-K evaluation in EoH, HiFo-Prompt, and LLaMEA across combinatorial and continuous black-box optimization. Under matched evaluation budgets, the results show improved search progress and held-out generalization in several settings.

Critical strengths

This work formulates instance-wise candidate evaluation in evolutionary LLM4AD as an adaptive resource-allocation problem, introduces a separate evaluation layer, and makes this idea applicable across different search frameworks.

Critical weaknesses
  1. The experiments match candidate-instance evaluation counts without accounting for the additional LLM inference cost of generating more candidates. The results therefore establish improved evaluation-budget efficiency, while the benefit under a fixed total runtime remains unclear.

  2. The submitted PDF modified the official ICLR 2027 template by shrinking the left and right margins.

Questions for authors
  1. In Algorithm 1, line 6 performs a cap elimination so that at least M candidates remain. Section 3.2 discusses refilling the surviving set to M, and Section 4.3 states that it uses previously eliminated candidates for refill. Can AdaEva-R eliminate candidates until fewer than M remain? If so, should we modify Algorithm 1? If not, when does population refill occur, and how are candidates selected for refill?

  2. Section 4.1 states that AdaEva-R ranks among the top two in 13 of 14 settings. Table 2 gives 10 of 14 for AdaEva-R. Is there a misstatement?

  3. What fractions of the total runtime are spent on LLM generation and candidate evaluation? Do the adaptive methods retain their advantage under the same total runtime?

2 comments
AuthorsOpenReview ↗

Response to Reviewer tiTD (Part 1/2)

We thank the reviewer for the careful reading and detailed feedback. We have revised the manuscript to address the concerns, including correcting the formatting issue, clarifying the methodology, and providing additional experiments and analyses. We respond to each point below.

 

The submitted PDF modified the official ICLR 2027 template by shrinking the left and right margins.

We thank the reviewer for catching this. The submission was built from an incorrect template file that altered the left and right margins. This was an unintentional error on our side and not an attempt to gain space. We sincerely apologize for the oversight.

The revised manuscript now uses the official ICLR 2027 style and complies with the 10-page limit for the rebuttal phase. We moved the detailed parameter sensitivity table and the candidate count figure (Table 4 and Figure 4 in the original submission) to Appendix E3 and G, respectively, while retaining the corresponding findings and conclusions in the main text.

Additionally, we have added a new Section 4.3: “Comparison with Cascading Partial Evaluation” (lines 455-485), to the main text to address concerns regarding the lack of comparison with existing partial evaluation mechanisms.

All newly added or modified content is highlighted in blue for the reviewer's convenience.

 

W1: The experiments match candidate-instance evaluation counts without accounting for the additional LLM inference cost of generating more candidates. The results therefore establish improved evaluation-budget efficiency, while the benefit under a fixed total runtime remains unclear.

We agree with the reviewer that matching candidate–instance evaluations does not necessarily imply matching total computational cost. Strategies that evaluate more candidates may also incur additional LLM inference costs. To address this, we have added a breakdown of LLM generation and candidate-evaluation costs (Figure 9), and a comparison under matched total runtime (Appendix E.2, Table 10), accounting for both LLM generation and candidate evaluation for the TSP experiments on EoH/HiFo-Prompt with two LLM backbones (Devstral and Qwen)

In those experiments, LLM generation accounts for 0.5-7% of the total runtime in the original setup, which is the full evaluation. Although the per-call prompting procedure is unchanged, the total generation cost varies with the number of candidates produced.

Importantly, the advantage of adaptive evaluation over fixed-depth evaluation persists when both costs are accounted for. Under the runtime-aligned comparison, at least one AdaEva variant outperforms all fixed-depth baselines in each of the four TSP framework–backbone settings. An AdaEva variant achieves the best held-out mean rank in three of the four settings and ranks among the top two in all four. These results provide evidence that the benefits of adaptive evaluation extend beyond evaluation-budget efficiency, although the runtime comparison is currently limited to TSP.

 

Q3: What fractions of the total runtime are spent on LLM generation and candidate evaluation? Do the adaptive methods retain their advantage under the same total runtime?

We thank the reviewer for this question, which is closely related to W1. The TSP results in the table below show that at least one AdaEva variant ranks among the top two methods in all four framework–backbone settings and outperforms every fixed-depth baseline. Thus, the advantage over fixed evaluation depths persists in these settings when LLM inference costs are included.

SettingLLMFixed-1Fixed-25%Fixed-50%FullAdaEva-SAdaEva-R
EoH / TSPDevstral4.0003.8334.0006.0002.1671.000
EoH / TSPQwen5.5003.1672.5004.5003.5001.833
HiFo-Prompt / TSPDevstral6.0004.0002.0003.3331.6674.000
HiFo-Prompt / TSPQwen6.0003.5003.3333.8332.3332.000
Mean5.3753.6252.9584.4172.4172.208
AuthorsOpenReview ↗

Response to Reviewer tiTD (Part 2/2)

Q1: In Algorithm 1, line 6 performs a cap elimination so that at least M candidates remain. Section 3.2 discusses refilling the surviving set to M, and Section 4.3 states that it uses previously eliminated candidates for refill. Can AdaEva-R eliminate candidates until fewer than M remain? If so, should we modify Algorithm 1? If not, when does population refill occur, and how are candidates selected for refill?

We thank the reviewer for identifying this inconsistency. In the default configuration, line 6 of Algorithm 1 caps the number of eliminated candidates at (∣V(t)∣−M)(|\mathcal V^{(t)}|-M). Consequently, elimination cannot reduce the survivor set below (MM), and no previously eliminated candidates need to be reintroduced.

The ablation previously labelled "Population Refill" examines the effect of removing this constraint. When disabled, the elimination rule can remove all candidates identified as inferior, potentially leaving fewer than (MM) survivors, which the host framework then carries forward as a smaller population. We agree that the original terminology was misleading and have clarified the distinction in the revised manuscript (Section 3.2 and Appendix G).

 

Q2: Section 4.1 states that AdaEva-R ranks among the top two in 13 of 14 settings. Table 2 gives 10 of 14 for AdaEva-R. Is there a misstatement?

We thank the reviewer for identifying this error. The original statement incorrectly attributed the combined results of the two AdaEva variants to AdaEva-R alone. As shown in Table 2, AdaEva-R achieves the best mean rank in 8 of the 14 settings and ranks among the top two in 10. When considering both variants together, at least one AdaEva variant achieves the best mean rank in 11 settings and ranks among the top two in 13.

We have corrected Section 4.1 (lines 318–320) to distinguish these results explicitly. We apologise for the inaccurate statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.