acceptodds
Under review as a conference paper at ICLR 2027

EvoAlloc: A Self-Evolving Resource Allocation Agent for Efficient Program Evolution

Abstract

LLM-based program evolution relies on evaluation feedback to guide the iterative search for high-performing programs. However, evaluation is often computationally expensive, making it essential to allocate limited resources to candidates that can most effectively advance the search. Existing LLM-based methods typically rely on fixed allocation strategies throughout the search, potentially wasting resources on low-value candidates while overlooking promising ones. We propose EvoAlloc, a self-evolving resource-allocation agent that learns from search experience to revise its strategy for allocating computational resources across candidates. EvoAlloc periodically consolidates prior search and allocation outcomes into reusable experience, which informs subsequent strategy revisions. It further uses a counterfactual exploration mechanism to occasionally evaluate candidates denied resources by the allocator, revealing their outcomes to enrich its experience for future strategy updates. Across coding and agent-harness optimization benchmarks, EvoAlloc requires 59–82% fewer full evaluations and 61–89% fewer total LLM tokens to reach baseline-level performance. Moreover, under the same full-evaluation budget, EvoAlloc achieves 8.7–12.0% higher final performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
program evolution, large language models, adaptive resource allocation, self-improving systems, sample efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jQVNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoAlloc, which adjusts candidate-evaluation strategies using search history to choose full evaluation, partial evaluation, or skipping during program evolution. It compares incumbent and challenger strategies through shared evaluations and randomly evaluates some skipped candidates to inform subsequent allocation. Five-seed mean results on ADAS-AIME and Circle Packing show improved final performance at the same full-evaluation budget, and fewer online evaluations and tokens for the mean search trajectory to reach the baseline score.

Critical strengths
  1. Random exploration addresses missing feedback from candidate filtering: skipped candidates would otherwise provide no outcomes, while exploration evaluates some of them and informs later strategies. It recovers 6 and 22 candidates that improve the current best score on the two tasks, demonstrating concrete search value.

  2. The complete method provides practical gains: at the same full-evaluation budget, mean final performance is 39.56% versus 35.33% and 2.547 versus 2.343 for ShinkaEvolve. Retaining explored candidates in search while removing only their explicit feedback lowers performance on both tasks, providing more targeted evidence for exploration feedback than simply removing exploration.

Critical weaknesses
  1. Online evolution is a central contribution, but the comparisons mix the effects of updating and initial guidance. One-Step removes both the strategy and experience; w/o Evolution uses manually refined fixed guidance but starts with a medium-selectivity strategy and seeded experience, whereas the full method starts with low selectivity and empty experience (Appendices A, B.1, and D.1). These results support the complete allocator over the selected alternatives, but do not clearly establish how much of the gain comes from online evolution itself.

  2. The resource advantage requires distinguishing target-reaching cost from the total investment in a complete search. Appendix D.3 discloses that EvoAlloc uses more tokens when the same full-evaluation budget is exhausted, for example 10.85M versus 6.11M for ShinkaEvolve on Circle Packing; the final-performance gain therefore cannot directly be interpreted as a gain at equal total compute. Table 5 and the token curves still support lower online target-reaching costs, but this advantage excludes the cost of obtaining 50 initialization examples, limiting the assessment of overall efficiency from scratch.

Questions for authors

None.

2 comments
AuthorsOpenReview ↗

Response to Reviewer jQVN (1/2)

We thank Reviewer jQVN for the careful reading of our paper. We address the comments below.

W1: Online evolution is a central contribution, but the comparisons mix the effects of updating and initial guidance. One-Step removes both the strategy and experience; w/o Evolution uses manually refined fixed guidance but starts with a medium-selectivity strategy and seeded experience, whereas the full method starts with low selectivity and empty experience (Appendices A, B.1, and D.1). These results support the complete allocator over the selected alternatives, but do not clearly establish how much of the gain comes from online evolution itself.

Answer:

We thank the reviewer for this helpful suggestion. We have added w/o Evolution (matched initialization), which disables online evolution while retaining the same initialization as EvoAlloc. With initialization held fixed, enabling online evolution improves mean performance by 17.1% on ADAS-AIME, as shown in the table below.

The w/o Evolution variant in the original submission used manually refined initialization with the aim of providing a competitive fixed allocator. We agree with the reviewer that matching initialization better isolates the contribution of online evolution, and have added this comparison to Appendix D.5 of the revised paper.

Table R3-1. Ablation of online evolution. Results are mean ±\pm std over five seeds.

VariantOnline evolutionInitializationADAS-AIME (%)Circle Packing
Full Method✓\checkmarkLow selectivity + empty Experiences39.56±6.80\mathbf{39.56_{\pm 6.80}}2.55±0.03\mathbf{2.55_{\pm 0.03}}
w/o Evolution (matched initialization; new)×\timesLow selectivity + empty Experiences33.78±2.1833.78_{\pm 2.18}2.49±0.052.49_{\pm 0.05}
w/o Evolution (refined initialization; in original submission)×\timesMedium selectivity + seeded Experiences35.78±1.0935.78_{\pm 1.09}2.46±0.092.46_{\pm 0.09}
AuthorsOpenReview ↗

Response to Reviewer jQVN (2/2)

W2.1: The resource advantage requires distinguishing target-reaching cost from the total investment in a complete search. Appendix D.3 discloses that EvoAlloc uses more tokens when the same full-evaluation budget is exhausted, for example 10.85M versus 6.11M for ShinkaEvolve on Circle Packing; the final-performance gain therefore cannot directly be interpreted as a gain at equal total compute.

Answer:

We thank the reviewer for this careful reading.

  • Our original comparison follows standard practice in program evolution [r1], evaluating performance under matched evaluation counts. EvoAlloc achieves better performance under this protocol.

  • We believe that token efficiency is more fairly assessed by comparing performance at the same token cost, rather than by comparing token cost under matched evaluation counts as in the reviewer's comment. Figure 3 in the original submission already reports performance against token cost; Table R3-2 summarizes the corresponding comparisons. EvoAlloc outperforms ShinkaEvolve by 8.1% on average at the same token cost, close to the 9.2% improvement under matched evaluation counts.

Table R3-2. Performance at the same token cost (mean ±\pm std over five seeds). For each benchmark, we use ShinkaEvolve's average token cost as a common budget and report each method's best performance within it.

MethodADAS-AIME (%)Circle PackingHotpotQA (%)SWE-PolyBench (%)
ShinkaEvolve34.89±2.0634.89_{\pm 2.06}2.335±0.2682.335_{\pm 0.268}63.20±1.5863.20_{\pm 1.58}16.25±1.6616.25_{\pm 1.66}
One-Step Alloc34.89±1.9434.89_{\pm 1.94}2.361±0.2522.361_{\pm 0.252}64.27±1.2464.27_{\pm 1.24}16.15±1.6516.15_{\pm 1.65}
Lineage-UCB34.00±0.8934.00_{\pm 0.89}2.406±0.1302.406_{\pm 0.130}65.42±2.4665.42_{\pm 2.46}15.00±1.6615.00_{\pm 1.66}
RPM33.33±1.2233.33_{\pm 1.22}2.120±0.1762.120_{\pm 0.176}64.76±2.2764.76_{\pm 2.27}15.94±1.5015.94_{\pm 1.50}
EvoAlloc (ours)37.11±4.69\mathbf{37.11_{\pm 4.69}}2.534±0.020\mathbf{2.534_{\pm 0.020}}67.02±1.72\mathbf{67.02_{\pm 1.72}}18.12±1.33\mathbf{18.12_{\pm 1.33}}

W2.2: Table 5 and the token curves still support lower online target-reaching costs, but this advantage excludes the cost of obtaining 50 initialization examples, limiting the assessment of overall efficiency from scratch.

Answer:

We agree that Strategy initialization costs should be included. As shown in Table R3-3, including these costs, EvoAlloc still reduces token cost by 56.4% on average across all four benchmarks, compared with 63.8% under the original accounting. The efficiency advantage therefore remains substantial.

The additional cost is limited because the 50 initialization examples are shared across all five seeds and partially overlap with each run's warm-up data (refer to Appendix A and D.4 for more details about warm-up).

We have clarified this in Appendix D.3 of the revised paper.

Table R3-3. Matched-target token cost (M tokens), with and without initialization costs.

MethodADAS-AIMECircle PackingHotpotQASWE-PolyBench
ShinkaEvolve89.8589.856.086.0891.0791.071,829.071,829.07
EvoAlloc (original accounting)35.3435.340.670.6719.5019.501,336.401,336.40
EvoAlloc (+ initialization)41.4641.460.750.7522.5322.531,666.201,666.20

Reference

[r1] R. T. Lange, et al. ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution. In ICLR, 2026.


Dear Reviewer jQVN ✨,

Thank you for your careful reading and for your positive assessment of our work. We hope the matched-initialization ablation and the complete token-cost accounting resolve the points you raised. We are happy to answer any further questions during the discussion period.

Reviewer_26kMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EvoAlloc, a self-evolving resource allocation agent for LLM-based program evolution. It adapts evaluation strategies using accumulated search experience and counterfactual exploration. Experiments on two benchmarks show improvements in evaluation efficiency and final performance.

Critical strengths

The paper addresses an important problem: reducing evaluation costs in program evolution. The proposed approach is intuitive, and the empirical results demonstrate promising efficiency gains.

Critical weaknesses

The proposed mechanisms are largely heuristic, relying on LLM-generated strategies and manually designed selection rules. Despite a formal problem formulation, the paper lacks a principled mechanism for deciding how to allocate evaluation resources.

The empirical evaluation also lacks comparisons with stronger recent methods, such as AdaEvolve (2026) https://arxiv.org/abs/2602.20133. Without these comparisons, it is difficult to assess the method's competitiveness against modern adaptive evolution approaches.

Questions for authors

See my weakness about baseline selections.

I particularly appreciate the counterfactual exploration mechanism. Although the overall approach remains heuristic, it provides a straightforward yet effective way to prevent search-space collapse. However, this seems more like a fix for a limitation introduced by the proposed method itself rather than an independent contribution. I would encourage the authors to clarify and justify this design choice.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 26kM (1/2)

We thank Reviewer 26kM for the constructive review. We address the comments below.

W1: The proposed mechanisms are largely heuristic, relying on LLM-generated strategies and manually designed selection rules. Despite a formal problem formulation, the paper lacks a principled mechanism for deciding how to allocate evaluation resources.

Answer:

We thank the reviewer for raising this concern. Some predefined choices are natural, such as the allocation action space (Full, Partial, Discard) commonly used in program evolution [r1, r2]. EvoAlloc’s two core mechanisms—self-evolution and counterfactual exploration—are grounded in the following two carefully considered principles:

  1. Allocation strategies should evolve with the program search. Whether and how extensively to evaluate a candidate depends on the changing search context and accumulated evidence. This motivates EvoAlloc's self-evolution mechanism, which distills observed outcomes into reusable Experiences to guide the LLM in revising its allocation Strategy. Each proposed revision is then compared with the current Strategy under shared evaluations and adopted only if it improves performance or saves evaluation resources.

  2. An imperfect allocator should remain capable of detecting and correcting its errors. An LLM-based agent can reject promising candidates without observing their performance, leaving its errors undetected. Without exploration, policy learning can remain trapped in suboptimal policies, as shown in prior work [r3]. This motivates EvoAlloc's counterfactual exploration mechanism, which randomly evaluates some otherwise rejected candidates to reveal their outcomes. These outcomes inform Experience updates and Strategy revisions, helping correct allocation errors.

Moreover, our ablations demonstrate that both mechanisms contribute substantially to performance (Section 5.3 and Appendix D.5 in the revised paper). We have clarified these principles and their implementation in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 26kM (2/2)

W2: The empirical evaluation also lacks comparisons with stronger recent methods, such as AdaEvolve (2026) https://arxiv.org/abs/2602.20133. Without these comparisons, it is difficult to assess the method's competitiveness against modern adaptive evolution approaches.

Answer:

We thank the reviewer for this suggestion and have added experiments with AdaEvolve [r4]. Note that both AdaEvolve and ShinkaEvolve are alternative frameworks for LLM-based program evolution, and EvoAlloc is designed as a plug-in whose core mechanisms can be integrated into either framework. We therefore compare AdaEvolve with and without EvoAlloc on two benchmarks.

As shown in Table R2-1, EvoAlloc improves final performance by 10.4% on ADAS-AIME, while requiring 50.9% fewer Full evaluations and 48.3% fewer LLM tokens. These results demonstrate that the benefits of EvoAlloc’s core mechanisms extend to a strong recent adaptive evolution framework.

We have included these results in Appendix E.1 of the revised paper.

Table R2-1. Comparison of AdaEvolve with and without EvoAlloc. Results are averaged over 5 seeds. The search curves are also available at https://evoalloc-anonymous.netlify.app/re2/. Note that on Circle Packing, most seeds in both variants already reach the best-known solution (≈2.636\approx 2.636), leaving limited room for further performance improvement.

BenchmarkMethodFinal PerformanceFull EvalsLLM Tokens (M)
Circle PackingAdaEvolve2.60±0.062.60_{\pm 0.06}1492.8
AdaEvolve + EvoAlloc2.61±0.03\mathbf{2.61_{\pm 0.03}}1092.6
ADAS-AIMEAdaEvolve35.44±5.9135.44_{\pm 5.91}5571.8
AdaEvolve + EvoAlloc39.11±7.38\mathbf{39.11_{\pm 7.38}}2737.2

Q1: I particularly appreciate the counterfactual exploration mechanism. Although the overall approach remains heuristic, it provides a straightforward yet effective way to prevent search-space collapse. However, this seems more like a fix for a limitation introduced by the proposed method itself rather than an independent contribution. I would encourage the authors to clarify and justify this design choice.

Answer:

We thank the reviewer for recognizing the value of counterfactual exploration. We clarify the motivation and scope of this design choice below.

  • The challenge addressed by counterfactual exploration is general, rather than specific to EvoAlloc. As explained in our response to W1, this mechanism aims to address selective-feedback bias: an allocator observes only the outcomes of candidates it evaluates, leaving mistaken rejections of promising candidates undetected. This problem can arise with any imperfect allocator, and selective feedback can reinforce suboptimal allocation policies without exploration [r3].

  • In addition, we'd like to highlight that counterfactual exploration is a general mechanism that can be integrated into other LLM-based resource allocators. Its core idea is to occasionally evaluate rejected alternatives and use the revealed outcomes to revise future allocation decisions. It is therefore applicable to other LLM-based allocators that can obtain such feedback and use it to update their allocation policies. As reported in our response to W2, EvoAlloc's mechanisms can be integrated into different program evolution frameworks, including AdaEvolve and ShinkaEvolve.

We have further clarified this general challenge in Section 3.3 and the motivation for counterfactual exploration in Section 4.3 of the revised paper.


Reference

[r1] A. Novikov et al. AlphaEvolve: A Coding Agent for Scientific and Algorithmic Discovery. arXiv preprint arXiv:2506.13131, 2025.

[r2] J. Zhang et al. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. In ICLR, 2026.

[r3] N. Kilbertus et al. Fair Decisions Despite Imperfect Predictions. In AISTATS, 2020.

[r4] M. Cemri et al. AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization. arXiv preprint arXiv:2602.20133, 2026.


Dear Reviewer 26kM ✨,

Thank you for your constructive review. We hope the clarified design principles and the new AdaEvolve comparisons address your concerns; if so, we would be grateful if you would consider reflecting this in your assessment. We are happy to answer any further questions.

Reviewer_HhaKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Since the value of evaluating a candidate changes as the search progresses and the budget shrinks, the paper argues that allocation should co-evolve with the search. It introduces EvoAlloc, an agent with an explicit strategy for deciding whether and how extensively to evaluate candidates. EvoAlloc periodically distills search history and allocation outcomes into reusable reflective experience, proposes strategy revisions, and adopts them when shared validation favors fewer missed new-best candidates and then lower evaluation cost. Because its own allocation choices determine which outcomes become visible, EvoAlloc also uses counterfactual exploration to sometimes evaluate candidates its strategy would otherwise deny, improving the experience available for future revisions. Across coding and agent-harness optimization benchmarks, EvoAlloc achieves higher final performance than baseline under the same full-evaluation budget.

Critical strengths

Unlike fixed allocation rules, EvoAlloc evolves an explicit strategy from accumulated experience, with stage-wise shifts in selectivity as search progresses.

Gains hold even on Circle Packing, where no partial evaluation is available, showing the benefit isn't just from an extra evaluation stage.

Experiments show higher final performance and cost savings. Under the same budget, EvoAlloc improves final performance over the baseline. It reaches baseline-level performance with fewer full evaluations and fewer LLM tokens.

Critical weaknesses

The study only examines allocation at the evaluation layer on two benchmarks, within a single search run, with generation, evaluators, and update mechanisms held fixed.

Because the agent's own choices decide which outcomes are visible, it depends on counterfactual exploration to avoid uncorrected pruning errors, which adds complexity and overhead.

Extending allocation to generation, letting evaluation/update mechanisms evolve, and retaining experience across runs are noted as future work rather than demonstrated benefits.

Validated only on a limited set of benchmarks and one search layer, broader generality remains to be shown.

Questions for authors

see above

3 comments
AuthorsOpenReview ↗

Response to Reviewer HhaK (1/3)

We thank Reviewer HhaK for the thoughtful review. We address the comments below.

W1: The study only examines allocation at the evaluation layer on two benchmarks, within a single search run, with generation, evaluators, and update mechanisms held fixed.

W3: Extending allocation to generation, letting evaluation/update mechanisms evolve, and retaining experience across runs are noted as future work rather than demonstrated benefits.

W4: Validated only on a limited set of benchmarks and one search layer, broader generality remains to be shown.

Answer:

We thank the reviewer and address W1, W3, and W4 together with additional experiments on new benchmarks and future directions outlined in our paper. These results further demonstrate EvoAlloc's generality across diverse benchmarks, evaluation protocols, and its applicability to generation allocation, cross-run experience transfer, and different evolutionary search frameworks.

  1. Two additional benchmarks. We have added HotpotQA and SWE-PolyBench, bringing our evaluation to four benchmarks spanning diverse domains and optimized components (Table R1-1-(1)). As shown in Table R1-1-(2) below and Fig. 3 in the revised paper, EvoAlloc achieves 9.2% higher final performance on average across all four benchmarks, while requiring 60.6% fewer Full evaluations and 63.8% fewer LLM tokens to reach baseline-level performance.

  2. Additional experiments on future directions. Our paper focuses on improving evaluation-resource allocation within a single search run, and the extensions raised by the reviewer were explicitly identified as future work beyond the paper's scope. Nevertheless, we conducted additional experiments to explore these directions and address the reviewer's concerns:

  • (a) Generation allocation. We use EvoAlloc's core principles to determine how many offspring each parent generates, with evaluation allocation disabled. The results on Circle Packing show 8.1% higher final performance, while requiring 57.4% fewer Full evaluations and 70.5% fewer LLM tokens, providing preliminary evidence that these principles extend beyond evaluation allocation (Table R1-2-(a)).

  • (b) Cross-run experience transfer. We transfer experience from completed runs to new runs and compare against EvoAlloc without prior experience. The results on Circle Packing show 35.6% fewer Full evaluations and 30.7% fewer LLM tokens, demonstrating the benefits of retaining experience across runs (Table R1-2-(b)).

  • (c) Compatibility with evolving mechanisms. In our original experiments, other adaptive mechanisms already evolve alongside EvoAlloc; “held fixed” means that we did not modify these mechanisms. To further assess generality, we evaluate EvoAlloc alongside different evolving mechanisms based on AdaEvolve [r1] on both benchmarks (Tables R1-2-(c1) and R1-2-(c2)). The results on ADAS-AIME show 10.4–12.0% higher final performance, with 50.9–59.4% fewer Full evaluations and 48.3–60.7% fewer LLM tokens across ShinkaEvolve and AdaEvolve, demonstrating that EvoAlloc remains effective when coupled with different evolving mechanisms.

Full experimental curves are available at https://evoalloc-anonymous.netlify.app/re1/, and have been included in the revised paper (Fig. 3 and Appendix E).

AuthorsOpenReview ↗

Response to Reviewer HhaK (2/3)

Table R1-1-(1). Overview of the four benchmarks.

BenchmarkDomainOptimized ComponentEvaluation InstancesAllows Partial Evaluation
Circle PackingNumerical OptimizationSolution Program1×\times
ADAS-AIMEMathematical ReasoningLLM Workflow30✓\checkmark
HotpotQA (New)Knowledge-intensive QARetrieval Agent Workflow150✓\checkmark
SWE-PolyBench (New)Software EngineeringCoding Agent Workflow64✓\checkmark

 


Table R1-1-(2). Performance and efficiency on the two new benchmarks. Final performance is reported as mean ±\pm std. over 5 seeds. Full Evaluations denotes the number of complete candidate evaluations; LLM Tokens (M) denotes cumulative LLM input and output tokens in millions. Both costs are measured when each method first reaches the target threshold; "---" denotes an unreached target. Higher (↑\uparrow) / lower (↓\downarrow) is better. These metric definitions also apply to Tables below.

BenchmarkMethodFinal Performance (%) ↑\uparrowFull Evaluations ↓\downarrowLLM Tokens (M) ↓\downarrow
SWE-PolyBenchShinkaEvolve16.25±1.6616.25_{\pm 1.66}60601,829.11,829.1
One-Step Alloc16.56±1.7216.56_{\pm 1.72}45452,019.02,019.0
Lineage-UCB15.42±1.7915.42_{\pm 1.79}------
RPM16.46±0.9716.46_{\pm 0.97}65652,621.72,621.7
EvoAlloc (ours)18.12±1.33\mathbf{18.12_{\pm 1.33}}44\mathbf{44}1,336.4\mathbf{1,336.4}
HotpotQAShinkaEvolve63.82±1.4363.82_{\pm 1.43}737391.191.1
One-Step Alloc64.27±1.2464.27_{\pm 1.24}666672.772.7
Lineage-UCB65.42±2.4665.42_{\pm 2.46}474739.939.9
RPM64.76±2.2764.76_{\pm 2.27}222235.035.0
EvoAlloc (ours)67.02±1.72\mathbf{67.02_{\pm 1.72}}19\mathbf{19}19.5\mathbf{19.5}

 


Table R1-2-(a). Generation allocation on Circle Packing. The evaluation allocator is disabled. Results are averaged over 5 seeds.

MethodFinal Performance ↑\uparrowFull Evaluations ↓\downarrowLLM Tokens (M) ↓\downarrow
ShinkaEvolve (uniform generation)2.34±0.252.34_{\pm 0.25}1481486.16.1
+ Generation allocation2.53±0.04\mathbf{2.53_{\pm 0.04}}63\mathbf{63}1.8\mathbf{1.8}

 


Table R1-2-(b). Cross-run experience transfer on Circle Packing. Results are averaged over 5 seeds. All methods use seeds 6–10, with prior experience transferred from seeds 1–5.

MethodFinal Performance ↑\uparrowFull Evaluations ↓\downarrowLLM Tokens (M) ↓\downarrow
ShinkaEvolve2.45±0.162.45_{\pm 0.16}1471475.45.4
EvoAlloc (w/o prior experience)2.54±0.032.54_{\pm 0.03}59592.152.15
EvoAlloc (w/ prior experience)2.58±0.04\mathbf{2.58_{\pm 0.04}}38\mathbf{38}1.49\mathbf{1.49}

 


Table R1-2-(c1). EvoAlloc under different evolving mechanisms on Circle Packing. Results are averaged over 5 seeds. Most seeds in both AdaEvolve variants already reach the best-known solution (≈2.636\approx 2.636), leaving limited room for further performance improvement.

MethodFinal Performance ↑\uparrowFull Evaluations ↓\downarrowLLM Tokens (M) ↓\downarrow
ShinkaEvolve2.34±0.252.34_{\pm 0.25}1481486.16.1
ShinkaEvolve + EvoAlloc2.55±0.03\mathbf{2.55_{\pm 0.03}}26\mathbf{26}0.7\mathbf{0.7}
AdaEvolve2.60±0.062.60_{\pm 0.06}1491492.82.8
AdaEvolve + EvoAlloc2.61±0.03\mathbf{2.61_{\pm 0.03}}109\mathbf{109}2.6\mathbf{2.6}

 


Table R1-2-(c2). EvoAlloc under different evolving mechanisms on ADAS-AIME. Results are averaged over 5 seeds.

MethodFinal Performance (%) ↑\uparrowFull Evaluations ↓\downarrowLLM Tokens (M) ↓\downarrow
ShinkaEvolve35.33±2.5735.33_{\pm 2.57}696989.889.8
ShinkaEvolve + EvoAlloc39.56±6.80\mathbf{39.56_{\pm 6.80}}28\mathbf{28}35.3\mathbf{35.3}
AdaEvolve35.44±5.9135.44_{\pm 5.91}555571.871.8
AdaEvolve + EvoAlloc39.11±7.38\mathbf{39.11_{\pm 7.38}}27\mathbf{27}37.2\mathbf{37.2}
AuthorsOpenReview ↗

Response to Reviewer HhaK (3/3)

W2: Because the agent's own choices decide which outcomes are visible, it depends on counterfactual exploration to avoid uncorrected pruning errors, which adds complexity and overhead.

Answer:

We thank the reviewer for raising this concern. We address the necessity, complexity, and overhead of counterfactual exploration below.

  1. Counterfactual exploration is necessary to address the inherent challenge of detecting and correcting pruning errors. As the reviewer correctly notes, an allocator's decisions determine which outcomes it observes. Erroneous rejections can therefore remain undetected without exploration. Prior work shows that, without exploration, agents can remain trapped in suboptimal policies when feedback is selectively observed [r2]. Our experiments further demonstrate its practical importance:

    • On Circle Packing, exploration identifies 22 candidates that outperform the best solution available at the time, all of which would otherwise have been discarded (Table 10 in the revised paper).
    • Removing exploration reduces final performance by 11.8% on ADAS-AIME and 4.6% on Circle Packing (Figure 4 (a-b)).
  2. Counterfactual exploration adds minimal complexity. It requires no additional LLM call, only a Bernoulli draw per discarded candidate controlled by a single hyperparameter ρ\rho. The parameter ρ\rho does not require careful tuning: performance remains stable over ρ∈[0.1,0.7]\rho \in [0.1, 0.7] (Figure 4c), and we fix ρ=0.3\rho = 0.3 across all benchmarks.

  3. Its benefits outweigh its evaluation overhead. Exploration incurs evaluation costs, but the errors it reveals guide Strategy revisions and help improve subsequent allocation decisions. With all exploration costs included, EvoAlloc still saves 60.6% of Full evaluations and 63.8% of LLM tokens to reach the target performance, averaged across all four benchmarks (Figure 3 and Table 6 in the revised paper).


Reference

[r1] M. Cemri, et al. AdaEvolve: Adaptive LLM Driven Zeroth-Order Optimization. arXiv preprint arXiv:2602.20133, 2026.

[r2] N. Kilbertus et al. Fair Decisions Despite Imperfect Predictions. In AISTATS, 2020.


Dear Reviewer HhaK ✨,

Thank you for your thoughtful review. We hope the new experiments address your concerns about generality and counterfactual exploration; if so, we would be grateful if you would consider reflecting this in your assessment. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.