- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EvoAlloc, which adjusts candidate-evaluation strategies using search history to choose full evaluation, partial evaluation, or skipping during program evolution. It compares incumbent and challenger strategies through shared evaluations and randomly evaluates some skipped candidates to inform subsequent allocation. Five-seed mean results on ADAS-AIME and Circle Packing show improved final performance at the same full-evaluation budget, and fewer online evaluations and tokens for the mean search trajectory to reach the baseline score.
Critical strengths
-
Random exploration addresses missing feedback from candidate filtering: skipped candidates would otherwise provide no outcomes, while exploration evaluates some of them and informs later strategies. It recovers 6 and 22 candidates that improve the current best score on the two tasks, demonstrating concrete search value.
-
The complete method provides practical gains: at the same full-evaluation budget, mean final performance is 39.56% versus 35.33% and 2.547 versus 2.343 for ShinkaEvolve. Retaining explored candidates in search while removing only their explicit feedback lowers performance on both tasks, providing more targeted evidence for exploration feedback than simply removing exploration.
Critical weaknesses
-
Online evolution is a central contribution, but the comparisons mix the effects of updating and initial guidance. One-Step removes both the strategy and experience; w/o Evolution uses manually refined fixed guidance but starts with a medium-selectivity strategy and seeded experience, whereas the full method starts with low selectivity and empty experience (Appendices A, B.1, and D.1). These results support the complete allocator over the selected alternatives, but do not clearly establish how much of the gain comes from online evolution itself.
-
The resource advantage requires distinguishing target-reaching cost from the total investment in a complete search. Appendix D.3 discloses that EvoAlloc uses more tokens when the same full-evaluation budget is exhausted, for example 10.85M versus 6.11M for ShinkaEvolve on Circle Packing; the final-performance gain therefore cannot directly be interpreted as a gain at equal total compute. Table 5 and the token curves still support lower online target-reaching costs, but this advantage excludes the cost of obtaining 50 initialization examples, limiting the assessment of overall efficiency from scratch.
Questions for authors
None.
2 comments
Response to Reviewer jQVN (2/2)
W2.1: The resource advantage requires distinguishing target-reaching cost from the total investment in a complete search. Appendix D.3 discloses that EvoAlloc uses more tokens when the same full-evaluation budget is exhausted, for example 10.85M versus 6.11M for ShinkaEvolve on Circle Packing; the final-performance gain therefore cannot directly be interpreted as a gain at equal total compute.
Answer:
We thank the reviewer for this careful reading.
-
Our original comparison follows standard practice in program evolution [r1], evaluating performance under matched evaluation counts. EvoAlloc achieves better performance under this protocol.
-
We believe that token efficiency is more fairly assessed by comparing performance at the same token cost, rather than by comparing token cost under matched evaluation counts as in the reviewer's comment. Figure 3 in the original submission already reports performance against token cost; Table R3-2 summarizes the corresponding comparisons. EvoAlloc outperforms ShinkaEvolve by 8.1% on average at the same token cost, close to the 9.2% improvement under matched evaluation counts.
Table R3-2. Performance at the same token cost (mean std over five seeds). For each benchmark, we use ShinkaEvolve's average token cost as a common budget and report each method's best performance within it.
| Method | ADAS-AIME (%) | Circle Packing | HotpotQA (%) | SWE-PolyBench (%) |
|---|---|---|---|---|
| ShinkaEvolve | ||||
| One-Step Alloc | ||||
| Lineage-UCB | ||||
| RPM | ||||
| EvoAlloc (ours) |
W2.2: Table 5 and the token curves still support lower online target-reaching costs, but this advantage excludes the cost of obtaining 50 initialization examples, limiting the assessment of overall efficiency from scratch.
Answer:
We agree that Strategy initialization costs should be included. As shown in Table R3-3, including these costs, EvoAlloc still reduces token cost by 56.4% on average across all four benchmarks, compared with 63.8% under the original accounting. The efficiency advantage therefore remains substantial.
The additional cost is limited because the 50 initialization examples are shared across all five seeds and partially overlap with each run's warm-up data (refer to Appendix A and D.4 for more details about warm-up).
We have clarified this in Appendix D.3 of the revised paper.
Table R3-3. Matched-target token cost (M tokens), with and without initialization costs.
| Method | ADAS-AIME | Circle Packing | HotpotQA | SWE-PolyBench |
|---|---|---|---|---|
| ShinkaEvolve | ||||
| EvoAlloc (original accounting) | ||||
| EvoAlloc (+ initialization) |
Reference
[r1] R. T. Lange, et al. ShinkaEvolve: Towards Open-Ended and Sample-Efficient Program Evolution. In ICLR, 2026.
Dear Reviewer jQVN ✨,
Thank you for your careful reading and for your positive assessment of our work. We hope the matched-initialization ablation and the complete token-cost accounting resolve the points you raised. We are happy to answer any further questions during the discussion period.
Response to Reviewer jQVN (1/2)
We thank Reviewer jQVN for the careful reading of our paper. We address the comments below.
Answer:
We thank the reviewer for this helpful suggestion. We have added w/o Evolution (matched initialization), which disables online evolution while retaining the same initialization as EvoAlloc. With initialization held fixed, enabling online evolution improves mean performance by 17.1% on ADAS-AIME, as shown in the table below.
The w/o Evolution variant in the original submission used manually refined initialization with the aim of providing a competitive fixed allocator. We agree with the reviewer that matching initialization better isolates the contribution of online evolution, and have added this comparison to Appendix D.5 of the revised paper.
Table R3-1. Ablation of online evolution. Results are mean ± std over five seeds.