- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper formalizes LLM-driven program evolution, as in AlphaEvolve, OpenEvolve and ShinkaEvolve, as an MDP. The action is the modular text prefix that conditions a frozen black-box LLM, not the program the LLM emits. Three modules produce this prefix: credit assignment, value estimation and adaptive exploration. A fourth module, experience memory, supplies retrieved context. The paper argues that prior methods correspond to a zero discount. AGAR implements these modules on a substrate with interchangeable modules and asynchronous execution, and the paper compares it with OpenEvolve and ShinkaEvolve on algorithm-discovery tasks.
Critical strengths
-
Each transferred RL tool is paired with the hand-set rule it replaces, and the estimators can be replaced or switched off without touching the controller. For example, replacing the LLM explore/exploit decision with a discounted-UCB bandit took one new file and no change to the controller.
-
The measurement of the TextGrad rewrite operator is useful. Inside the loop, the guideline block hit its character budget at the first update, and adjacent versions had low textual similarity. The guidelines were therefore resampled rather than improved. The replacement operator, a local edit plus deferred validation, follows from this measurement.
Critical weaknesses
-
The experiments do not show how much the transferred modules contribute. Sec. 3.4 explains why a per-module decomposition is not reported. The vanilla baseline, which Sec. 6.1 calls the scientifically important one, is not reported either, and the effect of the discount is left to future work. The related-work discussion could also cover MCTS-AHD (Zheng et al., ICML 2025), which backs up child values to their parents in a tree of LLM-generated heuristics.
-
Part of AGAR's improvement in Table 1 may be due to its larger LLM budget rather than its mechanisms. App. D.8 notes that at matched evaluations AGAR spends two to four times the baselines' LLM calls and that no LLM-call-matched arm is reported. On the three systems/ML tasks that are not near ceiling, AGAR and the stronger baseline are within a few percent of each other (App. D.10).
Questions for authors
-
How do vanilla and AGAR with γ = 0 compare with the default configuration at the same number of LLM calls? Clear gains would resolve the part of my first concern about the transfer.
-
How do OpenEvolve and ShinkaEvolve score with the same number of LLM calls as AGAR?
-
For the AtCoder tasks, are the reported scores computed on the same test cases that drive selection?
4 comments
Weakness 2 and Q1-Q3: the cost-matched comparison, and held-out re-scoring that changes no winner (0 of 20)
Q1: how do vanilla and the reported configuration compare at matched cost? Both arms were run, and our algorithms are what deliver the margin. With our token budget matched to each baseline's own, over all 13 non-saturated tasks:
| AGAR configuration | wins vs SE | median | wins vs OE | median |
|---|---|---|---|---|
| our configuration (modules on) | 10 / 13 | +4.7% | 11 / 13 | +6.8% |
| modules off (vanilla) | 6 / 13 | −0.1% | 10 / 13 | +1.1% |
The two rows separate the two halves of the contribution: with every module off, the bare substrate already matches ShinkaEvolve and is ahead of OpenEvolve, and switching the modules on is what produces the margin. Against ShinkaEvolve the separation shows in the win count; against OpenEvolve both arms take most tasks, but the median margin is six times larger with the modules on.
Q2: how do the baselines score at AGAR's budget? We take the cost-matched comparison to be the one at a common budget, and give it for both baselines. Per run on GLM, AGAR spends 2.44× ShinkaEvolve's tokens and an estimated 3.55× OpenEvolve's. Truncating our budget to each baseline's own:
| At the baseline's own token budget | AGAR wins | median |
|---|---|---|
| vs ShinkaEvolve, AtCoder family | 9 / 9 | +9.6% |
| vs OpenEvolve, AtCoder family | 8 / 9 | +13.2% |
| vs OpenEvolve, all 13 non-saturated tasks | 11 / 13 | +6.8% |
| vs ShinkaEvolve, all 13 non-saturated tasks | 10 / 13 | +4.7% |
We also compared the whole trajectory, not only the endpoint. Over a grid of 20 budget fractions, our best-so-far curve has a larger area on 8 of 9 AtCoder tasks (median +9.3%), and the median run reaches ShinkaEvolve's final score at 0.20 of its budget.
Q3: are the AtCoder scores computed on the same test cases that drive selection? They were — and on fresh cases the winner changes in 0 of 20 task–backend pairs, so our lead across the AtCoder family holds under held-out scoring. We re-scored the best program of all runs (10 tasks, both backends, three seeds, all systems) on 100 cases from new seeds.
| GLM, normalised mean | OpenEvolve | ShinkaEvolve | AGAR Combo | AGAR Base |
|---|---|---|---|---|
| on the public cases | 0.840 | 0.876 | 0.932 | 0.951 |
| on the held-out cases | 0.848 | 0.887 | 0.930 | 0.935 |
Both AGAR arms sit above both baselines before and after re-scoring, and the ordering is the same on DeepSeek. In absolute terms every system loses between −8.4% and −10.2% on the fresh cases, so selection on the public cases lifts all of them alike.
We hope this addresses the points raised. If anything remains unclear, or if the reviewer would like us to expand on any of it, we would be glad to continue the discussion.
Thank you for your rebuttal. Considering the paper quality, I will maintain my score.
Reply to Reviewer e48v's final comment
Dear Reviewer e48v,
Thank you for returning to the thread and for the time you have given this paper.
We note that the comment does not point to a concern that remains open, and we would not want to press the point further. If any part of the new evidence — the per-module ablation, the comparison at each baseline's own token budget, or the held-out re-scoring — is unconvincing on its merits, we would genuinely welcome knowing which, as that would be useful to us well beyond this review cycle.
Either way, we are grateful for the three questions, which led to experiments the paper is better for having.
Sincerely, The Authors
Weakness 1: every module earns its place (23 of 28 cells), and MCTS-AHD added
We thank the reviewer for the careful reading, and particularly for the three questions: each pointed at something we could test directly, and all three are now answered with new experiments. Thanks too for the MCTS-AHD pointer, which we have added. We take the weaknesses and questions in the order of the review.
On the contribution of the individual modules. All four cost score when removed, so the ablation supports the design of each of the four components and not only of the stack as a whole. In our reported configuration, six tasks × three seeds, token-matched:
On MCTS-AHD. Added. It backs child values up to parents and is the closest prior system to the recursion we formalize; it also bounds Observation 1, which we now state over population-selection systems rather than over all systems in this literature.