acceptodds
Under review as a conference paper at ICLR 2027

AGAR: A REINFORCEMENT LEARNING SUBSTRATE FOR LLM PROGRAM EVOLUTION

Abstract

Given a task and an evaluator, a language model can rewrite a candidate program while a search loop decides which rewrites survive — a practical route to algorithm discovery. But that loop is governed by five constants set by hand — which parent to select, how hard to mutate, how to keep diversity, what to remember,and a scalar score that never says which part of the program earned it — and reinforcement learning already has an estimator for each. The obstacle is that program evolution is not usually written down as a decision process. We formalize it as a Markov decision process whose action is the modular prefix the model is conditioned on, not the program it emits, so that credit assignment, value estimation,adaptive exploration and experience memory each attach to a distinct component.AGAR (Algorithm Generation As RL) provides the substrate this licenses: any estimator can be replaced or switched off without touching the controller, making the transfer auditable one mechanism at a time, with no gradient training of the backend model. Across 19 tasks, two backends and three seeds under one harness,AGAR improves on the stronger of two published baselines on most tasks, with the gains concentrated on the competitive-programming family. The formalization also yields a checkable reading of prior work: these systems are implicitly zero-discount, not by choice but because fitness is exogenous to an individual rather than a return over successors, leaving a discount factor nothing to act on.

Primary area:
foundation or frontier models, including LLMs
Keywords:
code optimization, evolutionary algorithms, reinforcement learning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_e48vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formalizes LLM-driven program evolution, as in AlphaEvolve, OpenEvolve and ShinkaEvolve, as an MDP. The action is the modular text prefix that conditions a frozen black-box LLM, not the program the LLM emits. Three modules produce this prefix: credit assignment, value estimation and adaptive exploration. A fourth module, experience memory, supplies retrieved context. The paper argues that prior methods correspond to a zero discount. AGAR implements these modules on a substrate with interchangeable modules and asynchronous execution, and the paper compares it with OpenEvolve and ShinkaEvolve on algorithm-discovery tasks.

Critical strengths
  1. Each transferred RL tool is paired with the hand-set rule it replaces, and the estimators can be replaced or switched off without touching the controller. For example, replacing the LLM explore/exploit decision with a discounted-UCB bandit took one new file and no change to the controller.

  2. The measurement of the TextGrad rewrite operator is useful. Inside the loop, the guideline block hit its character budget at the first update, and adjacent versions had low textual similarity. The guidelines were therefore resampled rather than improved. The replacement operator, a local edit plus deferred validation, follows from this measurement.

Critical weaknesses
  1. The experiments do not show how much the transferred modules contribute. Sec. 3.4 explains why a per-module decomposition is not reported. The vanilla baseline, which Sec. 6.1 calls the scientifically important one, is not reported either, and the effect of the discount is left to future work. The related-work discussion could also cover MCTS-AHD (Zheng et al., ICML 2025), which backs up child values to their parents in a tree of LLM-generated heuristics.

  2. Part of AGAR's improvement in Table 1 may be due to its larger LLM budget rather than its mechanisms. App. D.8 notes that at matched evaluations AGAR spends two to four times the baselines' LLM calls and that no LLM-call-matched arm is reported. On the three systems/ML tasks that are not near ceiling, AGAR and the stronger baseline are within a few percent of each other (App. D.10).

Questions for authors
  1. How do vanilla and AGAR with γ = 0 compare with the default configuration at the same number of LLM calls? Clear gains would resolve the part of my first concern about the transfer.

  2. How do OpenEvolve and ShinkaEvolve score with the same number of LLM calls as AGAR?

  3. For the AtCoder tasks, are the reported scores computed on the same test cases that drive selection?

4 comments
AuthorsOpenReview ↗

Weakness 1: every module earns its place (23 of 28 cells), and MCTS-AHD added

We thank the reviewer for the careful reading, and particularly for the three questions: each pointed at something we could test directly, and all three are now answered with new experiments. Thanks too for the MCTS-AHD pointer, which we have added. We take the weaknesses and questions in the order of the review.

On the contribution of the individual modules. All four cost score when removed, so the ablation supports the design of each of the four components and not only of the stack as a whole. In our reported configuration, six tasks × three seeds, token-matched:

Component removedfull config better onmedian
M1 credit assignment5 / 6+11.8%
M2 value estimation4 / 6+4.9%
M3 adaptive exploration5 / 6+5.7%
M4 experience memory5 / 6+4.8%
pooled, 28 cells23 / 28+5.5%

On MCTS-AHD. Added. It backs child values up to parents and is the closest prior system to the recursion we formalize; it also bounds Observation 1, which we now state over population-selection systems rather than over all systems in this literature.

AuthorsOpenReview ↗

Weakness 2 and Q1-Q3: the cost-matched comparison, and held-out re-scoring that changes no winner (0 of 20)

Q1: how do vanilla and the reported configuration compare at matched cost? Both arms were run, and our algorithms are what deliver the margin. With our token budget matched to each baseline's own, over all 13 non-saturated tasks:

AGAR configurationwins vs SEmedianwins vs OEmedian
our configuration (modules on)10 / 13+4.7%11 / 13+6.8%
modules off (vanilla)6 / 13−0.1%10 / 13+1.1%

The two rows separate the two halves of the contribution: with every module off, the bare substrate already matches ShinkaEvolve and is ahead of OpenEvolve, and switching the modules on is what produces the margin. Against ShinkaEvolve the separation shows in the win count; against OpenEvolve both arms take most tasks, but the median margin is six times larger with the modules on.

Q2: how do the baselines score at AGAR's budget? We take the cost-matched comparison to be the one at a common budget, and give it for both baselines. Per run on GLM, AGAR spends 2.44× ShinkaEvolve's tokens and an estimated 3.55× OpenEvolve's. Truncating our budget to each baseline's own:

At the baseline's own token budgetAGAR winsmedian
vs ShinkaEvolve, AtCoder family9 / 9+9.6%
vs OpenEvolve, AtCoder family8 / 9+13.2%
vs OpenEvolve, all 13 non-saturated tasks11 / 13+6.8%
vs ShinkaEvolve, all 13 non-saturated tasks10 / 13+4.7%

We also compared the whole trajectory, not only the endpoint. Over a grid of 20 budget fractions, our best-so-far curve has a larger area on 8 of 9 AtCoder tasks (median +9.3%), and the median run reaches ShinkaEvolve's final score at 0.20 of its budget.

Q3: are the AtCoder scores computed on the same test cases that drive selection? They were — and on fresh cases the winner changes in 0 of 20 task–backend pairs, so our lead across the AtCoder family holds under held-out scoring. We re-scored the best program of all runs (10 tasks, both backends, three seeds, all systems) on 100 cases from new seeds.

GLM, normalised meanOpenEvolveShinkaEvolveAGAR ComboAGAR Base
on the public cases0.8400.8760.9320.951
on the held-out cases0.8480.8870.9300.935

Both AGAR arms sit above both baselines before and after re-scoring, and the ordering is the same on DeepSeek. In absolute terms every system loses between −8.4% and −10.2% on the fresh cases, so selection on the public cases lifts all of them alike.

We hope this addresses the points raised. If anything remains unclear, or if the reviewer would like us to expand on any of it, we would be glad to continue the discussion.

Thank you for your rebuttal. Considering the paper quality, I will maintain my score.

AuthorsOpenReview ↗

Reply to Reviewer e48v's final comment

Dear Reviewer e48v,

Thank you for returning to the thread and for the time you have given this paper.

We note that the comment does not point to a concern that remains open, and we would not want to press the point further. If any part of the new evidence — the per-module ablation, the comparison at each baseline's own token budget, or the held-out re-scoring — is unconvincing on its merits, we would genuinely welcome knowing which, as that would be useful to us well beyond this review cycle.

Either way, we are grateful for the three questions, which led to experiments the paper is better for having.

Sincerely, The Authors

Reviewer_teNFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AGAR formulates LLM-driven program evolution as a decision process in which the action is the text prefix conditioning a frozen code-generating model. It organizes credit attribution, value estimation, exploration, and memory into replaceable modules, and reports improved best-found solutions across 19 tasks and two model backends.

Critical strengths
  1. The action-as-prefix formulation gives the system a clear organizing principle. The module interfaces and asynchronous execution details are described with unusual care for a systems paper.

  2. The evaluation spans several task families. I also appreciate that the authors report non-monotonic ablations and acknowledge where the current evidence does not support per-module attribution.

Critical weaknesses
  1. The paper’s RL framing is more developed than its demonstrated RL mechanism. The value estimate is not learned from realized returns, and the effect of a nonzero discount factor is not isolated experimentally. As a result, the experiments establish a useful RL-inspired architecture more clearly than they establish the benefit of multi-step value estimation.

  2. It remains difficult to identify what drives the gains. There is no leave-one-out ablation of the four modules, while the substrate ladder covers only five tasks, is non-monotonic, and derives most of its mean improvement from two tasks.

  3. Results are compared at equal numbers of fitness evaluations, but AGAR uses 2–4 times as many LLM calls as the external baselines. A call-, token-, or cost-matched comparison is needed to assess the practical advantage.

Questions for authors
  1. Could the authors clarify the relationship between Base/Combo and the substrate ladder? Section 3.3 says Base and Combo differ in the value estimator and exploration rule, whereas Section 6.3 identifies them with (L_2) and (L_3), where (L_3) adds sample-level credit assignment.

  2. How is a local guideline edit validated or reverted when its descendants arrive after a variable delay? Some details on the decision criterion would help assess whether the text-space updates improve the policy rather than respond to noisy short-run outcomes.

  3. Does the measured staleness correlate with value-estimation error or final solution quality? The delay measurements are interesting, but their consequence for the proposed decision rule remains unclear.

2 comments
AuthorsOpenReview ↗

What drives the gains: the module ablation, and the comparison at matched inference cost

We thank the reviewer for the close reading of the formulation and the module interfaces, and for the detailed engagement with the ablations. We take the weaknesses and questions in the order of the review.

On what drives the gains, and on the substrate ladder of Table 2. We have added a module ablation that reports the contribution of each module separately. In our reported configuration, six tasks × three seeds, token-matched:

Component removedfull config better onmedian
M1 credit assignment5 / 6+11.8%
M2 value estimation4 / 6+4.9%
M3 adaptive exploration5 / 6+5.7%
M4 experience memory5 / 6+4.8%
pooled, 28 cells23 / 28+5.5%

All four components cost score when they are removed, so the ablation supports the design of each of the four and not only of the stack as a whole.

On comparing at equal evaluations rather than equal cost. The reviewer is right, the comparison is now at matched tokens, and once cost is matched our algorithms remain ahead of both baselines. AGAR spends 2.44× ShinkaEvolve's tokens and an estimated 3.55× OpenEvolve's. With our token budget matched to each baseline's own, over all 13 non-saturated tasks:

AGAR configurationwins vs SEmedianwins vs OEmedian
our configuration (modules on)10 / 13+4.7%11 / 13+6.8%
modules off (vanilla)6 / 13−0.1%10 / 13+1.1%

The two rows separate the two halves of the contribution: with every module off, the bare substrate already matches ShinkaEvolve and is ahead of OpenEvolve, and switching the modules on is what produces the margin. On the AtCoder family alone we take 9 of 9 against ShinkaEvolve and 8 of 9 against OpenEvolve. The revision replaces the evaluation-matched main table with the token-matched one.

On where the gain shows. It shows in the trajectory as well as the endpoint: over 20 budget fractions our best-so-far curve has the larger area on 8 of 9 tasks (median +9.3%), and the median run reaches ShinkaEvolve's final score at 0.20 of its budget.

AuthorsOpenReview ↗

Q1-Q3: the configuration as the code runs it, how a guideline edit is validated, and staleness

Q1: how do Base/Combo relate to the substrate ladder? The reviewer is right to read §3.3 and §6.3 against each other, and here is the specification. The ladder levels instantiate none of the transferred modules: L3 is our substrate with every module off, which is the vanilla arm. They are a floor for the loop rather than instances of Base and Combo. Several DeepSeek cells of Table 1 were filled from ladder runs, so we rebuild that column below so it is uniform. On §3.3:

BaseCombo
M2 estimatorstatisticalstatistical — identical in both
M3 decision ruleLLM deciderdiscounted-UCB
M1 input formatfulldiff
novelty incubationoffon

Rebuilt from the modules-on runs, the DeepSeek column reads Base 0.968 / Combo 0.966 against ShinkaEvolve 0.943 and OpenEvolve 0.875 — the ranking is unchanged. The GLM column was always modules-on. Contribution 2 is therefore exercised by the Base arm. §3.3 and §6.3 are restated to this specification in the revision.

Q2: when descendants arrive late, how does a local guideline edit get validated or reverted, and what keeps that decision from chasing short-term swings? Every edit is judged on a held window against the incumbent measured at the same age, and scored so that no single rollout can swing the result. Across six tasks, about three quarters of proposed edits are reverted.

Those swings are measurable, and they are what the criterion is designed around. 5.2% of recorded parent→child outcomes are repair events of +0.8 or more, where a broken parent is rewritten wholesale. Whether one lands inside a given five-rollout window has nothing to do with the guideline under test:

A single repair event in a five-rollout window
chance the window contains one23%
shift it gives the window's raw mean≥ +0.16
shift it gives once returns are clipped at ±0.1+0.02

An unclipped window mean would therefore report that repair event rather than the edit:

StepWhat happens
1One candidate rule is installed, and the next five rollouts run under it
2That window is held and scored two updates later, once the lineage backfill has found descendants
3The score is the mean per-rollout return clipped to ±0.1, with win rate as tie-break, so no single repair event decides the verdict
4The reference is the incumbent's window scored at the same age; the candidate is kept if it matches it, reverted otherwise
5No new rule is proposed while a verdict is pending, which keeps "revert" unambiguous

The test rejects most of what reaches it. Over 26 runs on six tasks it returned 132 verdicts: 101 reverts, 31 keeps. The revision reports these counts and releases the accept logs.

What a guideline edit looks like, since that is the object a verdict is passed on. On ahc008 the exploration guideline was rewritten into six hard gates, checked in order:

GateCondition the rule writes
1Circuit-breakerparent delta < −0.10 → force EXPLOIT 2 iterations, exclude that parent
2Cooling-off tablenegatives per parent escalate the lock: 1→1 iter, 2→2, 3→5, 4→exclusion
3Consecutive-EXPLORE cap3 consecutive EXPLORE → force at least one EXPLOIT
4Diversity stall3+ consecutive EXPLORE with no diversity gain → force EXPLOIT
5Parent rotationno parent chosen more than twice in 5 iterations unless it gave a positive delta
6Top-1 protectionthe current best is never used for EXPLORE

The object under test is therefore inspectable,named conditions rather than an opaque prompt change which is what lets a verdict on it carry information.

Q3: does staleness correlate with value-estimation error, or with final quality? Our lag is the text-space analogue of policy lag in asynchronous RL, where rollouts arrive from a policy older than the one now being updated. Work in that setting either corrects the estimator or limits what a stale signal may decide; we do the latter. Because the accept test scores the incumbent's window at the same age as the candidate's, the lag falls on both sides of the comparison and cannot decide a verdict on its own. That leaves whether it costs final quality, which we can measure directly. A W-sweep on DeepSeek, two tasks with three seeds each:

W1248
ahc0110.690.490.630.53
EPLB0.1630.1640.1650.164

Non-monotone and within noise: no measurable quality cost of staleness at W ≤ 8. On the correlation with value-estimation error itself we do not yet have the per-rollout measurement, and we leave that to future work.

We hope this addresses the points raised. If anything remains unclear, or if the reviewer would like us to expand on any of it, we would be glad to continue the discussion.

Reviewer_uAjGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper repurposes the evolutionary search terminology in LLMs into RL terminology. This gives a modular framework for LLM-driven evolutionary program search, incorporating program selection, exploration, experience memory, prompt optimization, and asynchronous scheduling. The experiment reports some non-training results on two models.

Critical strengths

The writing of the paper makes the reviewer ineligible to review or identify the merit of the paper.

Critical weaknesses

The paper seems to be completely AI generated, but the bigger problem is, while I think I have decent expertise in the related area, this paper seems impossible for me to parse or understand. For example,

"The formalization also yields a checkable reading of prior work: these systems are implicitly zerodiscount, not by choice but because fitness is exogenous to an individual rather than a return over successors, leaving a discount factor nothing to act on."

"What we contribute is a general evolutionary substrate that carries this decomposition: an interface in which any one of the estimators can be replaced or switched off without touching the controller, so that the transfer is auditable one mechanism at a time."

Also if I understand correctly, the paper uses the RL terminology, but I could not find any experiment with RL training.

Finally, the current experiments do not fix the total inference cost, which makes the comparison less convincing.

Questions for authors

see weakness

1 comment
AuthorsOpenReview ↗

Point-by-point in the review's order: points 1, 3 and 4 answered from the submitted text, point 5 accepted and now measured

We address the five points in the order of the review.

1. On the summary's reading of the work. What we propose is a substrate that re-organises LLM program evolution along RL lines, together with the algorithms built on top of it so that the search evolves better — not the training of a model by RL. The review describes the paper as repurposing evolutionary-search terminology into RL terminology, and in the following sentence notes that this "gives a modular framework for LLM-driven evolutionary program search, incorporating program selection, exploration, experience memory, prompt optimization, and asynchronous scheduling". We would take the second sentence as the answer to the first: those components are what the modelling choice produces. §1 makes the same point directly — "doing so is not a re-labelling exercise: the modelling choices determine which machinery is applicable, and the machinery then determines the system" — and the choice is concrete: the action is the modular text prefix conditioning a frozen model, not the program it emits (§3.1, Eq. 1), which is what fixes where each estimator attaches. Appendix D.1 gives the data contract that makes the module boundary checkable, and the structure is measurable rather than nominal: switching off any one of M1-M4 lowers the score, and pooled over 28 component-task cells the full configuration comes out ahead in 23. Simply renaming the terminology would leave nothing to switch off.

2. On the paper appearing AI-generated: our use of AI is declared and bounded. We disclosed it at submission — the LLM-usage field records assistance with writing and polishing, and Appendix A states the scope: implementation, analysis scripts and prose editing, with the research questions, experimental designs and conclusions the authors'. We take responsibility for all content of the paper.

3. On the two sentences quoted as hard to parse, here is what each one states.

The sentenceWhat it states
prior systems are implicitly zero-discountthey select by immediate fitness, because no recursion over successor states is defined for them
the modules present an auditable interfaceeach estimator can be switched off independently, so that its contribution can be measured

Both sentences state claims the paper then tests: the first is Observation 1, now stated over population-selection systems; the second is what our leave-one-out ablation measures, where the full configuration comes out ahead in 23 of the 28 cells. We would be happy to expand on either if the reviewer would prefer further clarification.

4. On "no experiment with RL training". We do not claim one, and the submission says so. The abstract reads "...making the transfer auditable one mechanism at a time, with no gradient training of the backend model", and §4 is titled "Modular TextGrad: policy improvement in text space". What is optimised is the natural-language prefix conditioning a frozen model, whose weights are never updated. We will give this more prominence in the revised abstract.

5. On total inference cost. We have now measured it, and once total cost is matched we remain ahead of both baselines, so the gains come from the algorithms rather than from the extra budget. We found this point genuinely useful: acting on it sharpened what we can claim. AGAR spends 2.44× ShinkaEvolve's tokens and an estimated 3.55× OpenEvolve's. At each baseline's own token budget:

At the baseline's own token budgetAGAR winsmedian
vs ShinkaEvolve, AtCoder family9 / 9+9.6%
vs OpenEvolve, AtCoder family8 / 9+13.2%
vs OpenEvolve, all 13 non-saturated tasks11 / 13+6.8%
vs ShinkaEvolve, all 13 non-saturated tasks10 / 13+4.7%

We hope this addresses the points raised. If anything remains unclear, or if the reviewer would like us to expand on any of it, we would be glad to continue the discussion.

Comments on the paper

3 comments
AuthorsOpenReview ↗

(1/2) New experiments: at matched inference cost the algorithms still lead, reach the same quality five times sooner, and every module earns its place

We thank all three reviewers for their careful reading. AGAR has two parts: a substrate that decomposes LLM program evolution into interchangeable RL components, and the algorithms we supply for those components. The experiments we ran during the discussion period speak to both.

What it testsNew experimentWhat we find
the algorithms13 tasks, token budget matched to each baseline's9 of 9 on AtCoder and 10 of 13 overall, where the plain loop reaches 6 of 13
the algorithmsBest-so-far trajectory over 20 budget fractionsA larger area under the curve on 8 of 9 tasks; the median run matches ShinkaEvolve's final score at a fifth of its budget
bothLeave-one-out over M1–M4Removing any of the four costs score (23 of 28 cells)
validity761 programs re-scored on fresh test casesNo winner changes in any of the 20 pairs

1. At matched inference cost, the algorithms still lead. Per run on GLM, AGAR uses 2.44× ShinkaEvolve's tokens and an estimated 3.55× OpenEvolve's. Re-compared with our token budget matched to each baseline's own:

AGAR configurationwins vs SEmedianwins vs OEmedian
our configuration (modules on)10 / 13+4.7%11 / 13+6.8%
modules off (vanilla)6 / 13−0.1%10 / 13+1.1%

The two rows separate the two halves of the contribution. With every module switched off, the bare substrate already matches ShinkaEvolve (6 of 13, median −0.1%) and is ahead of OpenEvolve (10 of 13) — that is what the substrate alone buys. Switching the modules on is what produces the margin.

On the AtCoder family alone we take 9 of 9 against ShinkaEvolve and 8 of 9 against OpenEvolve, and our median margin over the plain loop is +12.3%. At each baseline's own cost we are ahead of both, and the comparison with the modules switched off shows our algorithms are what deliver that. The revision replaces the evaluation-matched main table with the token-matched one.

2. The same quality is reached about five times sooner. Across 20 budget fractions, our best-so-far curve has the larger area on 8 of 9 tasks (median +9.3%), and the median run matches ShinkaEvolve's final score at 0.20 of its budget. A score read at a fixed budget therefore understates a system that converges earlier.

3. Each module earns its place. We ran four arms, each differing from the full configuration by a single module and requiring no change to the controller, over six tasks and three seeds, token-matched. Removing M1 costs a median of 11.8%, M2 4.9%, M3 5.7% and M4 4.8%, and pooled over the 28 cells the full configuration comes out ahead in 23. Each component also accepts other rules — M2 ships with two estimators and M3 with two decision rules, each selected by a configuration field — so an alternative rule can be measured by this same procedure.

4. The results do not depend on scoring against the selection cases. We re-scored the best program of all 761 runs on 100 fresh cases: every system loses between −8.4% and −10.2%, and the winner changes in none of the 20 pairs. The revision adds a held-out column.

The configurations, specified. Part 2 sets out the ladder and the Base/Combo arms as the code runs them, with the DeepSeek column of Table 1 rebuilt so that it is uniform. The ranking is unchanged.

AuthorsOpenReview ↗

(2/2) The configurations as the code runs them, the scope of our claim, and the revision plan

The mechanism, as the code runs it

§3.3 and §6.3 describe the configurations at different levels of detail. Here is the specification; the ranking is unchanged under it.

ComponentWhat the configuration is
the substrate ladderThe levels instantiate none of the transferred modules; L3 is the substrate with every module off, which is the vanilla arm. They are a floor for the loop, not instances of Base and Combo.
Base vs ComboThe M2 estimator is identical in both. They differ in the M3 decision rule, the M1 input format, novelty incubation, and TextGrad — enabled in Base, disabled in Combo. Contribution 2 is therefore exercised by the Base arm.

Table 1's DeepSeek column. Several cells were filled from ladder runs; rebuilt from the modules-on runs so the column is uniform, it reads Base 0.968 / Combo 0.966 against ShinkaEvolve 0.943 and OpenEvolve 0.875 — ranking unchanged. The GLM column was always modules-on. §3.3 and §6.3 are restated to this specification.

Scope of the claim

The cost-matched runs cover all 13 non-saturated tasks. The other six of the 19 are saturated — all four systems within 1% — and are reported but not compared.

What the revision says

The title is unchanged. The abstract keeps its structure; its empirical sentence is replaced by the cost-matched statement. The main table changes basis from matched evaluations to matched tokens and gains a held-out column. §3.2, §6, §7, D.2 and D.7 are revised; the substrate ladder moves to the appendix with its limitations in the caption; the convergence analysis is added as a new section. The revision also positions the asynchronous scheduler against the policy-lag literature in asynchronous RL (AReaL; SAO, arXiv 2607.07508), of which our staleness measure is the text-space counterpart. We cite Oved et al. (arXiv 2610.23977) on token-matched comparison, Gideoni et al. (2602.16805) on simple baselines, and MCTS-AHD (Zheng et al., ICML 2025), which backs child values up to parents and bounds Observation 1 — now stated over population-selection systems rather than over all systems in this literature. We will upload the revised PDF with a change list.

AuthorsOpenReview ↗

Final Summary

We thank the AC and the reviewers for their work on this paper. The reviews converged on a small number of concrete, answerable questions, and each of them is now answered with a new experiment rather than with argument. A revised PDF is uploaded with every change marked in blue.

  1. Core Contribution

AGAR writes LLM program evolution down as a decision process in which the action is the modular prefix the model is conditioned on, not the program it emits. That choice places four RL estimators — credit assignment, value estimation, adaptive exploration, experience memory — at four specific points of the loop, each replaceable or removable without touching the controller. The backend model is frozen throughout and receives no gradient.

The contribution therefore has two halves, and the experiments separate them. With every module switched off, the bare substrate already matches ShinkaEvolve and leads OpenEvolve; switching the modules on is what produces the margin.

  1. Key Concerns Addressed (All in blue in the revised PDF)

Comparison at equal inference cost, not equal evaluation counts (teNF, e48v): we measured the gap — 2.44x ShinkaEvolve's tokens, an estimated 3.55x OpenEvolve's — and truncated every AGAR run at each baseline's own token budget. The configuration still leads on 10 of 13 non-saturated tasks against ShinkaEvolve and 11 of 13 against OpenEvolve, and on 9 of 9 and 8 of 9 within the competitive-programming family. (Section 6.3, Table 2)

Contribution of the individual modules (e48v, teNF): a leave-one-out ablation now reports each module separately. All four cost score when removed, 23 of 28 pooled cells favouring the full configuration (median +5.5%). This replaces the substrate ladder as the empirical argument; the ladder has moved to the appendix with its limitations in the caption. (Section 6.5, Table 3)

Validity of the competitive-programming scores (e48v): the best program of all 761 runs was re-scored on 100 cases generated from fresh seeds, disjoint from the cases that drive selection. The winner changes on 0 of 20 task-backend pairs; every system loses 8-10% alike. The fresh cases are released with the code. (Section 6.3)

Convergence, not only the endpoint (new): over a grid of 20 budget fractions, AGAR's best-so-far curve has the larger area on 8 of 9 tasks (median +9.3%), and the median run reaches ShinkaEvolve's final score at one fifth of its budget. (Section 6.4)

Specification of the mechanism (teNF): Sections 3.3 and 6.3 are restated to the exact configuration of Base and Combo, including the point that the substrate ladder instantiates none of the transferred modules. The decision criterion of the guideline accept test is given in full, together with its measured behaviour: 101 reverts out of 132 verdicts over 26 runs. (Sections 3.3, 4.2, 6.5)

Scope of the formal observation (e48v): the gamma=0 statement is now made over population-selection systems rather than over the literature as a whole, with MCTS-AHD cited as the system that bounds it. It has moved to the appendix; nothing in the main argument depends on it.

What the paper does and does not claim (uAjG): we do not claim to train a policy with reinforcement learning. The model is frozen and receives no gradient. The claim is that the loop can be written down as a decision process, that doing so places four named estimators at four specific points, and that each of them earns its place — which is what the ablation above tests.

  1. Reviewer Status

Reviewer e48v (Rating: 2): all three questions were answered with new experiments — the per-module ablation, the token-matched comparison, and the held-out re-scoring. The reviewer has acknowledged the rebuttal and maintains the score, without identifying a remaining technical concern.

Reviewer teNF (Rating: 2): all three weaknesses and all three questions were addressed, including the two points of specification the reviewer's reading of Sections 3.3 and 6.3 prompted us to state exactly. No response received to date.

Reviewer uAjG (Rating: 1): each of the five points was addressed, including the one we believe rests on a misreading — the paper does not claim to train a policy with reinforcement learning. No response received to date.

With one reviewer having returned to the thread and two not yet able to, we would be grateful if the AC would weigh the new evidence directly. The questions the reviews raised were empirical ones, and each is now settled by a measurement rather than by argument: a comparison at matched inference cost, a leave-one-out ablation over the four modules, a held-out re-scoring on fresh test cases, and a convergence comparison over the whole budget. All of it is marked in blue in the revised PDF, and the data and code are released with it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.