- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoTrias, a framework for LLM-driven program evolution, where the main idea is to separate the responsibilities of governance, proposal, and judgment. Basically, the authors argue that existing approaches either give too much autonomy to the coding agent or too little, and that by separating these responsibilities, one can get better performance and efficiency. The proposed framework combines an evolutionary controller, coding agents, and a protected evaluator. the authors evaluate it on a number of benchmarks comparing mainly to the baseline of CORAL-4, with some results on optimizing a large code repository.
Critical strengths
-
I think the problem that the paper is trying to address is relevant, specially with the recent progress in coding agents and LLM-based program evolution. Having a framework that can manage these agents and make them more efficient is certainly useful.
-
The experiments are relatively extensive, including comparisons with multiple existing methods and across different benchmarks and LLM backbones. I also appreciate that the authors have included quite a few ablation studies and implementation details in the appendix.
-
I found the experiment on evaluator exposure and how this can result in overfitting to be interesting, and I think this is one of the more convincing parts of the paper.
Critical weaknesses
-
The main research contribution is not very clear to me. My main concern with this paper is that it reads more like a technical report of a software framework rather than a research paper. I understand that the authors are proposing to separate governance, proposal, and judgment, however, I am not really convinced what is fundamentally novel about this. To me it seems that the proposed method is mostly putting together a number of already existing ideas and components in a particular way. I think the paper needs to better explain what is the key research insight here, beyond the fact that this particular architecture happens to work well.
-
The experiments do not really validate the main hypothesis of the paper. The authors argue that the allocation of autonomy is the main problem with existing approaches, however, I don't think the experiments really establish this. For instance, the authors argue that having agents manage the governance can lead to unreliable memory, premature narrowing of the search, and unnecessary computational costs. But I could not really find experiments that directly evaluate these claims.
Most of the comparisons are against other systems which differ in many different aspects, so it is difficult to know whether the improvements are actually because of the proposed separation of responsibilities or because of other design choices.
- The ablation studies are not sufficient to isolate the contribution. While the paper includes multiple ablations, most of them seem to be about removing individual components, such as Notes, Meta-guidance, or changing the initialization. I think these are useful, but they don't really answer the main question of the paper, which is whether separating governance from proposal is actually better than letting an agent manage both.
I think a much more interesting experiment would be to keep the same coding agent, evaluator, and budget, and only change how the governance and proposal responsibilities are allocated. Without something like this, I find it difficult to attribute the improvements to the central idea of the paper.
- The large repository experiment is interesting, but the comparison is not very convincing. The authors report improvements on OR-Tools, which is a large C++ repository. However, from what I understand, the actual editing surface is limited to four specified source files. Also, the main comparison seems to be against a single attempt of the coding agent.
I don't think this is a very strong baseline, since naturally running an evolutionary optimization for 50 iterations should have an advantage over a single attempt. I would expect comparisons against simpler approaches, such as running the coding agent multiple times independently or a basic iterative refinement loop, under the same computational budget. It would also be useful to evaluate the final solutions on held-out instances.
Questions for authors
-
What exactly is the main novel research contribution of this paper? I understand the proposed separation of governance, proposal, and judgment, but why should this be considered a new research insight rather than an architectural or engineering design choice? I think this distinction is important and currently not very clear to me.
-
Can the authors provide a more direct experiment to validate the autonomy allocation hypothesis? For instance, keeping the same agent, tools, and budget, and comparing a setup where the agent manages both governance and proposal against one where these responsibilities are separated. I think this would be much more convincing than comparisons between completely different frameworks.
6 comments
Response - 2
Q2: The experiments and ablations do not directly validate the autonomy-allocation hypothesis. A comparison that keeps the same agent, tools, and budget and only changes how governance and proposal are allocated is needed. (W2, W3, Q2)
A2: We are truly grateful for this suggestion, which has meant a great deal to our paper. Q2 does more than raise a concern: it lays out the exact experiment that settles it. We ran it just as you describe (new §6.3), with the same agent, tools, and budget and a single difference, namely who makes the search decisions. It tests the core motivation of our paper directly, and the paper is substantially stronger for it.
Setup.
- The single variable. The search decisions are which parent and inspirations each task starts from, which lessons enter the shared notes, and when to change direction. In configuration A (EvoTrias), Structured Governance makes them. In configuration B, the coding agent that writes the candidates makes them itself, with the full run history available (every candidate with its code, score, and lineage, every failed task, and every note). B's prompt describes its tools and leaves the strategy to the agent.
- Held constant. The coding agent (Claude Code), the model (GLM-5.2), the instructions for constructing a candidate, the protected evaluator, the budget of 50 tasks, the number of initial directions (4) and of concurrent executors (4), and the code version.
- Runs. Three runs per configuration on Circle Packing and Signal Processing. To count tokens in the same way for both configurations, we re-ran A alongside B, so A's scores here differ from Table 1.
Results (Table 19 in Appendix C.11 of the revised paper).
| Owner of the search | Score ↑ | Tokens / run | Tokens / task | Lineages | Top root | Notes / run | Small-Δ |
|---|---|---|---|---|---|---|---|
| Circle Packing | |||||||
| A: Structured Governance | 2.6340 ± 0.0026 | 25.9M | 321K → 383K | 3.7 | 0/3 | 4.0 | 0% |
| B: coding agent | 2.6253 ± 0.0126 | 33.2M | 364K → 675K | 1.7 | 3/3 | 35 | 91% |
| Signal Processing | |||||||
| A: Structured Governance | 0.7597 ± 0.0317 | 27.6M | 368K → 418K | 4.0 | 0/3 | 16.3 | 0% |
| B: coding agent | 0.6391 ± 0.0579 | 32.8M | 355K → 605K | 1.0 | 3/3 | 44.7 | 89% |
Score: final best, mean ± std over three runs. Tokens / task: iterations 5–15 → 40–50. Lineages: root lineages that receive a task after the root phase. Top root: runs whose final best descends from the root with the highest root-phase score. Notes / run: notes written in a run. Lineages and Notes / run are averaged over three runs. Small-Δ: share of notes from attempts within 2% of the parent's score.
Analysis. A scores higher on both tasks: 2.6340 against 2.6253 on Circle Packing, which is close to saturation, and 0.760 against 0.639 on Signal Processing, where A's lowest run (0.736) is above B's highest (0.689). The other columns measure the three costs that you found untested in W2:
- Unnecessary computational cost. B spends 19–28% more tokens per run. Its agent rereads a history that lengthens every iteration, so its tokens per task grow by ×1.70–1.86 from iterations 5–15 to iterations 40–50, against ×1.13–1.20 for A (new Figure 7).
- Premature narrowing. After the root phase, B's search collapses onto 1.0–1.7 of its four root lineages; on Signal Processing, all three runs take every later task from one lineage. In all six B runs the final best descends from the root that scored highest in the root phase, so a root that starts lower gets almost no chance. A keeps 3.7–4.0 lineages in use, and in all six A runs the final best comes from another root.
- Unreliable memory. B writes 35–44.7 notes per run, 89–91% of them from attempts that moved the score by less than 2%. A admits 4.0–16.3 notes per run, none below 2%.
Relation to the evaluator-exposure study. The study you found convincing (§6.3 in the submission, §6.4 in the revised paper) is the same kind of experiment for judge: it leaves part of judge's authority, the evaluator source, open to the coding agent and changes nothing else. The original score rises by 0.093 while the out-of-distribution gain shrinks to 0.009. The new experiment does the same for govern. Together, they show that on top of a division of roles, the boundaries around govern and judge are each necessary.
Revision. New §6.3 with Figure 7; the full table (Table 19) and the protocol are in Appendix C.11.
Response - 3(1)
Q3: The OR-Tools comparison is not convincing: (1) the editing surface is limited to four files; (2) the baseline is a single attempt of the coding agent; (3) there is no evaluation on held-out instances. (W4)
A3:
We really appreciate how carefully you examined the OR-Tools experiment. Your three points are specific and constructive, and addressing them has made this part of the paper considerably more convincing. We answer them in order; (2) and (3) each come with a new experiment.
(1) Editing surface.
The four files limit where a patch may be written. What the proposer has to read and get right is far larger, and this is what places the task beyond a single LLM call.
| Scope | Size |
|---|---|
| Four editable files | 698 + 1,213 + 807 + 2,355 = 5,073 lines; about 30 classes and structs and about 120 functions, none marked for editing |
| Code they depend on | 22 other OR-Tools headers included directly; a dependency closure of 39 files and about 24.9K lines |
| Code that drives them | cp_model_solver.cc (3,081 lines) constructs and schedules this local search; 13 parameters in sat_parameters.proto configure it |
| Module / repository | ortools/sat/ 234K lines / about 680K lines |
Edits inside the four files are also coupled. This local search maintains every constraint violation incrementally, so a change to jump scoring must be carried into the incremental updates and the weight updates; a patch that misses one of them fails to build or silently degrades the solver. Writing a patch that compiles and improves the solver therefore takes understanding code far outside the four files, then building the complete solver and repairing from its output. The edit surface is four files, but the code a working patch depends on spans about 25K lines in 39 files, well beyond what a single LLM call can read and get right without building or running anything. The allowlist itself only protects the build scripts, the evaluator, the instances, and the scoring code, and keeps every candidate a patch that can be reviewed directly.
(2) Baselines under the same budget.
We agree that 50 iterations should be compared with more than a single attempt. Following your suggestion, we added a basic iterative-refinement loop of the agent alone, and the revised submission now includes it (§6.1 and Figure 5; protocol in Appendix C.8).
Setup. The same host and model (Claude Code, DeepSeek-V4-Flash), task description, edit surface, and evaluator, and the same budget of 50 evaluated candidates, without EvoTrias. In each round the agent reads the evaluation result of its previous candidate, makes one change on top of that candidate, and evaluates the new patch; it keeps its own progress log across rounds.
| Method | Evaluated candidates | All 300 instances | 60-instance subset |
|---|---|---|---|
| Single pass of the agent (mean of 4 independent runs) | 1 | −0.1% | +1.3% |
| Iterative refinement by the agent alone | 50 | +4.6% | +2.2% |
| EvoTrias | 50 | +13.2% | +16.9% |
Gain over the unpatched solver. The two instance sets are separate experiments.
Analysis. Iterating does help the agent, as you expected: on the full set it moves from −0.1% to +4.6%. With the same 50 evaluated candidates, EvoTrias reaches +13.2% on the full set and +16.9% on the subset, against +4.6% and +2.2% for the agent alone. This loop is also the simplest form of an agent governing its own search: it decides what to try next from its own record. Its gap to EvoTrias on a repository-scale task is consistent with the controlled ablation in A2.
Response - 1
We sincerely thank Reviewer jtqw (R3) for the detailed and constructive review. We are greatly encouraged that you consider a framework that manages coding agents and makes them more efficient "certainly useful." We are also glad that you find both the evaluator-exposure study and the large-repository experiment "interesting." Below, we provide detailed responses to each of your comments.
Q1: The main research contribution is unclear. Why is the separation of governance, proposal, and judgment a research insight rather than a combination of existing components or an engineering design choice? (W1, Q1)
A1:
Thank you for this question. It gives us the chance to state our core insight and contribution more clearly. Dividing an agent system into roles by responsibility is indeed common practice. Our work, however, studies a fundamentally different problem that begins where this division ends, and to our knowledge it is the first to do so: once the proposer is a coding agent, do the powers stay divided, and what does it cost when they do not?
Before coding agents, the boundaries came for free. While propose was a single LLM call (OpenEvolve, AdaEvolve, EvoX), nobody had to ask. A call sees only its prompt: it cannot read the evaluator or the history or choose its parent, and equally cannot read a repository, build, or debug. A coding agent removes that limit, and its file and shell access removes the boundaries with it. Recent work embraces this and extends the agent's autonomy to the search itself: CORAL's agent also chooses what to build on, writes the shared memory, and finds the grader source in its working directory. Existing systems therefore come in two kinds: boundaries without autonomy, or autonomy without boundaries.
So what does autonomy without boundaries cost? As agents grow stronger, the field is heading this way, yet to our knowledge no prior work has measured the cost. Our paper does, one decision at a time:
The govern row is the premise of the paper: §1 argues that an agent owning the search pays a cognitive cost and a computational cost, and the submission supports this through the comparison with CORAL. The new §6.3, added at your suggestion, isolates it with a single variable: with the agent, model, evaluator, and budget unchanged, handing the search decisions to the agent lowers Signal Processing from 0.760 to 0.639 and raises the token cost of a run by 19–28% (details in A2).
Where the boundary belongs. This had no answer before; the current trend is to draw none and hand more decisions to a stronger agent. Our results give one: the boundary belongs between propose and the other two decisions. EvoTrias keeps the agent's full tool capability inside a task and rebuilds the boundaries outside the agent. To our knowledge, it is the first program-evolution system to do both. Plugging an agentic proposer into an existing controller is not enough: AdaEvolve and EvoX then change by only −1.2% to +1.9% (Appendix C.5).
The next question is how to build these boundaries while keeping the agent's full tool capability. Prompts alone do not hold them: as Appendix A.7 shows, even with the prohibition written in its prompt, one task executor still tried to read
evaluator.py, and another tried to read a sibling task's evaluation record. EvoTrias therefore defines a clear division of power among its three planes (Structured Governance, Open Proposal, and Protected Judgment), in which each decides alone inside its own boundary, and uses runtime hooks to keep these boundaries firm (§4, Appendix A.7).This division has two direct consequences. First, parallelism comes at almost no extra cost. Structured Governance makes the search decisions outside the agent, so running more tasks in parallel only needs more executors, each building one candidate: EvoTrias's cost varies by less than 5% from 1 to 8 concurrent executors, while CORAL's grows 3.9× from one agent to four. Second, EvoTrias runs as a plugin. Govern and judge live outside the agent, and an executor only builds one candidate from an assigned context, so any coding agent that supports MCP, hooks, and subagents can host EvoTrias unmodified.
Revision. New §6.3 reports the controlled ablation (setup and results in A2).