- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Agentic Evolutionary Search (AES), a framework for long-horizon automated research that keeps candidate solutions and evaluation outcomes in harness while reconstructing fresh contexts for an Advisor and parallel Workers. The Advisor uses a summary of results across search directions to assign concrete experiments, and Workers iteratively improve local candidates that may be retained even when they do not yet surpass the global best. The authors compare AES with EvoX, CORAL, and SwarmResearch on software engineering, kernel optimization, and algorithm-design tasks. They report the best full-budget result on every evaluated task, substantial token savings on a replicated kernel task, and checkpoint ablations suggesting that focused contexts and explicit assignments both contribute.
Critical strengths
- The paper makes an interesting observation that short evaluations can lead to different conclusions about which system performs better. The ranking on FrontierSWE provide concrete support for this point.
- The paper presents a clear and practical design that keeps research state outside agent conversations and constructs fresh contexts for the Advisor and Workers. This design is well motivated for long-horizon search because it limits context growth and is intended to reduce duplicated effort.
- The evaluation spans several types of tasks and shows improvements in final solution quality and token efficiency over recent baselines.
Critical weaknesses
- The paper does not explain in enough detail how historical evidence is selected, aggregated, and truncated for the Advisor, or how Advisor context resets are implemented. These omissions limit reproducibility of a central part of AES.
- The evaluation includes only subsets of FrontierSWE, SOL-ExecBench, and FrontierCS. This is appropriate for studying long-horizon search, but it leaves open whether AES maintains its solution quality and token efficiency on simpler or faster-saturating tasks.
- The paper uses a few traces to illustrate duplicated work without assignments and successful refinement across Workers. It does not measure how often these behaviors occur, so AES’s overall performance gains remains unclear.
Questions for authors
- Could the authors provide pseudocode for constructing the Advisor’s evidence summary, including selection criteria and token limits?
- Do the authors have results on the excluded tasks or on simpler, faster-saturating benchmarks? How does AES perform on them?
- Can the authors provide aggregate statistics over the full search traces, such as the frequency of duplicated Worker experiments and the frequency with which one Worker successfully builds on another Worker’s improvement? Such statistics would help establish how much these behaviors contribute to AES’s overall gains.
6 comments
#2. Results on simpler, faster-saturating tasks
We added three tasks used in related work such as EvoX and SwarmResearch: two math optimization tasks (signal processing and Erdős minimum overlap) and one systems optimization task (LLM-SQL). These tasks are simpler than the ones in our submission and saturate much faster, so we run each method for 20M tokens instead of 200M–1B. "–" means that the method has no valid result yet at that budget.
| Task | Method | 1M tokens | 5M tokens | 10M tokens | 20M tokens |
|---|---|---|---|---|---|
| Math: Signal processing ↑ | AES | 0.5020 | 0.5548 | 0.5627 | 0.5964 |
| CORAL | – | 0.5430 | 0.5445 | 0.5456 | |
| SwarmResearch | 0.5328 | 0.5357 | 0.5369 | 0.5371 | |
| Math: Erdős min overlap ↓ | AES | 0.381774 | 0.380948 | 0.380942 | 0.380940 |
| CORAL | – | 0.381748 | 0.381344 | 0.381341 | |
| SwarmResearch | 0.381253 | 0.381153 | 0.381153 | 0.381153 | |
| Systems: LLM-SQL ↑ | AES | 0.68614 | 0.71026 | 0.71663 | 0.71821 |
| CORAL | – | – | – | 0.68016 | |
| SwarmResearch | 0.68608 | 0.68608 | 0.69102 | 0.69992 |
We find that:
- Solution quality holds. From 5M tokens on, AES has the best result on all three tasks.
- Token efficiency holds. On all three tasks, AES at 5M tokens is already better than both baselines at 20M. It gets there with a quarter of the tokens.
- AES is not the fastest at the very start. At 1M tokens, SwarmResearch leads on the two math tasks, and the two methods are tied on LLM-SQL (0.68614 vs. 0.68608). This matches our note in §4.1 that a better final result does not always come with faster early progress.
- The tasks saturate in different ways. On Erdős minimum overlap, every method levels off, and AES changes by less than 0.00001 after 5M tokens. On LLM-SQL, AES and SwarmResearch both still improve slowly at 20M. On signal processing, AES keeps improving, from 0.555 at 5M to 0.596 at 20M, while CORAL and SwarmResearch each gain less than 0.003.
So the design choices that AES makes for long runs do not hurt it on short, simple tasks. These are single runs, so the smaller gaps should be read with care. We will add these results to the appendix.
#3. Statistics on duplicates and reuse from full experiment traces
We computed the following metrics from the full experiment traces of AES, CORAL, SwarmResearch, and EvoX. Here, an agent is an AES Worker, a CORAL agent, a SwarmResearch search agent, or a single EvoX generation. Each experiment starts from a parent program and changes it. For AES, the parent follows from the harness rules in §3.2: a redirected Worker starts from the global best, and otherwise from its own retained candidate. CORAL and EvoX record the parent directly. SwarmResearch does not record parents, so we use the most similar earlier program on the same branch or on the branch it was created from.
- Duplicate experiments. We compare the code change of each experiment (its diff from the parent, ignoring whitespace and line numbers) with the changes made by other agents. An experiment is an exact duplicate if another agent made the same change earlier or produced the same program. It is a near duplicate if another agent made a change at least 80% similar to it, compared line by line, within the previous 30 minutes. We report both as a percentage of all experiments. Note that an exact duplicate can still set a new global best when the same change is applied to a different parent, so we count these cases separately.
- Tokens spent on duplicates. The share of all tokens used by exact-duplicate experiments.
- Cross-Worker build-on rate. Among experiments that set a new global best, the share whose parent was written by a different agent. This shows how often one agent's improvement becomes the starting point for another agent's next improvement.
- Lineage depth of the final best solution. The number of code changes on the path from the seed program to the final best solution, and the number of agents that made them. This counts how many dependent steps the final solution needed.
Anthropic kernel (Codex), one run per method.
| Metric | AES | SwarmResearch | EvoX |
|---|---|---|---|
| Final cycles (this run) ↓ | 1109 | 1206 | 1347 |
| Experiments | 698 | 2,611 | 293 |
| Exact duplicates (% of experiments) ↓ | 2.1% (15) | 5.9% (153) | 5.1% (15) |
| Near duplicates within 30 min (% of experiments) ↓ | 1.4% (10) | 6.5% (170) | 13.7% (40) |
| Tokens spent on exact duplicates ↓ | 2.3% | 5.7% | 3.6% |
| Exact duplicates that set a new global best | 3 | 0 | 0 |
| Cross-Worker build-on rate ↑ | 36% (26/73) | 15% (15/101) | n/a* |
| Lineage depth of final best (agents involved) | 77 edits (3 Workers) | 124 edits (42 agents) | 24 edits (24 generations) |
*Each EvoX generation is a separate agent call, so every child builds on another agent's program by construction.
Lua Native Compiler (FrontierSWE, Codex), one run per method.
| Metric | AES | CORAL | SwarmResearch |
|---|---|---|---|
| Final score at 700M ↑ | 96.15 | 90.66 | 65.38 |
| Experiments | 632 | 257 | 972 |
| Exact duplicates (% of experiments) ↓ | 0.0% (0) | 2.7% (7) | 8.7% (85) |
| Near duplicates within 30 min (% of experiments) ↓ | 0.0% (0) | 4.3% (11) | 6.4% (62) |
| Tokens spent on exact duplicates ↓ | 0.0% | 3.3% | 11.7% |
| Exact duplicates that set a new global best | 0 | 2 | 0 |
| Cross-Worker build-on rate ↑ | 44% (28/64) | 18% (5/28) | 32% (20/62) |
| Lineage depth of final best (agents involved) | 63 edits (3 Workers) | 63 edits (4 agents) | 59 edits (27 agents) |
The traces show that:
- AES repeats less work. Exact duplicates are 2.1% (kernel) and 0% (Lua) of AES experiments, compared with 2.7–8.7% for the baselines. Near duplicates are 1.4% and 0%, compared with 4.3–13.7%. AES spends at most 2.3% of its tokens on exact duplicates, and the baselines spend up to 11.7%.
- AES Workers build on each other's progress more often. On Anthropic kernel, 36% of new global bests start from a program written by another Worker. On Lua, 44% do. For CORAL and SwarmResearch, the share is 15–32%.
- Fresh contexts do not cut progress short. The final AES solution is the end of a chain of 77 dependent edits on Anthropic kernel and 63 on Lua, made by several Workers. On Lua this equals the baselines (63 and 59), and on Anthropic kernel it lies between them (124 and 24).
Two limits apply. These numbers come from one run per method. The duplicate measure also counts only near-identical code. When two agents write the same feature with different code, as in Appendix C.2, it is not counted, so the real amount of repeated work is higher for every method. We therefore read duplicates as one part of the gap between AES and the baselines, not all of it. We will add these tables to the paper.
Rebuttal — Thank you for the questions and suggestions!
We thank the reviewer for the insightful feedback! We address each concern/question below.
#1. How the Advisor's evidence summary is built
Thank you for pointing this out. We will add the following details and the pseudocode below to the paper. Every epoch, the harness builds the Advisor's context from the stored research state with a fixed program; no model is involved. The context contains: (1) the current best score and the full code of the current best solution; (2) task diagnostics computed on that solution, when the task has them; (3) recent progress, i.e., how many epochs and attempts have passed since the last improvement, and the change in the best score over the last 1, 3, and 6 epochs; (4) the public task files; and (5) an evidence ledger. The ledger has three tables. The first lists each Worker's current direction, local best score, and last status. The second lists the last 24 attempts, one row each: the search direction, the measured score, the change in the global best, and an evidence label. The label is one of "improvement", "local improvement", "measured non-improvement", or "inconclusive" with its cause (output limit, timeout, implementation error, or correctness failure). The third table counts outcomes per search direction over the whole run, and shows the 20 most recently active directions. Directions older than these are no longer listed; what remains of them is the current best code. The ledger never contains rejected code, patches, Worker reports, or any conversation. Truncation is by these row limits and by character caps (240K characters for code, 100K for task files, 280K in total), which were not reached in our runs. In our Anthropic kernel run with Codex (698 experiments, 78 Advisor epochs), the Advisor's input is 19K–33K tokens per epoch. The ledger stays at about 5–6K characters from experiment 120 to experiment 698, while the full experiment record grows to 4.5M characters.
Pseudocode for the Advisor context: