acceptodds
Desk rejected at ICLR 2027

Stateful Search with Stateless Agents: Scaling Long-Horizon Automated Research

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Automated research systems increasingly run LLM agents over long horizons, but more inference does not by itself produce more progress: agents replay ever-growing histories, duplicate one another's work, or stop experimenting while token consumption continues. Yet most evaluations use short budgets or benchmarks that saturate early, leaving these failure modes untested. We introduce Agentic Evolutionary Search (AES), built on the principle of stateful search with stateless agents: the harness owns candidate solutions and measured outcomes, and reconstructs a fresh, role-specific context for every Advisor epoch and Worker trial. The Advisor turns accumulated evidence into concrete assignments for parallel Workers. We evaluate AES against three recent frameworks on software engineering, kernel optimization, and algorithm design at budgets of up to one billion cumulative tokens. AES achieves the best final result on every evaluated task and reaches the strongest kernel baseline's final performance with over 84% fewer tokens. Ablations from shared checkpoints show that focused contexts and explicit assignments each contribute, with effects that compound over full runs, while the Advisor consumes less than 0.6% of total tokens. These results argue for keeping durable research state out of agent conversations, and show that short evaluation horizons can misjudge both research systems and their components.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Automated Research, LLM Agents, Long-Horizon Search, Multi-Agent Systems, Inference-Time Scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZJnfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Agentic Evolutionary Search (AES), a framework for long-horizon automated research that keeps candidate solutions and evaluation outcomes in harness while reconstructing fresh contexts for an Advisor and parallel Workers. The Advisor uses a summary of results across search directions to assign concrete experiments, and Workers iteratively improve local candidates that may be retained even when they do not yet surpass the global best. The authors compare AES with EvoX, CORAL, and SwarmResearch on software engineering, kernel optimization, and algorithm-design tasks. They report the best full-budget result on every evaluated task, substantial token savings on a replicated kernel task, and checkpoint ablations suggesting that focused contexts and explicit assignments both contribute.

Critical strengths
  1. The paper makes an interesting observation that short evaluations can lead to different conclusions about which system performs better. The ranking on FrontierSWE provide concrete support for this point.
  2. The paper presents a clear and practical design that keeps research state outside agent conversations and constructs fresh contexts for the Advisor and Workers. This design is well motivated for long-horizon search because it limits context growth and is intended to reduce duplicated effort.
  3. The evaluation spans several types of tasks and shows improvements in final solution quality and token efficiency over recent baselines.
Critical weaknesses
  1. The paper does not explain in enough detail how historical evidence is selected, aggregated, and truncated for the Advisor, or how Advisor context resets are implemented. These omissions limit reproducibility of a central part of AES.
  2. The evaluation includes only subsets of FrontierSWE, SOL-ExecBench, and FrontierCS. This is appropriate for studying long-horizon search, but it leaves open whether AES maintains its solution quality and token efficiency on simpler or faster-saturating tasks.
  3. The paper uses a few traces to illustrate duplicated work without assignments and successful refinement across Workers. It does not measure how often these behaviors occur, so AES’s overall performance gains remains unclear.
Questions for authors
  1. Could the authors provide pseudocode for constructing the Advisor’s evidence summary, including selection criteria and token limits?
  2. Do the authors have results on the excluded tasks or on simpler, faster-saturating benchmarks? How does AES perform on them?
  3. Can the authors provide aggregate statistics over the full search traces, such as the frequency of duplicated Worker experiments and the frequency with which one Worker successfully builds on another Worker’s improvement? Such statistics would help establish how much these behaviors contribute to AES’s overall gains.
6 comments
AuthorsOpenReview ↗

Rebuttal — Thank you for the questions and suggestions!

We thank the reviewer for the insightful feedback! We address each concern/question below.

#1. How the Advisor's evidence summary is built

Thank you for pointing this out. We will add the following details and the pseudocode below to the paper. Every epoch, the harness builds the Advisor's context from the stored research state with a fixed program; no model is involved. The context contains: (1) the current best score and the full code of the current best solution; (2) task diagnostics computed on that solution, when the task has them; (3) recent progress, i.e., how many epochs and attempts have passed since the last improvement, and the change in the best score over the last 1, 3, and 6 epochs; (4) the public task files; and (5) an evidence ledger. The ledger has three tables. The first lists each Worker's current direction, local best score, and last status. The second lists the last 24 attempts, one row each: the search direction, the measured score, the change in the global best, and an evidence label. The label is one of "improvement", "local improvement", "measured non-improvement", or "inconclusive" with its cause (output limit, timeout, implementation error, or correctness failure). The third table counts outcomes per search direction over the whole run, and shows the 20 most recently active directions. Directions older than these are no longer listed; what remains of them is the current best code. The ledger never contains rejected code, patches, Worker reports, or any conversation. Truncation is by these row limits and by character caps (240K characters for code, 100K for task files, 280K in total), which were not reached in our runs. In our Anthropic kernel run with Codex (698 experiments, 78 Advisor epochs), the Advisor's input is 19K–33K tokens per epoch. The ledger stays at about 5–6K characters from experiment 120 to experiment 698, while the full experiment record grows to 4.5M characters.

Pseudocode for the Advisor context:

build_advisor_context(state, epoch):
    ctx  = best score, baseline score, epoch number
    ctx += full code of the current global best              # cap: 240K chars
    ctx += task diagnostics of the best solution, if available
    ctx += progress: epochs and attempts since the last improvement;
           for the last 1, 3, 6 epochs: change in best score,
           #improvements, #valid non-improvements, #failures
    ctx += public task files                                 # cap: 100K chars
    ctx += Worker roster: direction, local best, trials, last status
    ctx += last 24 attempts, one row each:
           (id, epoch, direction, status, score, change in best, evidence label)
    ctx += outcome counts per direction over ALL attempts,
           shown for the 20 most recently active directions:
           (#attempts, #valid, #improvements, #valid non-improvements,
            #inconclusive, strength label)
    return ctx                                               # cap: 280K chars

evidence_label(attempt):
    valid score, improved the global best      -> "improvement"
    valid score, improved the local candidate  -> "local improvement"
    valid score, no improvement                -> "measured non-improvement"
    no valid score -> "inconclusive: output limit | timeout | implementation | correctness"

strength_label(direction):
    any improvement of the global best  -> "has improvement"
    two or more valid scores, no win    -> "repeated negative evidence"
    one valid score                     -> "one sample"
    no valid score                      -> "implementation evidence only"

#2. Results on simpler, faster-saturating tasks

We added three tasks used in related work such as EvoX and SwarmResearch: two math optimization tasks (signal processing and Erdős minimum overlap) and one systems optimization task (LLM-SQL). These tasks are simpler than the ones in our submission and saturate much faster, so we run each method for 20M tokens instead of 200M–1B. "–" means that the method has no valid result yet at that budget.

TaskMethod1M tokens5M tokens10M tokens20M tokens
Math: Signal processing ↑AES0.50200.55480.56270.5964
CORAL–0.54300.54450.5456
SwarmResearch0.53280.53570.53690.5371
Math: Erdős min overlap ↓AES0.3817740.3809480.3809420.380940
CORAL–0.3817480.3813440.381341
SwarmResearch0.3812530.3811530.3811530.381153
Systems: LLM-SQL ↑AES0.686140.710260.716630.71821
CORAL–––0.68016
SwarmResearch0.686080.686080.691020.69992

We find that:

  • Solution quality holds. From 5M tokens on, AES has the best result on all three tasks.
  • Token efficiency holds. On all three tasks, AES at 5M tokens is already better than both baselines at 20M. It gets there with a quarter of the tokens.
  • AES is not the fastest at the very start. At 1M tokens, SwarmResearch leads on the two math tasks, and the two methods are tied on LLM-SQL (0.68614 vs. 0.68608). This matches our note in §4.1 that a better final result does not always come with faster early progress.
  • The tasks saturate in different ways. On Erdős minimum overlap, every method levels off, and AES changes by less than 0.00001 after 5M tokens. On LLM-SQL, AES and SwarmResearch both still improve slowly at 20M. On signal processing, AES keeps improving, from 0.555 at 5M to 0.596 at 20M, while CORAL and SwarmResearch each gain less than 0.003.

So the design choices that AES makes for long runs do not hurt it on short, simple tasks. These are single runs, so the smaller gaps should be read with care. We will add these results to the appendix.

#3. Statistics on duplicates and reuse from full experiment traces

We computed the following metrics from the full experiment traces of AES, CORAL, SwarmResearch, and EvoX. Here, an agent is an AES Worker, a CORAL agent, a SwarmResearch search agent, or a single EvoX generation. Each experiment starts from a parent program and changes it. For AES, the parent follows from the harness rules in §3.2: a redirected Worker starts from the global best, and otherwise from its own retained candidate. CORAL and EvoX record the parent directly. SwarmResearch does not record parents, so we use the most similar earlier program on the same branch or on the branch it was created from.

  1. Duplicate experiments. We compare the code change of each experiment (its diff from the parent, ignoring whitespace and line numbers) with the changes made by other agents. An experiment is an exact duplicate if another agent made the same change earlier or produced the same program. It is a near duplicate if another agent made a change at least 80% similar to it, compared line by line, within the previous 30 minutes. We report both as a percentage of all experiments. Note that an exact duplicate can still set a new global best when the same change is applied to a different parent, so we count these cases separately.
  2. Tokens spent on duplicates. The share of all tokens used by exact-duplicate experiments.
  3. Cross-Worker build-on rate. Among experiments that set a new global best, the share whose parent was written by a different agent. This shows how often one agent's improvement becomes the starting point for another agent's next improvement.
  4. Lineage depth of the final best solution. The number of code changes on the path from the seed program to the final best solution, and the number of agents that made them. This counts how many dependent steps the final solution needed.

Anthropic kernel (Codex), one run per method.

MetricAESSwarmResearchEvoX
Final cycles (this run) ↓110912061347
Experiments6982,611293
Exact duplicates (% of experiments) ↓2.1% (15)5.9% (153)5.1% (15)
Near duplicates within 30 min (% of experiments) ↓1.4% (10)6.5% (170)13.7% (40)
Tokens spent on exact duplicates ↓2.3%5.7%3.6%
Exact duplicates that set a new global best300
Cross-Worker build-on rate ↑36% (26/73)15% (15/101)n/a*
Lineage depth of final best (agents involved)77 edits (3 Workers)124 edits (42 agents)24 edits (24 generations)

*Each EvoX generation is a separate agent call, so every child builds on another agent's program by construction.

Lua Native Compiler (FrontierSWE, Codex), one run per method.

MetricAESCORALSwarmResearch
Final score at 700M ↑96.1590.6665.38
Experiments632257972
Exact duplicates (% of experiments) ↓0.0% (0)2.7% (7)8.7% (85)
Near duplicates within 30 min (% of experiments) ↓0.0% (0)4.3% (11)6.4% (62)
Tokens spent on exact duplicates ↓0.0%3.3%11.7%
Exact duplicates that set a new global best020
Cross-Worker build-on rate ↑44% (28/64)18% (5/28)32% (20/62)
Lineage depth of final best (agents involved)63 edits (3 Workers)63 edits (4 agents)59 edits (27 agents)

The traces show that:

  • AES repeats less work. Exact duplicates are 2.1% (kernel) and 0% (Lua) of AES experiments, compared with 2.7–8.7% for the baselines. Near duplicates are 1.4% and 0%, compared with 4.3–13.7%. AES spends at most 2.3% of its tokens on exact duplicates, and the baselines spend up to 11.7%.
  • AES Workers build on each other's progress more often. On Anthropic kernel, 36% of new global bests start from a program written by another Worker. On Lua, 44% do. For CORAL and SwarmResearch, the share is 15–32%.
  • Fresh contexts do not cut progress short. The final AES solution is the end of a chain of 77 dependent edits on Anthropic kernel and 63 on Lua, made by several Workers. On Lua this equals the baselines (63 and 59), and on Anthropic kernel it lies between them (124 and 24).

Two limits apply. These numbers come from one run per method. The duplicate measure also counts only near-identical code. When two agents write the same feature with different code, as in Appendix C.2, it is not counted, so the real amount of repeated work is higher for every method. We therefore read duplicates as one part of the gap between AES and the baselines, not all of it. We will add these tables to the paper.

Reviewer_wrvdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce AES (Agentic Evolutionary Search), a harness focused search algorithm using LLM agents. The study focuses on handling context of the different agents and evaluates how that translates to gains in discovery and token consumption. Rather than providing the advisor agent with long conversation histories, it is provided with a fresh new context of global summaries of all experiments performed and their results. The worker agents only obtains the context needed to perform the local experiments. A persistent research memory is kept alive throughout the study. AES shows improved performance in three benchmarks over other search algorithms. The authors have also presented ablation studies pertaining to context of worker, advisor, number of workers etc.

Critical strengths
  1. The manuscript is well written and easy to follow.
  2. The authors have shown improved performance over other competing frameworks
  3. The design of experiments is done in a way to highlight the major claims of the paper.
  4. The authors show that AES achieves best performance earlier than other frameworks, with respect to the tokens used.
Critical weaknesses
  1. The evaluation covers software engineering, kernel optimization, and algorithm design. While these tasks can involve extended search, the paper does not characterize how much they require dependencies across stages or retention and reuse of earlier findings. The results therefore support sustained computational search more directly than broader long horizon research, such as scientific discovery or multi-stage theorem proving.
  2. Cumulative token count is a useful measure of inference budget, but does not by itself establish task horizon. A simple code fix can consume many tokens through repeated failures or unproductive loops without requiring long horizon reasoning. The paper should distinguish prolonged search from intrinsically long horizon tasks and complement token budgets with measures of sequential depth and dependencies between experiments.
  3. The paper suggests that preventing duplicate work contributes to AES gains, but does not report duplicate attempt rates for AES or the baselines. Such statistics would help substantiate this explanation.
  4. The paper does not highlight the need for the advisor deciding the experiments to run instead of simpler alternatives like workers choosing their own experiments and recording in shared memory to prevent duplication.
Questions for authors
  1. Can there be a better definition or measure of long horizon tasks? Maybe using length of dependent causal chains?
  2. Can the authors provide some ablations on prompts?
  3. Please provide some statistics on the duplicates in full AES and baseline experiments.
  4. Can authors provide some experiments to highlight the need for advisor in comparison to giving autonomy to the workers to decide their own experiment?
5 comments
AuthorsOpenReview ↗

Rebuttal — Thank you for the questions and suggestions!

We thank the reviewer for the insightful feedback! We address each concern/question below.

#1. A better definition of long-horizon tasks

We thank the reviewer for the concrete suggestion in Q1.

Tokens measure the budget, not the task horizon. A run can spend many tokens without doing long-horizon work. Our Figure 2(b) is an example: in that CORAL run, the last improvement arrives at 231M tokens, and the remaining tokens of the 1B budget produce none. We use cumulative tokens because they give a controlled budget axis that is comparable across methods (§2.2). They do not show by themselves that a task needs many dependent steps. We will state this distinction in the paper.

A measure of dependent chains. Following your suggestion, we now report the lineage depth of the final best solution: the number of code changes on the path from the seed program to the final best solution. Every step on this path is a change that was kept and that a later step builds on. We also report how often a new best solution starts from a program written by a different agent, which measures retention and reuse of earlier findings. Definitions and full tables are in #3.

TaskMethodLineage depth of final bestNew bests that build on another agent's program
Anthropic kernelAES77 edits36% (26/73)
SwarmResearch124 edits15% (15/101)
EvoX24 editsn/a
Lua Native CompilerAES63 edits44% (28/64)
CORAL63 edits18% (5/28)
SwarmResearch59 edits32% (20/62)

These traces show that the final solutions emerged through long sequences of code refinement, with measurable reuse across Workers. Lineage depth describes the observed search trajectory; it does not establish the minimum number of necessary steps or prove that every edit is causally required.

Scope of our claims. We agree that our results support sustained, evaluator-grounded search more directly than broader research such as scientific discovery or multi-stage theorem proving. All our tasks have executable evaluators with fast feedback, and the dependencies we measure are between code changes, not between research stages. In the revision, we will:

  1. define "long-horizon" in §2.2 with two parts: a large token budget on a task that still has headroom, and a long chain of dependent improvements;
  2. report lineage depth next to the token budgets for our main tasks;
  3. say "long-horizon search" where that is what we mean, and name scientific discovery and theorem proving as future work in the limitations.

#2. Prompt ablation

Thank you for the suggestion. We ran AES on Anthropic kernel with two more prompt variants, using Claude Code with Claude Opus 4.8:

  • General prompt. The prompts in Appendix D contain task details, such as how the VLIW machine is defined. This variant removes them. Note that the agent should still be able to infer these details from the given starter code by itself, so it has full information to search and find good solutions. This experiment tests whether the gain of AES comes from task knowledge in the prompt.
  • Shortened prompt. We manually cut the Advisor and Worker prompts by about 30%, from about 2.7K to 2.0K tokens. The short version keeps the role split, the output format, and the integrity rules. It removes the instructions for agents with limited tools, the recovery steps for a Worker that hits its output limit, and other guidelines and strategy hints that could be helping performance.

Prompt ablation on Anthropic kernel (Claude Code, Claude Opus 4.8). Cycles (↓), one run per prompt. The baseline row is from Table 1.

PromptCycles at 250M↓Cycles at 1B↓Tokens to match best baseline (M)↓
Best baseline (CORAL)12051130888.3
AES, original prompt10791039138.5
AES, general prompt (no task-specific details)1048101491.5
AES, shortened prompt (strategy hints removed)10991093162.9
  • The advantage of AES does not depend on one prompt. All three prompts beat every baseline in Table 1 at both budgets, and each reaches CORAL's final result with 5–10× fewer tokens.
  • The gain does not come from task details in the prompt. The general prompt, which has none, does best (1014 cycles).
  • The prompt does change the size of the gain. The shortened prompt ends at 1093 cycles, 54 behind the original, so the shortened prompt performed worse in this run. With one run per prompt, we cannot say how much of this spread is run-to-run variation.

We will add this ablation and both prompt variants to the appendix.

#3. Statistics on duplicates and reuse from full experiment traces

We computed the following metrics from the full experiment traces of AES, CORAL, SwarmResearch, and EvoX. Here, an agent is an AES Worker, a CORAL agent, a SwarmResearch search agent, or a single EvoX generation. Each experiment starts from a parent program and changes it. For AES, the parent follows from the harness rules in §3.2: a redirected Worker starts from the global best, and otherwise from its own retained candidate. CORAL and EvoX record the parent directly. SwarmResearch does not record parents, so we use the most similar earlier program on the same branch or on the branch it was created from.

  1. Duplicate experiments. We compare the code change of each experiment (its diff from the parent, ignoring whitespace and line numbers) with the changes made by other agents. An experiment is an exact duplicate if another agent made the same change earlier or produced the same program. It is a near duplicate if another agent made a change at least 80% similar to it, compared line by line, within the previous 30 minutes. We report both as a percentage of all experiments. Note that an exact duplicate can still set a new global best when the same change is applied to a different parent, so we count these cases separately.
  2. Tokens spent on duplicates. The share of all tokens used by exact-duplicate experiments.
  3. Cross-Worker build-on rate. Among experiments that set a new global best, the share whose parent was written by a different agent. This shows how often one agent's improvement becomes the starting point for another agent's next improvement.
  4. Lineage depth of the final best solution. The number of code changes on the path from the seed program to the final best solution, and the number of agents that made them. This counts how many dependent steps the final solution needed.

Anthropic kernel (Codex), one run per method.

MetricAESSwarmResearchEvoX
Final cycles (this run) ↓110912061347
Experiments6982,611293
Exact duplicates (% of experiments) ↓2.1% (15)5.9% (153)5.1% (15)
Near duplicates within 30 min (% of experiments) ↓1.4% (10)6.5% (170)13.7% (40)
Tokens spent on exact duplicates ↓2.3%5.7%3.6%
Exact duplicates that set a new global best300
Cross-Worker build-on rate ↑36% (26/73)15% (15/101)n/a*
Lineage depth of final best (agents involved)77 edits (3 Workers)124 edits (42 agents)24 edits (24 generations)

*Each EvoX generation is a separate agent call, so every child builds on another agent's program by construction.

Lua Native Compiler (FrontierSWE, Codex), one run per method.

MetricAESCORALSwarmResearch
Final score at 700M ↑96.1590.6665.38
Experiments632257972
Exact duplicates (% of experiments) ↓0.0% (0)2.7% (7)8.7% (85)
Near duplicates within 30 min (% of experiments) ↓0.0% (0)4.3% (11)6.4% (62)
Tokens spent on exact duplicates ↓0.0%3.3%11.7%
Exact duplicates that set a new global best020
Cross-Worker build-on rate ↑44% (28/64)18% (5/28)32% (20/62)
Lineage depth of final best (agents involved)63 edits (3 Workers)63 edits (4 agents)59 edits (27 agents)

The traces show three things.

  • AES repeats less work. Exact duplicates are 2.1% (kernel) and 0% (Lua) of AES experiments, compared with 2.7–8.7% for the baselines. Near duplicates are 1.4% and 0%, compared with 4.3–13.7%. AES spends at most 2.3% of its tokens on exact duplicates, and the baselines spend up to 11.7%.
  • AES Workers build on each other's progress more often. On Anthropic kernel, 36% of new global bests start from a program written by another Worker. On Lua, 44% do. For CORAL and SwarmResearch, the share is 15–32%.
  • Fresh contexts do not cut progress short. The final AES solution is the end of a chain of 77 dependent edits on Anthropic kernel and 63 on Lua, made by several Workers. On Lua this equals the baselines (63 and 59), and on Anthropic kernel it lies between them (124 and 24).

Two limits apply. These numbers come from one run per method. The duplicate measure also counts only near-identical code. When two agents write the same feature with different code, as in Appendix C.2, it is not counted, so the real amount of repeated work is higher for every method. We therefore read duplicates as one part of the gap between AES and the baselines, not all of it. We will add these tables to the paper.

#4. Why the Advisor is needed

To justify the importance of the Advisor, we implemented a simple Advisor-free parallel-agent baseline in the AES harness, with fresh contexts, shared global-best code, compact feedback, and self-chosen experiments. Unlike full AES, it restarts every trial from the global best rather than retaining per-Worker local candidates.

  • Parallel agents, no Advisor. Three agents run in parallel, the same number as AES Workers.
  • Context resets. Each agent starts every trial with a fresh context, as in AES.
  • Shared global best. The harness keeps one best solution across all agents. Every agent sees it and starts each trial from it, so an improvement found by one agent becomes the starting point for all agents. Progress is shared, but without an Advisor.
  • Compact feedback. Each agent also sees the status and score of its own previous trial.
  • Self-chosen experiments. No one assigns work. Each agent decides for itself what to explore next.

Both methods use Codex with GPT-5.6-terra at high reasoning effort, with the same seed solutions, evaluators, and token budgets as in the paper. This model is newer than the GPT-5.5 used in the paper, so we reran AES with it. The numbers below should be compared within this table, not with Tables 1–2.

MethodAnthropic kernel: Cycles at 250M↓Anthropic kernel: Cycles at 1B↓libexpat: Score at 175M↑libexpat: Score at 700M↑Git→Zig: Score at 175M↑Git→Zig: Score at 700M↑
Parallel agents (no Advisor)138213624.3718.1217.5123.22
AES (ours)1111109126.8438.7920.5633.79

AES substantially outperforms this simpler baseline. At the full budget, AES needs 20% fewer cycles on Anthropic kernel (1091 vs. 1362), scores 2.1× higher on libexpat (38.79 vs. 18.12), and scores 10.6 points higher on Git→Zig (33.79 vs. 23.22). On Anthropic kernel and libexpat, AES at 25% of the budget is already ahead of the baseline at the full budget.

Shared memory alone does not prevent duplication in the observed CORAL traces. The alternative you describe, where agents choose their own experiments and record them in shared memory, is close to how CORAL works: its agents share attempts and notes. CORAL differs from AES in other ways too, so this is not a clean ablation, but its traces show why shared memory is not enough. Shared notes show what has been done, not what another agent is about to do. In Appendix C.2, an agent posts a note claiming a direction, and a second agent still implements the same feature, because it last read the notes before the claim was posted. The statistics in #3 agree: on Lua, CORAL has 2.7% exact duplicates and an 18% build-on rate, compared with 0% and 44% for AES. The Advisor avoids this problem because it splits the work before any Worker starts, and it is the only agent that sees results across all directions.

Finally, we really appreciate your insightful suggestions, which have already helped us strengthen both the analysis and presentation of this work. We hope the above clarifications resolve the concerns. If the response is helpful, we would be grateful if you could consider raising your score. We are committed to incorporating all promised additions in the revised version to make the paper as valuable as possible to the community.

Reviewer_xVoUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AES, a framework for long-horizon automated research. It stores search history outside agent conversations and uses an Advisor to assign tasks to Workers with fresh contexts. Experiments show that AES achieves better final results than the evaluated baselines and that method rankings can change as the search budget increases.

Critical strengths

The long-budget evaluation is useful. It shows that results from short runs may not reflect how well a method performs over longer searches.

The comparisons use the same models, starting solutions, and evaluators. The repeated kernel experiments and component ablations provide useful support for the design.

Critical weaknesses

The methodological novelty seems limited. The approach combines external memory, context resets, and Advisor–Worker coordination. Although the baselines are relevant, a simpler baseline with context resets and compact feedback is missing. This makes it difficult to judge whether the full design is necessary.

Most tasks have only one run, making the reliability of smaller improvements unclear. Most ablations also start from AES checkpoints, so they provide limited evidence about how each component affects a complete search.

Questions for authors
  1. How does AES compare with a simple agent, or parallel agents, using context resets, a stored best solution, and compact feedback without an Advisor?
  2. Which part of the method do the authors consider the main methodological novelty over existing frameworks?
  3. Would the conclusions hold if the ablations were run from the beginning?
  4. How do the methods compare in total monetary cost, including model calls and execution costs?
6 comments
AuthorsOpenReview ↗

Rebuttal — Thank you for the questions and suggestions!

We thank the reviewer for the insightful feedback! We address each concern/question below.

#1. Novelty compared to existing frameworks

We agree that external memory, context resets, and a coordinator each appear in earlier systems, and we do not claim any of them alone as new. Our contribution is how they are set up for long runs, and the evidence that this setup matters. The paper follows one line: long runs expose two challenges (§2.2), each challenge gets one design (§3.1, §3.2), and each design is tested by ablations. We see two methodological contributions and one evaluation contribution.

Methodological novelty 1: harness-controlled context (§3.1). In prior agentic frameworks, agents manage their own context. CORAL agents resume their own sessions and choose what to read from shared notes. SwarmResearch's Shepherd runs as one long session, and its Optimizer agents continue from their parent's conversation. A reset or compaction does not change this, because what survives is the agent's own summary. In AES, no conversation survives. The harness owns the research state (code and measured scores) and rebuilds every context from it. The Advisor gets outcomes grouped by search direction. Each Worker gets only its assignment, its retained candidate, and local feedback. The sandbox enforces these boundaries.

This matters in long runs:

  • Worker contexts. In full runs (#3), letting Workers read the full history ends at 1336 cycles on Anthropic kernel (AES: 1112) and at 20.28 on libexpat (AES: 27.76). The same change cost only 5–11 cycles in the 100M-token continuations of Table 4, so the cost builds up over a run.
  • Advisor context. Removing the rebuilt summary ends at 1152 cycles and 25.69 (#3). The summary also stays small. Its evidence ledger holds at 5–6K characters while the full record grows to 4.5M, and the Advisor uses 0.24–0.51% of all tokens, compared with 8.4–10.7% for SwarmResearch's Shepherd (Table 5).

Methodological novelty 2: evidence-driven work assignment (§3.2). The two closest frameworks choose not to let a central agent assign experiments. CORAL agents pick their own. SwarmResearch's Shepherd picks parents and agent types but is instructed not to assign specific ideas. In AES, the Advisor is the only agent that sees evidence across all directions. It gives each Worker a concrete experiment and revises it every epoch. The harness also keeps each Worker's local candidate, so a new direction has time to mature before it must beat the global best.

New trace statistics and experiment numbers support this design:

  • Less repeated work. AES has fewer exact-duplicate experiments than every baseline on both tasks and spends fewer tokens on them (table below). These counts cover only near-identical code. They miss cases like Appendix C.2, where two CORAL agents implement the same two features within 13 minutes, although one had posted a note claiming the second.
  • More building on each other. New global bests in AES start from another Worker's program about twice as often as in CORAL and SwarmResearch on Anthropic kernel, and more often on Lua as well (table below).

Trace statistics (Codex, one run per method). An exact duplicate is an experiment whose code change another agent already made. The build-on rate is the share of new global bests that start from another agent's program. Definitions and full tables are in our response to Reviewer wrvd, #3.

MetricKernel: AESKernel: SwarmResearchKernel: EvoXLua: AESLua: CORALLua: SwarmResearch
Exact duplicates (% of experiments) ↓2.1%5.9%5.1%0.0%2.7%8.7%
Tokens spent on exact duplicates ↓2.3%5.7%3.6%0.0%3.3%11.7%
Cross-Worker build-on rate ↑36%15%n/a44%18%32%
  • The simpler parallel-agent baseline performs substantially worse. Parallel agents with the same context resets and stored best solution, but no Advisor, finish far behind AES on all three tasks (#2): 1362 vs. 1091 cycles on Anthropic kernel, 18.12 vs. 38.79 on libexpat, and 23.22 vs. 33.79 on Git→Zig. On the first two, AES at 25% of the budget already beats this baseline at the full budget. Withholding only the per-Worker instructions has a smaller effect (25.81 vs. 27.76 on libexpat; #3).

Evaluation novelty. Beyond the method, we hope readers take away how to evaluate long-horizon automated research.

  • Token budgets and tasks with headroom. Prior evaluations count iterations, model calls, or wall-clock time, and a commonly used benchmark saturates early: on 26-circle packing, four methods are within 2.3×10⁻¹⁰ of each other by 5M tokens (Figure 2(a)). We measure cumulative tokens, run up to 1B, and choose tasks that still improve at that budget.
  • Short runs can mislead by giving an incomplete picture of methods, components, and configurations.
    • Methods: with three runs per method (#5), AES has the best mean at 700M on all four FrontierSWE tasks, and its lead over the strongest baseline is larger at 700M than at 175M on each of them. The growth is largest on Dart→Haskell (+7.5 to +17.6 points) and Lua (+13.7 to +21.2). On Git→Zig, AES and CORAL are within noise at 175M (21.84 ± 1.91 vs. 22.50 ± 3.10), and AES is ahead at 700M (26.51 vs. 24.30). A short run would therefore understate the gap between methods.
    • Components: removing Worker context isolation costs only 5–11 cycles in the 100M-token continuations of Table 4, but 224 cycles over a full 1B-token run (#3). A short ablation would have judged this component as minor.
    • Configurations: wider Worker pools trail by up to 337 cycles at 25% of the budget but finish within 6 cycles of each other at the full budget (Table 6). A short run would have favored fewer Workers.
  • Controlled ablations. Because the harness owns the research state, ablations can resume from the same checkpoint and differ only in the component under test (Table 4).

We therefore recommend reporting results at several cumulative token budgets. In the revision, we will state these contributions more directly in §1 and add an appendix table that compares AES with prior frameworks.

#2. A simpler baseline without an Advisor

Thank you for suggesting additional baselines. We implemented a simple parallel-agent baseline in the AES harness, with fresh contexts, shared global-best code, compact feedback, and self-chosen experiments. Unlike full AES, it restarts every trial from the global best rather than retaining per-Worker local candidates.

  • Parallel agents, no Advisor. Three agents run in parallel, the same number as AES Workers.
  • Context resets. Each agent starts every trial with a fresh context, as in AES.
  • Shared global best. The harness keeps one best solution across all agents. Every agent sees it and starts each trial from it, so an improvement found by one agent becomes the starting point for all agents. Progress is shared, but without an Advisor.
  • Compact feedback. Each agent also sees the status and score of its own previous trial.
  • Self-chosen experiments. No one assigns work. Each agent decides for itself what to explore next.

Both methods use Codex with GPT-5.6-terra at high reasoning effort, with the same seed solutions, evaluators, and token budgets as in the paper. This model is newer than the GPT-5.5 used in the paper, so we reran AES with it. The numbers below should be compared within this table, not with Tables 1–2.

MethodAnthropic kernel: Cycles at 250M↓Anthropic kernel: Cycles at 1B↓libexpat: Score at 175M↑libexpat: Score at 700M↑Git→Zig: Score at 175M↑Git→Zig: Score at 700M↑
Parallel agents (no Advisor)138213624.3718.1217.5123.22
AES (ours)1111109126.8438.7920.5633.79

AES is better on every task and at both budgets. At the full budget, AES needs 20% fewer cycles on Anthropic kernel (1091 vs. 1362), scores 2.1× higher on libexpat (38.79 vs. 18.12), and scores 10.6 points higher on Git→Zig (33.79 vs. 23.22). On Anthropic kernel and libexpat, AES at 25% of the budget is already ahead of the baseline at the full budget. On Git→Zig, the gap grows from 3.1 points at 175M to 10.6 points at 700M.

This shows that context resets and a stored best solution are not enough for long runs. The baseline keeps each context short, but no agent sees results across all directions or turns them into different next experiments for each agent. This is the second challenge in §2.2, and it is what the Advisor is for. This baseline removes the Advisor entirely. In #3 we remove one component at a time, also over full runs.

#3. Full-run ablations

Thank you for this question. We ran each ablation from the beginning for the full budget, on the two tasks of Table 4. The setup matches the paper: Codex with GPT-5.5 at high reasoning effort, the same seed solutions, 1B tokens on Anthropic kernel and 700M on libexpat. The three variants are defined as in §4.2:

  • w/o Advisor context reconstruction: the Advisor reads the full record of past experiments, with no summary computed by the harness.
  • w/o Worker context isolation: Workers can read the full research history and other Workers' workspaces.
  • w/o Advisor work assignments: the Advisor's experiment instructions are withheld from Workers; everything else is unchanged.
ConfigurationAnthropic kernel: Cycles at 1B↓libexpat: Score at 700M↑
AES (full)111227.76
AES w/o Advisor context reconstruction1152 (+40)25.69 (−2.07)
AES w/o Worker context isolation1336 (+224)20.28 (−7.48)
AES w/o Advisor work assignments1119 (+7)25.81 (−1.95)

The conclusions from prior ablation studies still hold. Full AES has the best final result in all six comparisons, as in the checkpoint ablations. The sizes differ by component:

  • Worker context isolation has the largest effect: 224 cycles on Anthropic kernel and 7.5 points on libexpat. In the 100M-token continuations of Table 4, the same ablation cost only 5–11 cycles, so this cost builds up over a full run.
  • Advisor context reconstruction costs 40 cycles on Anthropic kernel and 2.07 points on libexpat.
  • Advisor work assignments have the smallest effect in full runs: 7 cycles and 2.0 points. Note that this ablation withholds only the per-Worker instructions, and Workers still receive the rest of the Advisor's output. When the Advisor is removed entirely (#2), the gap is much larger.

We will add these results to the paper.

#4. Total monetary cost, including model calls and execution

Thank you for this question. The submission reported model cost only for the coordinator (Table 5) and left out execution cost, which we listed as a limitation. We now report the full cost on two tasks: Anthropic kernel, where an evaluation is almost free, and Lua Native Compiler (FrontierSWE), where every evaluation runs in a Modal container. Model cost uses GPT-5.5 list prices: $5 / $0.50 / $30 per 1M uncached input / cached input / output tokens.

Anthropic kernel, 1B-token budgetAESSwarmResearchEvoX
Final cycles ↓110912061347
Model cost ($)1,138989942
… of which Advisor ($)14 (1.2%)––
Evaluations6982,611293
Execution cost ($)*<1<1<1
Total cost ($)≈1,138≈989≈942
Wall-clock time (h)39.216.317.7
AES cost to reach this method's final score ($)–≈150≈60

*Each evaluation runs a CPU simulator for under a second. SwarmResearch's 2,611 evaluations took 0.8 CPU-hours in total.

Lua Native Compiler, 700M-token budgetAESCORALSwarmResearch
Final score ↑96.1590.6665.38
Model cost ($)815813611
… of which Advisor ($)18 (2.2%)––
Evaluations632257972
Evaluation container time (h)19.6≈1140.0
Execution cost ($)*5≈2–310
Total cost ($)820≈816621
Wall-clock time (h)21.910.113.7
AES cost to reach this method's final score ($)–435≈140

*Modal at standard rates with a 4-core, 8 GB container per evaluation.

What the tables show:

  • At the same token budget, AES costs more. Its total cost is 15% above SwarmResearch on Anthropic kernel (1,138 vs. 989) and 32% above SwarmResearch on Lua (820 vs. 621). There are two reasons. Fresh Worker sessions get fewer cache hits (94.3% vs. 95.6% of input tokens on Anthropic kernel), and AES agents write more output tokens (13.0M vs. 9.8M).
  • AES reaches each baseline's final score at a lower cost. On Anthropic kernel, AES passes SwarmResearch's final result after about 150 dollars (SwarmResearch's full run: 989) and EvoX's after about 60 dollars (EvoX: 942). On Lua, AES passes CORAL's final score after 435 dollars (CORAL: 816) and SwarmResearch's after about 140 dollars (SwarmResearch: 621). AES then keeps improving to the best final score on both tasks.
  • Execution cost is small. It is under 1 dollar on Anthropic kernel and at most 10 dollars on Lua, which is 1.6% of that run's total. Including it does not change the comparison.
  • The Advisor is cheap. It accounts for 1.2–2.2% of AES's model cost.
  • Wall-clock time. With our default of three Workers, AES takes about twice as long as the baselines. This is a configuration choice. With 15 Workers, the same 1B-token kernel run takes 14.8 hours and ends at 1107 cycles (Table 6).

We will add these tables to the appendix and remove the matching sentence from the limitations.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion and Rebuttal Summary

Dear Area Chair and Reviewers,

We would like to briefly summarize the progress made during the rebuttal and discussion period.

All three reviewers (xVoU, wrvd, ZJnf) recommended weak acceptance. Each of them replied to our rebuttal, kept their rating, and raised no further concerns.

How reviewer concerns are addressed

  1. Novelty and the need for the Advisor (xVoU, wrvd). We added the requested simpler baseline: parallel agents with context resets, a shared best solution, and compact feedback, but no Advisor. AES is better on all three tasks at both budgets. At the full budget: 1091 vs. 1362 cycles on Anthropic kernel, 38.79 vs. 18.12 on libexpat, and 33.79 vs. 23.22 on Git→Zig. We also stated our two method contributions (harness-controlled context, evidence-driven work assignment) and our evaluation contribution more directly.
  2. Single runs and checkpoint ablations (xVoU). We reran all four FrontierSWE tasks with three runs per method. AES has the best mean at the full budget on all four, with an average lead of 11.6 points. We also reran the ablations from the beginning for the full budget. Full AES is best in all six comparisons, so the earlier conclusions hold.
  3. Total cost (xVoU). We now report model plus execution cost. At the same token budget AES costs up to 32% more, but it reaches every baseline's final score at a lower cost. Execution is at most 1.6% of the total.
  4. Duplicate and reuse statistics (wrvd, ZJnf). We computed these from the full traces. AES has fewer exact duplicates (0–2.1% vs. 2.7–8.7% of experiments), and its Workers build on each other more often (36–44% vs. 15–32% of new global bests).
  5. Definition of long-horizon and scope of claims (wrvd). We now report lineage depth as a measure of dependent steps (77 and 63 edits behind the final AES solutions). We will define "long-horizon" more clearly in §2.2.
  6. Prompt ablation (wrvd). We tested two more prompt variants. Both beat every baseline at both budgets, and the variant with no task-specific details does best. This suggests that the gain of AES does not come from a specific prompt.
  7. Advisor context details (ZJnf). We gave a full description and pseudocode of how the evidence summary is built, including the selection rules and size caps.
  8. Simpler, faster-saturating tasks (ZJnf). We added three such tasks. From 5M tokens on, AES has the best result on all three.

We are sincerely grateful for the reviewers’ detailed feedback, which has already helped us significantly strengthen the paper, and we remain fully committed to incorporating all promised additions and clarifications into the final version.

Thank you for your time and consideration.

Sincerely,
Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.