acceptodds
Under review as a conference paper at ICLR 2027

What Do Evolutionary Coding Agents Evolve?

Abstract

Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model's internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of 121 evolutionary coding runs spanning four frameworks, five models, and 16 tasks across mathematics and algorithm design, comprising 10,672 unique programs. We also develop EvoReplay, which reconstructs local search states and supports held-out rescoring, same-prompt replay, and fixed-structure parameter tuning. Using a multi-label LLM judge validated against blind human re-annotation, we assign parent–child edits to nine recurring categories. The most frequent edit categories are not those most strongly associated with beating the parent. The median run reintroduces about 30% of its added lines from code deleted earlier in the same lineage, and these edits more often repair weak parents than advance the run best. Fixed-structure tuning can recover improvements otherwise reached through later LLM edits, exposed-evaluator gains can reverse on held-out instances, and same-prompt replay often recovers much of the original score without reproducing the winning code. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure.

Primary area:
datasets and benchmarks
Keywords:
evolutionary coding agents, llms, ai4math, quality diversity search

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ejpHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents EvoTrace and EvoReplay, a dataset and an analysis framework for studying how evolutionary coding agents improve programs. EvoTrace contains 121 evolutionary runs across four frameworks, five models, and 16 tasks, with more than 10,000 generated programs. EvoReplay supports code-change analysis, edit classification, fixed-structure parameter tuning, held-out evaluation, and prompt replay. The authors report several findings, including frequent reintroduction of previously deleted code, differences between public and private evaluation scores, improvements achievable through parameter tuning alone, and limited reproducibility of generated programs.

Critical strengths

The dataset and replay infrastructure are useful. Collecting complete search histories across multiple frameworks and models provides an opportunity to study the behavior of evolutionary coding agents in detail.

Critical weaknesses
  1. The paper does not establish a sufficiently clear new research contribution. The general problem of understanding LLM-guided evolutionary search has already been studied, including trajectory analysis, fitness landscape analysis, search stagnation, and comparisons with simple optimization baselines. The paper does not clearly identify which important question remains unanswered by these studies. Although EvoTrace provides a larger collection of search histories and EvoReplay supports additional analyses, most findings concern known issues, such as evaluator overfitting, the effectiveness of parameter tuning, and the variability of LLM-generated code. The paper presents several observations, but it is unclear what substantial new understanding of evolutionary coding agents they provide beyond existing empirical studies.

  2. The analysis does not sufficiently establish the causes of performance improvements. The central goal of the paper is to understand how evolutionary coding agents achieve improvements. However, much of the analysis is based on correlations between code changes and evaluation scores rather than controlled experiments that identify the actual causes of improvement. For example, the edit taxonomy shows that certain types of changes are associated with higher chances of beating their parents. However, these changes often have multiple labels, and the experiments do not establish whether a particular type of modification causes the improvement. The paper includes some interventions, but they do not sufficiently isolate the main mechanisms. For example, the Bayesian optimization experiment shows that additional parameter tuning can achieve strong results from an existing program structure. However, this does not establish whether the original evolutionary improvements came from parameter tuning rather than structural changes. A more convincing study would compare different search strategies starting from the same intermediate program, under matched evaluation budgets. Without such controlled comparisons, the paper provides limited insight into why evolutionary search succeeds or fails.

  3. Some key measurements do not adequately support the conclusions drawn from them. The paper reports that approximately 30% of added code lines had previously been deleted from the same lineage. However, the set of previously deleted lines naturally grows over time, making repeated matches increasingly likely. The paper does not establish an appropriate baseline to separate this effect from genuine search cycling. Moreover, reintroducing identical lines does not necessarily mean repeating the same algorithmic behavior. Similarly, the prompt replay experiment reports that regenerated programs achieve a median of 76% of the original program's score. However, this measures absolute performance rather than the fraction of the original improvement that was reproduced. A regenerated program may achieve a similar score to the parent without reproducing the breakthrough at all. These limitations weaken the interpretation of the reported findings as evidence about search dynamics and improvement mechanisms.

  4. The evidence for the main conclusions is limited. Although EvoTrace contains more than 10,000 programs, several important conclusions rely on relatively small subsets. The Bayesian optimization and prompt replay experiments each use 36 selected targets, while the code-recycling suppression experiment covers only ten paired runs. The paper also studies only two main task domains. It is unclear how consistently the reported patterns hold across different tasks, models, and search configurations.

Questions for authors
  1. What is the main new insight about evolutionary coding agents that this paper establishes beyond prior trajectory and search-behavior studies? Which previously unresolved question do the proposed analyses answer?

  2. Can the authors provide stronger causal evidence for the reported improvement mechanisms? For example, can they compare continued LLM evolution, fixed-structure Bayesian optimization, and a combined approach from the same intermediate programs under matched evaluation budgets?

  3. How do the authors distinguish genuine code cycling from the natural increase in repeated-line matches as search history grows? Can they compare the observed rates with an appropriate baseline and examine whether repeated lines correspond to repeated program behavior?

  4. Does prompt replay actually reproduce performance improvements? Can the authors measure the reproduced gain relative to the original parent or previous best program, rather than using the ratio of absolute scores?

Reviewer_X88ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EvoTrace, a dataset of evolutionary coding traces, and EvoReplay, a toolkit for analyzing code reuse, evaluator overfitting, parameter tuning, and replay stability. The study provides a useful unified representation of programs, prompts, parent-child relations, and evaluation logs, but its evidence is based on only 16 tasks from two narrow domains.

However, the paper does not sufficiently establish that the curated dataset can be reused in independent studies, and EvoReplay remains primarily a post hoc diagnostic tool without showing that its findings can improve subsequent evolutionary coding runs.

Critical strengths

The paper studies an underexplored question: Rather than evaluating evolutionary coding agents only by their final scores, it asks what kinds of changes actually produce progress during the search process.

Critical weaknesses
  1. The contribution of the curated EvoTrace dataset alone may not be sufficiently substantial for this track. Although the paper describes a unified schema and provides the EvoReplay package, it does not convincingly demonstrate how EvoTrace can be reused by independent studies or how the dataset supports evaluations beyond the authors' own analyses. The long-term value and reusability of the dataset, therefore, remain unclear.
  2. The evaluation covers a narrow range of tasks. The paper includes 16 tasks in total: 10 C++ competitive-programming tasks from ALE-bench Lite and 6 Python mathematical-discovery tasks. It does not cover other important domains, such as machine-learning engineering (e.g., MLE-Bench) or GPU-kernel optimization. This limited task coverage narrows the scope and generalizability of the reported observations.
  3. EvoReplay functions primarily as a post hoc diagnostic toolkit. The paper identifies several useful patterns in evolutionary coding traces, but it does not show how these observations can be fed back into the evolutionary process to improve coding agents. For example, the authors do not demonstrate that using EvoReplay's diagnostics leads to better search efficiency, final performance, or robustness in a subsequent evolutionary run.
  4. The practical impact of these research questions is unclear. The paper does not sufficiently explain what concrete decisions, system improvements, or real-world benefits follow from identifying the reported trace-level patterns.
Questions for authors

Please see the weaknesses.

Reviewer_KP8LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at what LLM-driven evolutionary program-search systems, such as FunSearch and AlphaEvolve, actually change during search. Its main point is that final benchmark scores alone cannot tell us whether a system discovers new structures, retunes parameters, recombines familiar ideas, or overfits the evaluator.

To study this, the authors release EvoTrace, a dataset of 121 search runs across four frameworks, five LLMs, and 16 tasks. It records programs, prompts, lineage, and evaluation results in a common format, with environments for replay. They also introduce EvoReplay, a toolkit for analyzing edits, tracking reused code, replaying breakthroughs, and testing how much progress can come from tuning a fixed program.

The findings suggest that much of the search involves hyperparameter tuning, while rarer structural edits tend to deliver more value per edit. About 30% of added lines bring back previously deleted code. Replaying breakthroughs rarely produces the same program, but recovers a median 76% of the original score. On math tasks, Bayesian optimization on a mid-run program matches or beats the final search result in 13 of 15 cases. The authors support these findings with annotation checks, replay and tuning experiments, and private-test evaluation.

The paper’s main value is making search traces something researchers can inspect and experimentally test, rather than any single headline result. Its scope is limited to single-call, non-agentic mutations; self-improving agents such as DGM, HGM, and HyperAgents are neither studied nor cited.

Critical strengths
  1. The paper makes evolutionary search easier to study through experiments. Its biggest contribution is making search runs replayable, so researchers can investigate where gains come from. For example, tuning a frozen program helps separate parameter tuning from structural discovery, while same-prompt replay tests whether a breakthrough depends on the exact program or its context. This provides useful infrastructure for understanding search behavior beyond final scores and gives the paper more value than a standard framework comparison.

  2. The shared analysis leads to useful evaluation practices. Applying the same analysis across four frameworks, five LLMs, and two domains makes recurring patterns—such as cycling, tuning dominance, and the gap between edit frequency and utility—more convincing than findings from one framework alone. It also offers two practical recommendations: compare math-task results against a single-program tuning baseline, and check ALE public scores against private tests.

Critical weaknesses
  1. The framing is broader than the experiments. The title refers to “coding agents,” but the experiments study program search where an LLM acts as a single-call mutation operator. Related work on self-improving agents, including DGM, HGM, and HyperAgents, is neither covered nor cited, despite overlapping questions about lineage and credit assignment. It is also unclear whether the findings carry over to mutation operators that use tools and execution feedback. The authors should narrow the framing to “evolutionary program search” or explicitly discuss these boundaries.

  2. Several claims need more direct tests. The paper treats cycling and dead branches as potential waste, but does not show that they hurt performance or test the proposed deletion-aware filters. The toolkit already supports context ablations and structural-only cycling analysis, yet neither is reported. These experiments would help explain what drives the gains and whether cycling reflects meaningful repetition or mostly parameter changes. The infrastructure supports stronger causal analysis than the paper currently delivers.

  3. The dataset needs clearer documentation and stronger validation. The paper does not report how many traces were excluded for non-replayability or provide a complete framework × model × task breakdown. Without repeated runs, framework differences are also hard to separate from search variance. Configurations are author-chosen without validation from the framework developers, and judge validation relies on one annotator who also designed the taxonomy. There are reproducibility gaps too: the README lists three canonical backends while the paper reports four, and breakthrough replay still depends on the original search framework.

Questions for authors
  1. Scope and positioning Do any of the four frameworks use tools, multi-turn interaction, or execution feedback during mutation? If so, could the authors report results separately for agentic and single-call mutation? If not, could they narrow the title and framing to “evolutionary program search” and explain how it differs from DGM, HGM, and HyperAgents in the related work?

  2. Claims needing direct tests Does cycling actually hurt performance? Could the authors report the structural-only cycling rate, using the existing --collapse-numbers --exclude-hyperparams options, and its correlation with final scores? If cycling largely disappears after removing trivial edits and parameter tuning, or has little connection to performance, the “wasted search budget” interpretation would need reconsidering.

  3. Dataset validation Could the authors provide the full framework × model × task run matrix, including repeat counts, and report how many traces were excluded because they could not be replayed and where those exclusions occurred? If each combination has only one run, how can framework differences be distinguished from search variance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.