What Do Evolutionary Coding Agents Evolve?

Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model's internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of 121 evolutionary coding runs spanning four frameworks, five models, and 16 tasks across mathematics and algorithm design, comprising 10,672 unique programs. We also develop EvoReplay, which reconstructs local search states and supports held-out rescoring, same-prompt replay, and fixed-structure parameter tuning. Using a multi-label LLM judge validated against blind human re-annotation, we assign parent–child edits to nine recurring categories. The most frequent edit categories are not those most strongly associated with beating the parent. The median run reintroduces about 30% of its added lines from code deleted earlier in the same lineage, and these edits more often repair weak parents than advance the run best. Fixed-structure tuning can recover improvements otherwise reached through later LLM edits, exposed-evaluator gains can reverse on held-out instances, and same-prompt replay often recovers much of the original score without reproducing the winning code. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure.