- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Population-Centered Evolution (PCE), a framework that uses population-wide experience to guide agent self-improvement. The authors identify a gap between the tasks solved collectively by an evolutionary population and those solved by its best individual agent. To address this, PCE distils execution records and modification histories across agent variants into a population block. This block is then used to diagnose and improve a selected parent's harness. Experiments on SWE-bench Verified and Polyglot show that PCE outperforms Group-Evolving Agents (GEA) by 3.2 and 5.9 percentage points. PCE also reduces the population coverage gap and uses fewer tokens during evolution.
Critical strengths
The paper highlights a useful gap between population coverage and individual agent performance. PCE extends experience sharing beyond selected parents and uses cross-agent evidence to guide harness modifications. This is a reasonable direction for improving evolutionary agent systems.
The experiments compare PCE with GEA under matched models and evolution settings. PCE improves performance on both benchmarks and retains gains on held-out tasks. The parent-only distillation ablation provides useful evidence for the value of population-wide information. The reduction in token consumption is also encouraging.
Critical weaknesses
1. Limited baseline comparisons and insufficient validation of the proposed mechanism.
The evaluation focuses primarily on GEA. Results from DGM and other methods in Table 1 use different experimental settings, while controlled comparisons with recent methods such as Self-Harness and HarnessEvolve are missing. The ablations also do not isolate the contributions of individual population-block components or compare against simpler experience aggregation strategies. More importantly, the claimed capability consolidation is supported mainly by a reduced population coverage gap. This does not directly establish whether agents acquire capabilities discovered by other variants. These limitations leave the specific advantages of PCE insufficiently demonstrated.
2. Limited experimental robustness and generalizability.
SWE-bench includes only two independent evolution runs per method, while Polyglot includes just one. The five repeated evaluations measure the variability of selected agents but do not establish the stability of the evolution process. This is particularly concerning given the reported 100% Polyglot search score. The experiments are also limited to one model family and ten generations. Furthermore, the substantial token savings coincide with shorter execution trajectories, but their source is not clearly established. These limitations raise concerns about the reliability and generalizability of the reported improvements.
Questions for authors
-
Could the authors provide stronger baseline comparisons and more comprehensive ablations? For example, comparing PCE with Self-Harness or HarnessEvolve under matched settings would help establish its advantages over existing methods. Ablating individual population-block components and comparing against simpler strategies such as trajectory retrieval or failure clustering would also clarify which parts of PCE are necessary.
-
Could the authors provide additional independent evolution runs on Polyglot? Since the reported 100% search score and zero coverage gap are based on a single run, it remains unclear whether these results are reproducible across different evolution trajectories.
-
Can the authors measure how often a child solves tasks that its selected parents failed but other population members previously solved? Comparing this rate against GEA would provide more direct evidence of cross-agent capability consolidation.
-
Could the authors break down token consumption by execution, diagnosis, self-modification, and distillation?
-
Do the findings hold across longer evolution horizons, larger populations, or different model families? For example, extending evolution to 20–30 generations and evaluating with other models such as GPT or Kimi would help assess generalizability. Also testing on more recent challenging benchmarks would provide further evidence beyond the current evaluation settings.
12 comments
Weakness 1 & Question 1: Ablations and consolidation
2. Ablations. Our main argument is that agent evolution should draw on population-level evidence. An ablation of each component in the population block takes this argument one step further, showing where the gain comes from. We run this ablation on Polyglot, keeping distillation unchanged and varying only the components that diagnosis receives. Trajectory retrieval and failure clustering usually serve to improve a single agent iteratively rather than to evolve a population. The two approaches are complementary, and each suits different scenarios. To test this kind of strategy within population evolution, we add member summaries. They summarize the record of every population member separately, so diagnosis sees the whole population but none of the population-block components.
The results show that the gain does not depend on any single component. Outcome evidence alone and the observation alone each reach 98.3%, and together they reach 100%. In contrast, variants without population-level evidence reach 90.0% to 93.3% after ten generations. The gain therefore comes from evidence about how the whole population performs on a task, and each component supplies this evidence.
| Population-level evidence | Variant | By generation 2 | After ten generations |
|---|---|---|---|
| Without | GEA | 85.0 | 90.0 |
| Parent-only distillation | 80.0 | 91.7 | |
| Member summaries | 83.3 | 93.3 | |
| With | Outcome evidence only | 98.3 | 98.3 |
| Cross-agent observation only | 95.0 | 98.3 | |
| Outcome evidence and observation | 96.7 | 100.0 | |
| Full population block | 96.7 | 100.0 |
Best search score so far on Polyglot Held-in 60, in percent, from Figure 4a and Table 6 of the revised paper.
- Retrieved records. GEA gives diagnosis the records of the selected parents, and parent-only distillation adds an analysis step over the same records. Over the generations, the records that GEA reads add up to about as much evidence as member summaries provide. GEA, parent-only distillation, and member summaries therefore end close together. Neither the number of trajectories nor additional analysis is the bottleneck.
- Outcome evidence. Outcome evidence reports the distribution of task outcomes across the population. With outcome evidence alone, the best search score reaches 98.3% by generation 2. This already exceeds the 90.0% that GEA reaches after ten generations.
- Cross-agent observation. The cross-agent observation provides a hypothesis about why the outcomes of population members differ. With the observation alone, the best search score reaches 95.0% by generation 2 and 98.3% by generation 5.
- Candidate files. Candidate files identify parts of the harness that may implement the relevant behavior. They guide where diagnosis looks, so we evaluate them together with the other two components. Outcome evidence and the observation together reach 100%, and so does the full population block with candidate files.
Why population distillation works. As Section 3.1 describes, each component gives diagnosis evidence about how the whole population performs on a task. With this evidence, diagnosis can locate the problem in the harness that most needs to change. Parent-only distillation and member summaries lack this evidence, and they end close to GEA. Outcome evidence and the observation therefore each achieve most of the gain on their own, and together the components reach 100%.
3. Consolidation. We especially thank the reviewer for proposing a direct measure of consolidation, which we find very useful. We adopt it in the revised paper as the consolidation rate, a new column of Table 2 defined in Appendix D.1. On Polyglot, PCE children solve 72.9% of the tasks that their parents fail but earlier population members solve, against 44.6% for GEA.
Therefore, the gain of PCE comes from population-level evidence, which every component provides, rather than from retrieved or summarized records. The consolidation rate gives the direct evidence that the reviewer asked for. In the revised paper, Figure 4a and Section 5.3 report the ablation, and Table 6 in Appendix A lists every child of each variant. Table 2 and Appendix D.1 report the consolidation rate. We hope this answer addresses the reviewer's concern.
Weakness 2 & Question 2: Robustness of the evolution process
W2. Limited experimental robustness and generalizability. SWE-bench includes only two independent evolution runs per method, while Polyglot includes just one. The five repeated evaluations measure the variability of selected agents but do not establish the stability of the evolution process. This is particularly concerning given the reported 100% Polyglot search score. The experiments are also limited to one model family and ten generations. Furthermore, the substantial token savings coincide with shorter execution trajectories, but their source is not clearly established. These limitations raise concerns about the reliability and generalizability of the reported improvements.
Q2. Could the authors provide additional independent evolution runs on Polyglot? Since the reported 100% search score and zero coverage gap are based on a single run, it remains unclear whether these results are reproducible across different evolution trajectories.
A2: We thank the reviewer for this suggestion. All three reviewers raised this question, so we address it in detail. Our setting follows DGM and GEA, whose main results each come from a single evolution run per benchmark. As both papers report, one such run costs DGM about USD 22,000 on SWE-bench. For GEA it costs about USD 13,000 on SWE-bench and USD 1,500 on Polyglot. Our submission reported two SWE-bench runs per method and one Polyglot run.
We agree that randomness is inherent in agent evolution. We therefore spend additional budget and compute on new runs and report them in the revised paper. The improvement of PCE over GEA is reproducible and does not rest on a single favorable run. The evidence below covers independent runs and twenty-generation runs, and it also shows that the evolved harness transfers to other execution models.
- The best PCE agent beats the best GEA agent in every run. The table below lists the best search score, coverage gap, and consolidation rate of each run, together with the means from Table 2.
| Benchmark | Run | Best search score GEA | Best search score PCE | Coverage gap GEA | Coverage gap PCE | Consolidation rate GEA | Consolidation rate PCE |
|---|---|---|---|---|---|---|---|
| SWE-bench | Run 1 | 71.7 | 75.0 | 15.0 | 10.0 | 25.5 | 41.9 |
| Run 2 | 68.3 | 75.0 | 13.3 | 10.0 | 40.6 | 29.2 | |
| Mean ± sd | 70.0 ± 2.4 | 75.0 ± 0.0 | 14.2 ± 1.2 | 10.0 ± 0.0 | 33.1 ± 10.7 | 35.6 ± 9.0 | |
| Polyglot | Run 1 | 90.0 | 100.0 | 10.0 | 0.0 | 37.0 | 59.1 |
| Run 2 | 85.0 | 98.3 | 15.0 | 1.7 | 45.5 | 81.0 | |
| Run 3 | 81.7 | 100.0 | 18.3 | 0.0 | 51.2 | 78.7 | |
| Mean ± sd | 85.6 ± 4.2 | 99.4 ± 1.0 | 14.4 ± 4.2 | 0.6 ± 1.0 | 44.6 ± 7.1 | 72.9 ± 12.0 |
- The gain exceeds the variation between runs. On Polyglot, the lowest PCE run reaches 98.3% and the highest GEA run reaches 90.0%. On SWE-bench, both PCE runs reach 75.0% and the GEA runs reach 71.7% and 68.3%. The weakest PCE run therefore beats the strongest GEA run on both benchmarks, as Tables 5 and 6 in Appendix A show.
- Strong agents recur over twenty generations. In the twenty-generation run of Section 5.5 and Figure 4b, PCE produces four agents that solve all 60 held-in tasks. They appear at generations 2, 4, 4, and 11. The best GEA agent over the same twenty generations reaches 91.7%.
- The evolved harness transfers to other execution models. The PCE agent runs unchanged under seven other execution models from five providers. It outperforms the initial agent under every model, by 25.7 points on average, as Section 5.5 and Figure 4c report.
- PCE also leads in repeated and held-out evaluation. Over five repeated evaluations on Polyglot, the selected PCE agent averages 96.0%, against 92.0% for the selected GEA agent. The selected PCE agents also beat GEA on every held-out task set, including 97.0% against 90.3% on Polyglot Held-out 165, as Table 8 reports.
Therefore, the advantage of PCE is reproducible across independent runs, since its best agent outperforms that of GEA in every independent run. The perfect search score holds across runs. Two of the three Polyglot runs reach 100%, and four agents solve all 60 held-in tasks over twenty generations. In the ablation of Section 5.3, the variant with outcome evidence and the observation also reaches 100%. The advantage of PCE persists over twenty generations, and the evolved harness transfers to seven other execution models. The token savings come from shorter task executions. In the twenty-generation Polyglot runs, task execution saves 2,037.0M tokens, while population distillation adds 160.1M. PCE episodes are far shorter on both solved and unsolved tasks. Episodes above one million tokens fall from 611 to 94, as Appendix E shows. In the revised paper, Section 4 states the number of runs, and Tables 1 and 2 report the mean over all runs. Figures 3a and 3b plot the additional runs. We hope these results address the reviewer's concern.
Weakness 1 & Question 1: Baselines
A1: We thank the reviewer for these suggestions. We address each point separately.
1. Baselines. PCE and Self-Harness/HarnessEvolve answer different questions, so a stronger improvement step could be used within PCE rather than compete with it. PCE asks which evidence should inform the step that creates a child from its parents. We use GEA as the baseline to isolate the effect of population distillation. GEA gives diagnosis the records of the selected parents, and PCE adds population-level evidence. Any difference between the two methods therefore comes from this evidence. Self-Harness and HarnessEvolve address a lower-level question. Each refines a single active harness by accepting or rejecting candidate changes, rather than evolving a population. Neither reports a comparison with DGM or GEA. Their contribution concerns how to carry out one improvement step, which DGM, GEA, and PCE all repeat in every generation.
A stronger step of this kind could therefore be used within DGM, GEA, or PCE. Comparing it with PCE would change the improvement step and the evolution method at the same time.
Therefore, PCE and the methods that refine a single improvement step work at different levels, and a stronger improvement step could be used within PCE.