acceptodds
Under review as a conference paper at ICLR 2027

The View from Nowhere: Population-Centered Agent Evolution

Abstract

Self-improving AI systems can revise the designs that govern their own behavior. Yet existing approaches center improvement on selected parent agents and their evolutionary experience, leaving the broader population largely underused. We identify a substantial gap between population coverage (i.e., the tasks solved by at least one evolved variant) and the performance of the best individual agent. This gap shows that useful capabilities and diagnostic evidence emerge across the population without being consolidated into its leading agents. These observations motivate the view from nowhere, a principle for agent evolution: an agent should be improved by interpreting its behavior through population-wide patterns of success, failure, and prior self-modification. We instantiate this principle as Population-Centered Evolution (PCE). PCE distils task records and modification histories from the population into a population block, which captures recurring failures, complementary successes, and the effects of previously attempted modifications to guide improvements to a selected parent’s harness. Under matched models, initial harnesses, and evolution schedules, PCE achieves 72.1% success on SWE-bench Verified and 96.7% on Polyglot, improving over a strong group-evolution baseline by 3.2 and 5.9 percentage points. In our evaluation, PCE reduces the gap between population coverage and best-agent performance by one-third on SWE-bench and closes it on Polyglot. These results suggest that treating the population as evidence—not merely as candidates for selection—provides a complementary principle for agent evolution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving Agents, Open-Ended Evolution, Evolutionary Search

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_H2HVOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Population-Centered Evolution (PCE), a framework that uses population-wide experience to guide agent self-improvement. The authors identify a gap between the tasks solved collectively by an evolutionary population and those solved by its best individual agent. To address this, PCE distils execution records and modification histories across agent variants into a population block. This block is then used to diagnose and improve a selected parent's harness. Experiments on SWE-bench Verified and Polyglot show that PCE outperforms Group-Evolving Agents (GEA) by 3.2 and 5.9 percentage points. PCE also reduces the population coverage gap and uses fewer tokens during evolution.

Critical strengths

The paper highlights a useful gap between population coverage and individual agent performance. PCE extends experience sharing beyond selected parents and uses cross-agent evidence to guide harness modifications. This is a reasonable direction for improving evolutionary agent systems.

The experiments compare PCE with GEA under matched models and evolution settings. PCE improves performance on both benchmarks and retains gains on held-out tasks. The parent-only distillation ablation provides useful evidence for the value of population-wide information. The reduction in token consumption is also encouraging.

Critical weaknesses

1. Limited baseline comparisons and insufficient validation of the proposed mechanism.

The evaluation focuses primarily on GEA. Results from DGM and other methods in Table 1 use different experimental settings, while controlled comparisons with recent methods such as Self-Harness and HarnessEvolve are missing. The ablations also do not isolate the contributions of individual population-block components or compare against simpler experience aggregation strategies. More importantly, the claimed capability consolidation is supported mainly by a reduced population coverage gap. This does not directly establish whether agents acquire capabilities discovered by other variants. These limitations leave the specific advantages of PCE insufficiently demonstrated.

2. Limited experimental robustness and generalizability.

SWE-bench includes only two independent evolution runs per method, while Polyglot includes just one. The five repeated evaluations measure the variability of selected agents but do not establish the stability of the evolution process. This is particularly concerning given the reported 100% Polyglot search score. The experiments are also limited to one model family and ten generations. Furthermore, the substantial token savings coincide with shorter execution trajectories, but their source is not clearly established. These limitations raise concerns about the reliability and generalizability of the reported improvements.

Questions for authors
  1. Could the authors provide stronger baseline comparisons and more comprehensive ablations? For example, comparing PCE with Self-Harness or HarnessEvolve under matched settings would help establish its advantages over existing methods. Ablating individual population-block components and comparing against simpler strategies such as trajectory retrieval or failure clustering would also clarify which parts of PCE are necessary.

  2. Could the authors provide additional independent evolution runs on Polyglot? Since the reported 100% search score and zero coverage gap are based on a single run, it remains unclear whether these results are reproducible across different evolution trajectories.

  3. Can the authors measure how often a child solves tasks that its selected parents failed but other population members previously solved? Comparing this rate against GEA would provide more direct evidence of cross-agent capability consolidation.

  4. Could the authors break down token consumption by execution, diagnosis, self-modification, and distillation?

  5. Do the findings hold across longer evolution horizons, larger populations, or different model families? For example, extending evolution to 20–30 generations and evaluating with other models such as GPT or Kimi would help assess generalizability. Also testing on more recent challenging benchmarks would provide further evidence beyond the current evaluation settings.

12 comments
AuthorsOpenReview ↗

Weakness 1 & Question 1: Baselines

W1. Limited baseline comparisons and insufficient validation of the proposed mechanism. The evaluation focuses primarily on GEA. Results from DGM and other methods in Table 1 use different experimental settings, while controlled comparisons with recent methods such as Self-Harness and HarnessEvolve are missing. The ablations also do not isolate the contributions of individual population-block components or compare against simpler experience aggregation strategies. More importantly, the claimed capability consolidation is supported mainly by a reduced population coverage gap. This does not directly establish whether agents acquire capabilities discovered by other variants. These limitations leave the specific advantages of PCE insufficiently demonstrated.

Q1. Could the authors provide stronger baseline comparisons and more comprehensive ablations? For example, comparing PCE with Self-Harness or HarnessEvolve under matched settings would help establish its advantages over existing methods. Ablating individual population-block components and comparing against simpler strategies such as trajectory retrieval or failure clustering would also clarify which parts of PCE are necessary.

A1: We thank the reviewer for these suggestions. We address each point separately.

1. Baselines. PCE and Self-Harness/HarnessEvolve answer different questions, so a stronger improvement step could be used within PCE rather than compete with it. PCE asks which evidence should inform the step that creates a child from its parents. We use GEA as the baseline to isolate the effect of population distillation. GEA gives diagnosis the records of the selected parents, and PCE adds population-level evidence. Any difference between the two methods therefore comes from this evidence. Self-Harness and HarnessEvolve address a lower-level question. Each refines a single active harness by accepting or rejecting candidate changes, rather than evolving a population. Neither reports a comparison with DGM or GEA. Their contribution concerns how to carry out one improvement step, which DGM, GEA, and PCE all repeat in every generation.

LevelQuestion it answersMethods
Population-level evolutionhow parents are kept and chosen, and which evidence informs each improvementDGM, GEA, PCE
Single improvement stephow one harness becomes a better harness from given evidenceSelf-Harness, HarnessEvolve

A stronger step of this kind could therefore be used within DGM, GEA, or PCE. Comparing it with PCE would change the improvement step and the evolution method at the same time.

Therefore, PCE and the methods that refine a single improvement step work at different levels, and a stronger improvement step could be used within PCE.

AuthorsOpenReview ↗

Weakness 1 & Question 1: Ablations and consolidation

2. Ablations. Our main argument is that agent evolution should draw on population-level evidence. An ablation of each component in the population block takes this argument one step further, showing where the gain comes from. We run this ablation on Polyglot, keeping distillation unchanged and varying only the components that diagnosis receives. Trajectory retrieval and failure clustering usually serve to improve a single agent iteratively rather than to evolve a population. The two approaches are complementary, and each suits different scenarios. To test this kind of strategy within population evolution, we add member summaries. They summarize the record of every population member separately, so diagnosis sees the whole population but none of the population-block components.

The results show that the gain does not depend on any single component. Outcome evidence alone and the observation alone each reach 98.3%, and together they reach 100%. In contrast, variants without population-level evidence reach 90.0% to 93.3% after ten generations. The gain therefore comes from evidence about how the whole population performs on a task, and each component supplies this evidence.

Population-level evidenceVariantBy generation 2After ten generations
WithoutGEA85.090.0
Parent-only distillation80.091.7
Member summaries83.393.3
WithOutcome evidence only98.398.3
Cross-agent observation only95.098.3
Outcome evidence and observation96.7100.0
Full population block96.7100.0

Best search score so far on Polyglot Held-in 60, in percent, from Figure 4a and Table 6 of the revised paper.

  • Retrieved records. GEA gives diagnosis the records of the selected parents, and parent-only distillation adds an analysis step over the same records. Over the generations, the records that GEA reads add up to about as much evidence as member summaries provide. GEA, parent-only distillation, and member summaries therefore end close together. Neither the number of trajectories nor additional analysis is the bottleneck.
  • Outcome evidence. Outcome evidence reports the distribution of task outcomes across the population. With outcome evidence alone, the best search score reaches 98.3% by generation 2. This already exceeds the 90.0% that GEA reaches after ten generations.
  • Cross-agent observation. The cross-agent observation provides a hypothesis about why the outcomes of population members differ. With the observation alone, the best search score reaches 95.0% by generation 2 and 98.3% by generation 5.
  • Candidate files. Candidate files identify parts of the harness that may implement the relevant behavior. They guide where diagnosis looks, so we evaluate them together with the other two components. Outcome evidence and the observation together reach 100%, and so does the full population block with candidate files.

Why population distillation works. As Section 3.1 describes, each component gives diagnosis evidence about how the whole population performs on a task. With this evidence, diagnosis can locate the problem in the harness that most needs to change. Parent-only distillation and member summaries lack this evidence, and they end close to GEA. Outcome evidence and the observation therefore each achieve most of the gain on their own, and together the components reach 100%.

3. Consolidation. We especially thank the reviewer for proposing a direct measure of consolidation, which we find very useful. We adopt it in the revised paper as the consolidation rate, a new column of Table 2 defined in Appendix D.1. On Polyglot, PCE children solve 72.9% of the tasks that their parents fail but earlier population members solve, against 44.6% for GEA.

Therefore, the gain of PCE comes from population-level evidence, which every component provides, rather than from retrieved or summarized records. The consolidation rate gives the direct evidence that the reviewer asked for. In the revised paper, Figure 4a and Section 5.3 report the ablation, and Table 6 in Appendix A lists every child of each variant. Table 2 and Appendix D.1 report the consolidation rate. We hope this answer addresses the reviewer's concern.

AuthorsOpenReview ↗

Weakness 2 & Question 2: Robustness of the evolution process

W2. Limited experimental robustness and generalizability. SWE-bench includes only two independent evolution runs per method, while Polyglot includes just one. The five repeated evaluations measure the variability of selected agents but do not establish the stability of the evolution process. This is particularly concerning given the reported 100% Polyglot search score. The experiments are also limited to one model family and ten generations. Furthermore, the substantial token savings coincide with shorter execution trajectories, but their source is not clearly established. These limitations raise concerns about the reliability and generalizability of the reported improvements.

Q2. Could the authors provide additional independent evolution runs on Polyglot? Since the reported 100% search score and zero coverage gap are based on a single run, it remains unclear whether these results are reproducible across different evolution trajectories.

A2: We thank the reviewer for this suggestion. All three reviewers raised this question, so we address it in detail. Our setting follows DGM and GEA, whose main results each come from a single evolution run per benchmark. As both papers report, one such run costs DGM about USD 22,000 on SWE-bench. For GEA it costs about USD 13,000 on SWE-bench and USD 1,500 on Polyglot. Our submission reported two SWE-bench runs per method and one Polyglot run.

We agree that randomness is inherent in agent evolution. We therefore spend additional budget and compute on new runs and report them in the revised paper. The improvement of PCE over GEA is reproducible and does not rest on a single favorable run. The evidence below covers independent runs and twenty-generation runs, and it also shows that the evolved harness transfers to other execution models.

  • The best PCE agent beats the best GEA agent in every run. The table below lists the best search score, coverage gap, and consolidation rate of each run, together with the means from Table 2.
BenchmarkRunBest search score GEABest search score PCECoverage gap GEACoverage gap PCEConsolidation rate GEAConsolidation rate PCE
SWE-benchRun 171.775.015.010.025.541.9
Run 268.375.013.310.040.629.2
Mean ± sd70.0 ± 2.475.0 ± 0.014.2 ± 1.210.0 ± 0.033.1 ± 10.735.6 ± 9.0
PolyglotRun 190.0100.010.00.037.059.1
Run 285.098.315.01.745.581.0
Run 381.7100.018.30.051.278.7
Mean ± sd85.6 ± 4.299.4 ± 1.014.4 ± 4.20.6 ± 1.044.6 ± 7.172.9 ± 12.0
  • The gain exceeds the variation between runs. On Polyglot, the lowest PCE run reaches 98.3% and the highest GEA run reaches 90.0%. On SWE-bench, both PCE runs reach 75.0% and the GEA runs reach 71.7% and 68.3%. The weakest PCE run therefore beats the strongest GEA run on both benchmarks, as Tables 5 and 6 in Appendix A show.
  • Strong agents recur over twenty generations. In the twenty-generation run of Section 5.5 and Figure 4b, PCE produces four agents that solve all 60 held-in tasks. They appear at generations 2, 4, 4, and 11. The best GEA agent over the same twenty generations reaches 91.7%.
  • The evolved harness transfers to other execution models. The PCE agent runs unchanged under seven other execution models from five providers. It outperforms the initial agent under every model, by 25.7 points on average, as Section 5.5 and Figure 4c report.
  • PCE also leads in repeated and held-out evaluation. Over five repeated evaluations on Polyglot, the selected PCE agent averages 96.0%, against 92.0% for the selected GEA agent. The selected PCE agents also beat GEA on every held-out task set, including 97.0% against 90.3% on Polyglot Held-out 165, as Table 8 reports.

Therefore, the advantage of PCE is reproducible across independent runs, since its best agent outperforms that of GEA in every independent run. The perfect search score holds across runs. Two of the three Polyglot runs reach 100%, and four agents solve all 60 held-in tasks over twenty generations. In the ablation of Section 5.3, the variant with outcome evidence and the observation also reaches 100%. The advantage of PCE persists over twenty generations, and the evolved harness transfers to seven other execution models. The token savings come from shorter task executions. In the twenty-generation Polyglot runs, task execution saves 2,037.0M tokens, while population distillation adds 160.1M. PCE episodes are far shorter on both solved and unsolved tasks. Episodes above one million tokens fall from 611 to 94, as Appendix E shows. In the revised paper, Section 4 states the number of runs, and Tables 1 and 2 report the mean over all runs. Figures 3a and 3b plot the additional runs. We hope these results address the reviewer's concern.

Reviewer_QgZgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Population-Centered Evolution (PCE), which uses experience from different evolved agent versions to guide further improvement. A distillation agent compares task outcomes, execution traces, and previous modifications, then summarizes useful findings for the next update. Experiments on SWE-bench Verified and Polyglot show improvements over GEA under the same models and evolution settings, including on held-out tasks. The authors also report lower total token usage and compare population distillation with analysis limited to the selected parents.

Critical strengths
  1. The paper is overall easy to follow, and using experience from different agent versions to guide improvement appears reasonable.
  2. The comparison with GEA uses matched evolution settings, and the parent-only ablation supports using experience beyond the selected parents. The authors also evaluate held-out tasks and include distillation in the reported token costs.
Critical weaknesses
  1. Combining results from many agent versions may cover more tasks simply because there are more attempts. Even an unchanged agent can solve different tasks across runs, so the gap between population coverage and the best agent's score does not necessarily show complementary capabilities.

  2. The experiments include only two independent evolution runs per method on SWE-bench and one on Polyglot, while the main performance and cost comparisons use one run. Evaluating the selected agent five times measures variation in its execution, but does not show whether the evolution algorithm consistently finds better agents.

  3. It is unclear whether distilling population experience works better than directly retrieving or sampling historical records. The parent-only ablation compares sources of experience rather than the need for distillation, and the raw-trajectory comparison only examines proposals from three contexts.

  4. Model coverage and baseline comparability are limited. PCE is evaluated only within the DeepSeek family. Table 1 includes DGM and other methods using older or different models and evaluation protocols, so those results do not establish PCE’s advantage over these approaches. SWE-bench and Polyglot also use different execution models—DeepSeek-V4-Flash and V4.1-Flash, making it harder to separate benchmark effects from model effects. It also remains unclear whether the gains persist with different executor or diagnosis models, or whether the evolved harnesses transfer across model families.

Questions for authors
  1. Population-wide analysis seems likely to cost more than parent-only analysis. Could the authors break down the cost of population distillation and diagnosis relative to the parent-only baseline? How does this overhead grow with population size, and how much of the overall saving comes from shorter downstream executions?

  2. Please see the Weaknesses section.

6 comments
AuthorsOpenReview ↗

Weakness 1: Population coverage versus repeated attempts

W1. Combining results from many agent versions may cover more tasks simply because there are more attempts. Even an unchanged agent can solve different tasks across runs, so the gap between population coverage and the best agent's score does not necessarily show complementary capabilities.

A1: We thank the reviewer for this point, and we would like to clarify how repeated attempts relate to the gains of PCE. The number of tasks solved across repeated attempts does grow with the number of attempts, and test-time methods build on this effect. However, PCE works at a different level, improving the agent itself, and our comparisons hold the number of attempts fixed.

  • 1. Complementary to test-time methods. Test-time methods gain from repeated attempts, for example through best-of-N sampling, majority voting, or reranking with a verifier. These methods act at inference time, while PCE changes the harness of the agent. PCE and these methods are therefore complementary, and these methods can also be applied to an agent that PCE evolves.
  • 2. Repeated attempts of one agent. In Table 3, we evaluate the same selected agent five times, which matches the setting the reviewer describes. The PCE agent averages 96.0% over the five evaluations, against 92.0% for GEA. It also solves 88.3% of the tasks in every evaluation, against 85.0% for GEA. The PCE agent therefore solves more tasks reliably than baselines, and its advantage does not depend on combining attempts.
  • 3. Attempts by different agents in the population. In each evolution run, population coverage counts the tasks that any child solves as Table 2 reports. The two methods reach similar population coverage on both benchmarks. Yet the coverage gap of GEA is 14.2 points against 10.0 for PCE on SWE-bench, and 14.4 against 0.6 on Polyglot. The two populations find nearly the same successes, but the best PCE agent solves more of them on its own.
  • 4. A direct measure. Motivated by the reviews, we add the consolidation rate. It measures how often a child solves tasks that its parents fail but earlier population members solve. PCE achieves a higher rate than GEA on both benchmarks, as Table 2 reports. On Polyglot, PCE reaches 72.9% against 44.6% for GEA, and the lowest PCE run exceeds the highest GEA run.

Therefore, repeated attempts do not explain the advantage of PCE. The selected PCE agent solves more tasks reliably. We hope this answer addresses the reviewer's concern.

AuthorsOpenReview ↗

Weakness 2: Number of independent evolution runs

W2. The experiments include only two independent evolution runs per method on SWE-bench and one on Polyglot, while the main performance and cost comparisons use one run. Evaluating the selected agent five times measures variation in its execution, but does not show whether the evolution algorithm consistently finds better agents.

A2: We thank the reviewer for this point. All three reviewers raised this question, so we address it in detail. Our setting follows DGM and GEA, whose main results each come from a single evolution run per benchmark. As both papers report, one such run costs DGM about USD 22,000 on SWE-bench. For GEA it costs about USD 13,000 on SWE-bench and USD 1,500 on Polyglot. Our submission reported two SWE-bench runs per method and one Polyglot run.

We agree that randomness is inherent in agent evolution. We therefore spend additional budget and compute on new runs and report them in the revised paper. The improvement of PCE over GEA is reproducible and does not rest on a single favorable run. The evidence below covers independent runs and twenty-generation runs.

  • The best PCE agent beats the best GEA agent in every run. The table below lists the best search score, coverage gap, and consolidation rate of each run, together with the means from Table 2.
BenchmarkRunBest search score GEABest search score PCECoverage gap GEACoverage gap PCEConsolidation rate GEAConsolidation rate PCE
SWE-benchRun 171.775.015.010.025.541.9
Run 268.375.013.310.040.629.2
Mean ± sd70.0 ± 2.475.0 ± 0.014.2 ± 1.210.0 ± 0.033.1 ± 10.735.6 ± 9.0
PolyglotRun 190.0100.010.00.037.059.1
Run 285.098.315.01.745.581.0
Run 381.7100.018.30.051.278.7
Mean ± sd85.6 ± 4.299.4 ± 1.014.4 ± 4.20.6 ± 1.044.6 ± 7.172.9 ± 12.0
  • The gain exceeds the variation between runs. On Polyglot, the lowest PCE run reaches 98.3% and the highest GEA run reaches 90.0%. On SWE-bench, both PCE runs reach 75.0% and the GEA runs reach 71.7% and 68.3%. The weakest PCE run therefore beats the strongest GEA run on both benchmarks, as Tables 5 and 6 in Appendix A show.
  • Strong agents recur over twenty generations. In the twenty-generation run of Section 5.5 and Figure 4b, PCE produces four agents that solve all 60 held-in tasks. They appear at generations 2, 4, 4, and 11. The best GEA agent over the same twenty generations reaches 91.7%.
  • The cost advantage holds beyond the main run. In the twenty-generation Polyglot runs, PCE uses 698.1M tokens against 2,786.0M for GEA, about a quarter, as Appendix E reports.

Therefore, PCE consistently finds better agents than GEA, since its best agent outperforms that of GEA in every independent run. The advantage also persists over twenty generations. In the revised paper, Section 4 states the number of runs, and Tables 1 and 2 report the mean over all runs. Figures 3a and 3b plot the additional runs. We hope these results address the reviewer's concern.

AuthorsOpenReview ↗

Weakness 3: Distillation versus direct retrieval

W3. It is unclear whether distilling population experience works better than directly retrieving or sampling historical records. The parent-only ablation compares sources of experience rather than the need for distillation, and the raw-trajectory comparison only examines proposals from three contexts.

A3: We thank the reviewer for this point. Our main argument is that agent evolution should draw on population-level evidence. An ablation of each component in the population block takes this argument one step further, showing where the gain comes from. We run this ablation on Polyglot, keeping distillation unchanged and varying only the components that diagnosis receives. To test direct retrieval within population evolution, we add member summaries. They retrieve and summarize the record of every population member separately, so diagnosis sees the whole population but none of the population-block components. Unlike the comparison in Appendix B, which examines proposals from three fixed contexts, every variant here runs a full ten-generation evolution.

The results show that distilled population-level evidence outperforms retrieved records, even when the records cover every population member. Member summaries reach 93.3% after ten generations. Outcome evidence alone and the observation alone each reach 98.3%, and together they reach 100%.

Population-level evidenceVariantBy generation 2After ten generations
WithoutGEA85.090.0
Parent-only distillation80.091.7
Member summaries83.393.3
WithOutcome evidence only98.398.3
Cross-agent observation only95.098.3
Outcome evidence and observation96.7100.0
Full population block96.7100.0
  • Retrieved records. GEA gives diagnosis the records of the selected parents, and parent-only distillation adds an analysis step over the same records. Over the generations, the records that GEA reads add up to about as much evidence as member summaries provide. GEA, parent-only distillation, and member summaries therefore end close together. Neither the number of trajectories nor additional analysis is the bottleneck.
  • Outcome evidence. Outcome evidence reports the distribution of task outcomes across the population. With outcome evidence alone, the best search score reaches 98.3% by generation 2. This already exceeds the 93.3% that member summaries reach after ten generations.
  • Cross-agent observation. The cross-agent observation provides a hypothesis about why the outcomes of population members differ. With the observation alone, the best search score reaches 95.0% by generation 2 and 98.3% by generation 5.
  • Candidate files. Candidate files identify parts of the harness that may implement the relevant behavior. They guide where diagnosis looks, so we evaluate them together with the other two components. Outcome evidence and the observation together reach 100%, and so does the full population block with candidate files.

Why population distillation works. As Section 3.1 describes, each component gives diagnosis evidence about how the whole population performs on a task. With this evidence, diagnosis can locate the problem in the harness that most needs to change. Parent-only distillation and member summaries lack this evidence, and they end close to GEA. Outcome evidence and the observation therefore each achieve most of the gain on their own, and together the components reach 100%.

Therefore, the gain of PCE comes from distilling the records into population-level evidence, which every component provides, rather than from retrieving raw or summarized records. In the revised paper, Figure 4a and Section 5.3 report the ablation, and Table 6 in Appendix A lists every child of each variant. We hope this answer addresses the reviewer's question.

Reviewer_Wpc7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Population-Centered Evolution (PCE), which uses execution records and modification histories from the entire agent population to guide self-improvement. The method summarizes population-wide successes and failures into a population block and uses it to diagnose and modify a selected parent's harness. Experiments on SWE-bench Verified and Polyglot show improvements over GEA in both performance and token efficiency.

Critical strengths
  1. The paper identifies an interesting limitation of existing agent evolution methods. The gap between population coverage and best-agent performance provides a useful motivation for learning from the broader population rather than only selected parents.

  2. The experimental results are promising. PCE improves over GEA on both benchmarks, and the gains also extend to held-out tasks. The reduction in total token consumption is another encouraging result.

  3. The parent-only distillation ablation helps isolate the benefit of population-wide information from simply adding another analysis step. The case study also gives a concrete example of how population-level observations can lead to better harness modifications.

Critical weaknesses
  1. The experiments are relatively limited. SWE-bench uses only two evolution runs per method, while Polyglot reports a single run. Given the randomness involved in agent evolution, more independent runs would help establish the reliability of the improvements.

  2. The method combines outcome statistics, cross-agent observations, and candidate file suggestions into a single population block. It is unclear which components contribute most to the performance gains. A more detailed ablation would help explain why population distillation works.

Questions for authors
  1. Can the authors provide an ablation of the different components in the population block, particularly the outcome statistics, cross-agent observations, and candidate file suggestions?

  2. How sensitive are the results to the number and diversity of agents in the population? Does population distillation still help when only a small number of variants have attempted the same task?

3 comments
AuthorsOpenReview ↗

Weakness 1: Number of independent evolution runs

W1. The experiments are relatively limited. SWE-bench uses only two evolution runs per method, while Polyglot reports a single run. Given the randomness involved in agent evolution, more independent runs would help establish the reliability of the improvements.

A1: We thank the reviewer for this suggestion. All three reviewers raised this question, so we address it in detail. Our setting follows DGM and GEA, whose main results each come from a single evolution run per benchmark. As both papers report, one such run costs DGM about USD 22,000 on SWE-bench. For GEA it costs about USD 13,000 on SWE-bench and USD 1,500 on Polyglot. Our submission reported two SWE-bench runs per method and one Polyglot run.

We agree that randomness is inherent in agent evolution. We therefore spend additional budget and compute on new runs and report them in the revised paper. The improvement of PCE over GEA is reproducible and does not rest on a single favorable run. The evidence below covers independent runs and twenty-generation runs, and it also shows that the evolved harness transfers to other execution models.

  • The best PCE agent beats the best GEA agent in every run. The table below lists the best search score, coverage gap, and consolidation rate of each run, together with the means from Table 2.
BenchmarkRunBest search score GEABest search score PCECoverage gap GEACoverage gap PCEConsolidation rate GEAConsolidation rate PCE
SWE-benchRun 171.775.015.010.025.541.9
Run 268.375.013.310.040.629.2
Mean ± sd70.0 ± 2.475.0 ± 0.014.2 ± 1.210.0 ± 0.033.1 ± 10.735.6 ± 9.0
PolyglotRun 190.0100.010.00.037.059.1
Run 285.098.315.01.745.581.0
Run 381.7100.018.30.051.278.7
Mean ± sd85.6 ± 4.299.4 ± 1.014.4 ± 4.20.6 ± 1.044.6 ± 7.172.9 ± 12.0
  • The gain exceeds the variation between runs. On Polyglot, the lowest PCE run reaches 98.3% and the highest GEA run reaches 90.0%. On SWE-bench, both PCE runs reach 75.0% and the GEA runs reach 71.7% and 68.3%. The weakest PCE run therefore beats the strongest GEA run on both benchmarks, as Tables 5 and 6 in Appendix A show.
  • Strong agents recur over twenty generations. In the twenty-generation run of Section 5.5 and Figure 4b, PCE produces four agents that solve all 60 held-in tasks. They appear at generations 2, 4, 4, and 11. The best GEA agent over the same twenty generations reaches 91.7%.
  • The evolved harness transfers to other execution models. The PCE agent runs unchanged under seven other execution models from five providers. It outperforms the initial agent under every model, by 25.7 points on average, as Section 5.5 and Figure 4c report.
  • PCE also leads in repeated and held-out evaluation. Over five repeated evaluations on Polyglot, the selected PCE agent averages 96.0%, against 92.0% for the selected GEA agent. The selected PCE agents also beat GEA on every held-out task set, including 97.0% against 90.3% on Polyglot Held-out 165, as Table 8 reports.

Therefore, the improvement of PCE over GEA holds across independent runs and over twenty generations. The harness that PCE evolves also transfers to other execution models. As the ablation in Section 5.3 shows, this gain comes from population-level evidence rather than from more records or an additional analysis step. In the revised paper, Section 4 states the number of runs, and Tables 1 and 2 report the mean over all runs. Figures 3a and 3b plot the additional runs. We hope these results address the reviewer's concern.

AuthorsOpenReview ↗

Weakness 2 & Question 1: Ablation of population-block components

W2 & Q1. The method combines outcome statistics, cross-agent observations, and candidate file suggestions into a single population block. It is unclear which components contribute most to the performance gains. A more detailed ablation would help explain why population distillation works. Can the authors provide an ablation of the different components in the population block, particularly the outcome statistics, cross-agent observations, and candidate file suggestions?

A2: We thank the reviewer for this suggestion. Our main argument is that agent evolution should draw on population-level evidence. An ablation of each component in the population block takes this argument one step further, showing where the gain comes from. We run this ablation on Polyglot, keeping distillation unchanged and varying only the components that diagnosis receives. As controls, parent-only distillation analyzes only the records of the selected parents. Member summaries summarize the record of every population member separately, so diagnosis sees the whole population but none of the three components. The results show that the gain does not depend on any single component. Outcome evidence alone and the observation alone each reach 98.3%, and together they reach 100%. In contrast, variants without population-level evidence reach 90.0% to 93.3% after ten generations. The gain therefore comes from evidence about how the whole population performs on a task, and each component supplies this evidence.

Population-level evidenceVariantBy generation 2After ten generations
WithoutGEA85.090.0
Parent-only distillation80.091.7
Member summaries83.393.3
WithOutcome evidence only98.398.3
Cross-agent observation only95.098.3
Outcome evidence and observation96.7100.0
Full population block96.7100.0

Best search score so far on Polyglot Held-in 60, in percent, from Figure 4a and Table 6 of the revised paper.

  • 1. Outcome statistics. Outcome evidence reports the distribution of task outcomes across the population. With outcome evidence alone, the best search score reaches 98.3% by generation 2. This already exceeds the 90.0% that GEA reaches after ten generations.
  • 2. Cross-agent observations. The cross-agent observation provides a hypothesis about why the outcomes of population members differ. With the observation alone, the best search score reaches 95.0% by generation 2 and 98.3% by generation 5.
  • 3. Candidate file suggestions. Candidate files identify parts of the harness that may implement the relevant behavior. They guide where diagnosis looks, so we evaluate them together with the other two components. Outcome evidence and the observation together reach 100%, and so does the full population block with candidate files.

Why population distillation works. As Section 3.1 describes, each component gives diagnosis evidence about how the whole population performs on a task. With this evidence, diagnosis can locate the problem in the harness that most needs to change. Parent-only distillation and member summaries lack this evidence, and they end close to GEA. Outcome evidence and the observation therefore each achieve most of the gain on their own, and together the components reach 100%.

Therefore, the gain of PCE comes from population-level evidence, which every component provides, rather than from one component in particular. In the revised paper, Figure 4a and Section 5.3 report the ablation, and Table 6 in Appendix A lists every child of each variant. We hope this answer addresses the reviewer's question.

AuthorsOpenReview ↗

Question 2: Sensitivity to population size and diversity

Q2. How sensitive are the results to the number and diversity of agents in the population? Does population distillation still help when only a small number of variants have attempted the same task?

A3: We thank the reviewer for this question. PCE already shows its advantage when the population is small, and the advantage persists as the population grows.

  • 1. Small populations. Distillation runs once a task has at least three records, as Section 3.3 states. By generation 2, each task has four or five records. At that point, every variant with population-level evidence scores between 95.0% and 98.3%. GEA, parent-only distillation, and member summaries score between 80.0% and 85.0%, as Figure 4a shows.
  • 2. Large populations. In the twenty-generation run of Section 5.5, the population grows to 41 members. PCE still produces an agent that solves all 60 held-in tasks at generation 11, when the population has 23 members. Distillation tokens per child stay flat, as Appendix E reports.
  • 3. Diversity. Population diversity depends on many factors and lies outside the scope of this paper. Our runs nevertheless show how PCE behaves when most of the population descends from one agent. In Runs 2 and 3, one generation-1 agent is a parent of almost every later child. PCE still evolves its best agents from this parent, four of them in Run 2 and three in Run 3, as Appendix D.2 shows. Within ten generations, the largest gain of a PCE child over its inheriting parent is 17, 19, and 18 tasks in Runs 1 to 3. For GEA it is 6, 6, and 8 tasks. These PCE parents score below the population median at the time. A single modification that solves 17 to 19 more tasks captures a general capability, rather than a rare skill of one member. Population distillation helps diagnosis find such a capability, because it shows which failures recur across the whole population.

Therefore, PCE helps with small populations and with populations that descend mostly from one agent. Population-level evidence lets diagnosis improve ordinary parents in large steps. In the revised paper, Figure 4a and Section 5.3 report the early generations, and Section 5.5 reports the twenty-generation runs. Appendix D.2 analyzes the lineages and the parents of the largest gains, and Appendix E reports the cost. We hope this answer addresses the reviewer's question.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for their careful reading and thoughtful questions. Their feedback centers on the reproducibility of the gains, the source of these gains, and a finer measure of consolidation. Working through these questions has substantially strengthened PCE. We are genuinely excited about what this discussion has brought to PCE.

Reviewers Wpc7 and H2HV find the gap between population coverage and the best agent a useful motivation. The weaknesses listed in the reviews concern the experimental evidence, and the reviewers consider them minor or fixable. To address them, we add new experiments and analyses to the revised paper. We summarize the main ones below.

  • A direct measure of consolidation. We are especially grateful to Reviewer H2HV for proposing a new metric. It measures how often a child solves tasks that its parents fail but earlier population members solve. The coverage gap reflects the outcome of evolution for the whole population. The consolidation rate works at the level of each child and shows whether the child acquires earlier successes of the population. PCE achieves a higher rate than GEA on both benchmarks. On Polyglot, a PCE child solves about three in four such tasks, while a GEA child solves fewer than half.
  • Independent runs. All three reviewers asked for more independent evolution runs. We now report three runs per method on Polyglot and two on SWE-bench. The best PCE agent outperforms the best GEA agent in every run, and the weakest PCE run still exceeds the strongest GEA run on both benchmarks.
  • Component ablation. Reviewers Wpc7, QgZg, and H2HV asked which parts of the population block matter. We ablate each component on Polyglot. As a control, member summaries condense the record of every population member for diagnosis. With outcome evidence alone or the cross-agent observation alone, PCE reaches 98.3%, and with both it reaches 100%. Variants without population-level evidence score between 90.0% and 93.3% after ten generations. The gain therefore comes from population-level evidence rather than from more records or an additional analysis step.
  • Longer evolution and other models. Over twenty generations, the population grows to 41 members and PCE keeps its advantage. The harness that PCE evolves with DeepSeek Flash also remains effective with seven other execution models from five model families.
  • Cost. A new token analysis in Appendix E shows that population distillation costs 7.7% of the tokens that PCE saves. Almost all of the savings come from shorter task executions.

The revised paper marks all new content in blue. In each reviewer thread, we answer every weakness and question point by point. We believe these results address the weaknesses raised in the reviews. If any point remains open, we would be glad to discuss it further. We would be grateful if the reviewers could take the new results into account in their final assessment. We thank the reviewers again for their time and thoughtful engagement, which has shaped the revised paper.

AuthorsOpenReview ↗

Follow-up on our responses

As the discussion period ends soon, we would appreciate hearing whether our replies address the concerns of reviewers. The general response above summarizes the new experiments, and our replies in each reviewer thread address the specific concerns in detail. We would be happy to clarify any remaining questions before the deadline.

AuthorsOpenReview ↗

Update from the discussion with Reviewer H2HV

Our follow-up discussion with Reviewer H2HV adds two results that also relate to the questions of Reviewers Wpc7 and QgZg.

  • What each field of the population block adds. Beyond the best agent, we compare the search scores of all children of each ablation variant. The cross-agent observation raises the consolidation rate from 28.4% with outcome evidence alone to 58.5%. With both fields, every child from generation 2 on scores between 96.7% and 100%.
  • Full evolution with another model family. We have started full evolution of both methods with Claude Haiku 5.5 on Polyglot, and we share the progress of these runs.

Our responses to Reviewer H2HV give the details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.