acceptodds
Under review as a conference paper at ICLR 2027

Evaluation-Free Recursive Self-Improvement for Coding Agents

Abstract

Recent self-improving coding agents have achieved strong benchmark performance by iteratively modifying their own executable implementations. A common pattern uses downstream task-derived evidence to decide which candidates or lineages continue evolving, which we term evaluate-and-select. Obtaining this signal requires repeatedly executing evolved agents on downstream tasks, so broader exploration incurs increasing evaluation cost, while repeated optimization against a fixed evaluation distribution may encourage benchmark-specific adaptation. We ask a more fundamental question: can coding agents self-improve without downstream task evaluation inside the evolution loop? We introduce Explore-then-Integrate (ELITE), an explore-then-integrate framework that integrates implementation mechanisms rather than selecting a single exploratory child to continue the lineage. At each generation, ELITE proposes diverse focused improvements, implements them independently from the same parent, and inspects the resulting programs to identify complementary, redundant, and conflicting mechanisms before reimplementing a compatible subset as one coherent descendant. No downstream benchmark task, task-solving trajectory, or candidate performance score is used to guide evolution; only deterministic checks of structural and operational validity are used inside the loop. On SWE-bench Pro, ELITE improves the initial agent from 60.0% to 85.6%, compared with 86.7% for the Darwin Gödel Machine (DGM) and Huxley-Gödel Machine (HGM), while using zero candidate task evaluations and reducing observed wall-clock evolution time by 8.8x and 3.2x, respectively, under the evaluated configurations. Without further adaptation, ELITE averages 73.3% on SWE-bench Verified and 69.3% on Polyglot. Together, these results demonstrate the feasibility of coding-agent self-improvement without downstream candidate task evaluation in our setting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Self-Improving Agents, LLM Agents, Agent Evolution, Automated Agent Design, Coding Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8KiqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DGM/HyperAgents, but without doing any evals. Instead it just asks the agents to improve the agent without ever testing it and giving that feedback to the meta-agent.

Critical strengths

This is an interesting, novel paper. The core idea is simple, yet rather radical. It suggests we can make progress on ADAS/DGM without doing any evals. That's a new take, and the results are surprising. I find the main results and the transfer results quite interesting and shocking.

Critical weaknesses
  1. The main weakness, as the authors acknowledge, is a lack of replication. There is only one (!) run of the baselines. Worse, they are high-variance methods, so we don't know if the results are signal or noise. I recommend the authors add replicates and/or harvest results from those methods from their original papers.

  2. Relatedly, there are zero statistics in the paper. We thus do not know what is signal and what is noise. Even with a single replicate, there are some cases where you may be able to use some fancy stats to try to help us differentiate signal from noise, but you'd have to know enough to use those complex stats appropriately.

1 and 2 are a shame, because the paper is interesting.

  1. There is not enough discussion of WHY this might work. It's rather radical to believe we can improve our performance on a task without testing on that task? So, why does this work? Does it only work up to a point? Surely we can't expect it to keep outperforming empirical tests for long, right?

  2. How do we know that the models you are using did not read the DGM/ADAS/HGM/HyperAgents work and their code, and that's why it is able to do this remarkable thing?

  3. I may have missed it, but as far as I saw, the critical gates are not described. What are these? Scripts? libraries? That you made or others did? Or LLMs? More minor things:

  4. Line 48-49 this language seems more to describe generational population methods (e.g. classic evolutionary algorithms), but DGM and similar use archives.

  5. Lines 82-84 and elsewhere. HyperAgents can see the whole archive of past solutions and can (and did) crossover/combine subcomponents of different solutions when making a new solution, so this writing is inaccurate. DGM also looked at the whole history of its lineage and all the patches that led to it. So both can study mechanisms/features and choose to combine, fix, repair, etc. irrespective of a reward score if they choose. I thus don't think it's accurate so say your method is the first to allow crossover/picking specific ideas/features and merging them/etc. Please correct this language.

  6. Line 71-74: these descriptions are too dense to understand what these works did and how it is different from yours.

I like this paper and want to see it published eventually, but I cannot vote for acceptance without stats and proper rigorous science to differentiate signal from noise. If you can address most or all of my key weaknesses, I would be delighted to raise my score.

Questions for authors

See my questions in the Weaknesses section above.

Also, by only having one solution chosen as the parent of each new crop, you abandon the principles of open-endedness that are important for solving very hard problems. Why do that? Did you consider or try a more archive or population-based approach? It is not incompatible with your idea. At a minimum, I recommend pointing that out as a possible future work direction.

What do you tell the model at the start? What is it's prompt?

In figures like Fig 2, why did you not also compare to HyperAgents?

For single exploration, did you give it the same budget in terms of generated solutions?

22 comments
AuthorsOpenReview ↗

Response to Reviewer 8Kiq (1/11)

Summary of our response

We thank the reviewer for the thoughtful comments on statistical reliability, the mechanism of evaluation-free improvement, and implementation clarity. Our revisions address four areas. First, Sections 5.1-5.2 and the main Table 1 now reflect three independent DeepSeek runs per method; Appendix A.1.2 (Table 7) reports the added baseline runs and sample standard deviations. Second, Sections 5.3-5.4 (Tables 3-4) add the mini-swe-agent trajectory and two ablations, while Appendix A.7 connects these results to implementation-level reasoning, evolution dynamics, and evaluation scope. Third, Appendix A.5.2 (Table 12) details the deterministic gates and ablation settings; Appendix A.3 documents the prompt templates reproduced below. Finally, Sections 1-2 clarify archive-based prior work and ELITE's distinction, and the Limitations paragraph and Appendix A.7 identify archive-based search as a future direction. Below, we also clarify the role of existing model knowledge and distinguish matched exploration breadth from matched total computation.

AuthorsOpenReview ↗

Response to Reviewer 8Kiq (2/11)

W1: The main weakness, as the authors acknowledge, is a lack of replication. There is only one (!) run of the baselines. Worse, they are high-variance methods, so we don't know if the results are signal or noise. I recommend the authors add replicates and/or harvest results from those methods from their original papers.

W2: Relatedly, there are zero statistics in the paper. We thus do not know what is signal and what is noise. Even with a single replicate, there are some cases where you may be able to use some fancy stats to try to help us differentiate signal from noise, but you'd have to know enough to use those complex stats appropriately.

Response to W1 and W2 about Independent Replications and Statistical Reliability

We thank the reviewer for emphasizing the importance of independent repeated runs and statistical analysis in assessing the reliability of our results.

The original submission reported three independent ELITE runs per backbone, but only one run per baseline. We have added two complete independent evolution runs each for DGM and HGM under DeepSeek, giving all three methods three runs in this configuration. Each complete evolution run is the unit of replication. The table below reproduces Table 7 from Appendix A.1.2; Run 1 contains the original baseline results, and Runs 2 and 3 are the additions. We report means and sample standard deviations, with Avg. equally weighting the three benchmarks. ELITE's individual runs for both backbones appear in Table 6 of the revised manuscript.

Table 7: DeepSeek baseline repetitions and comparison with ELITE. Success rates are percentages; sample SDs are percentage points. Run 1 reports the original baseline results.

Method and runProVerifiedPolyglotAvg.
DGM Run 186.764.062.070.9
DGM Run 285.064.062.070.3
DGM Run 388.362.064.071.4
DGM Mean ±\pm SD86.7±1.786.7\pm1.763.3±1.263.3\pm1.262.7±1.262.7\pm1.270.9±0.670.9\pm0.6
HGM Run 186.768.064.072.9
HGM Run 288.362.064.071.4
HGM Run 390.066.066.074.0
HGM Mean ±\pm SD88.3±1.788.3\pm1.765.3±3.165.3\pm3.164.7±1.264.7\pm1.272.8±1.372.8\pm1.3
ELITE Mean ±\pm SD85.6±1.085.6\pm1.073.3±1.273.3\pm1.269.3±2.369.3\pm2.376.1±0.876.1\pm0.8

ELITE's mean success rates on Verified and Polyglot are 73.3% and 69.3%, higher than DGM's 63.3% and 62.7% and HGM's 65.3% and 64.7%. Its three-benchmark average is 76.1±0.876.1\pm0.8, exceeding DGM and HGM by 5.2 and 3.3 percentage points, respectively. DGM and HGM retain higher mean Pro scores. The added runs therefore quantify run-to-run variability and preserve the observed advantage on the evaluated transfer benchmarks and their three-benchmark average.

We have also updated the main Table 1 and Section 5.2 to use the three-run DeepSeek means for DGM and HGM. Section 5.1 specifies the repetition counts, and Appendix A.1.2 provides the per-run results and sample SDs. Qwen baseline results remain single-run observations. These statistics characterize variation across the available independent runs; the Limitations paragraph and Appendix A.7 describe the scope of the comparisons and the value of further repetitions and broader task coverage.

AuthorsOpenReview ↗

Response to Reviewer 8Kiq (3/11)

W3: There is not enough discussion of WHY this might work. It's rather radical to believe we can improve our performance on a task without testing on that task? So, why does this work? Does it only work up to a point? Surely we can't expect it to keep outperforming empirical tests for long, right?

Response to W3 about Why Evaluation-Free Evolution Can Work and Its Limits

We thank the reviewer for raising this fundamental question. We have expanded the explanation of the mechanism and its limits, supported by the additional ablations, the mini-swe-agent experiment, and the existing implementation analyses.

(1) Why can improvement occur without candidate task feedback?

Downstream task scores are one source of information for improving coding agents; another is the combination of the LLM's existing programming knowledge and inspectable source code. The LLM parameters remain fixed, while the scaffold governs tool invocation, execution feedback, and the problem-solving process. Concrete limitations, such as missing exit-status feedback, blocking output reads, or incomplete parsing, can be identified from the implementation without first evaluating an evolved candidate on downstream tasks. Improving these shared mechanisms can benefit task execution across different instances.

ELITE's Explore-then-Integrate process turns proposals into inspectable implementations. Independent exploration exposes code changes, interfaces, and dependencies; integration uses this evidence to combine complementary mechanisms and reconcile conflicts; deterministic checks filter some structural and operational errors. These steps provide a route from existing programming knowledge to scaffold improvement, although operational validity alone does not guarantee downstream gains.

(2) What evidence supports this explanation?

The revised Table 4 adds Evaluation-based Select-one and Direct joint implementation and groups the comparisons by child selection, exploration and integration, and other components. We reproduce the complete table below.

Table 4: Design ablations of ELITE. All methods use DeepSeek. Scores are success rates (%); Avg. equally weights the three benchmarks. Full ELITE and the no-gates ablation report three-run means; all other variants use one run. Bold numbers indicate column maxima.

MethodProVerifiedPolyglotAvg.
Full ELITE (K=7K=7)85.673.369.376.1
Alternative selection strategies
Select-one (LLM-based)83.368.064.071.8
SIFT-style Select-one85.072.046.067.7
Evaluation-based Select-one91.770.062.074.6
Exploration and integration ablations
Single exploration (K=1K=1)81.764.064.069.9
Direct joint implementation78.372.066.072.1
Other component ablations
w/o hypothesis generation85.068.060.071.0
w/o deterministic gates59.451.345.352.0

The results provide three complementary observations. First, full ELITE averages 76.1%, compared with 71.8% for Select-one, 67.7% for SIFT-style Select-one, and 72.1% for Direct joint implementation. The selection variants retain seven independently implemented children, while Direct joint implementation generates seven proposals and implements them jointly as one child. These comparisons support the roles of executable exploration and integration relative to the evaluated alternatives. Second, removing the deterministic gates reduces the average to 52.0%, showing the importance of operational validity constraints in this setting.

Third, Evaluation-based Select-one evaluates each of seven children on the same 30 Pro tasks per round and selects the highest-scoring child without integration. It achieves higher Pro performance (91.7% versus 85.6%), while ELITE has higher Verified (73.3% versus 70.0%) and Polyglot (69.3% versus 62.0%) scores. This comparison changes both the selection signal and the next-agent construction rule, so it compares the two strategies rather than isolating task feedback alone. The repetition counts are stated in the table caption.

Section 5.3 and Appendix A.1.4 provide a concrete implementation example: two exploratory children separately introduce continuous output reading and exit-status feedback. Their integrated descendant returns all 100,000 output lines and the correct exit code of 7 in a local probe. This demonstrates compatible functionality in the resulting code; it does not attribute aggregate benchmark gains to either modification individually.

Reviewer_AgpfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ELITE, a framework for recursively improving coding agents without using downstream task evaluations during evolution. Instead of evaluating and selecting the best agent, it generates several modifications independently and then integrates compatible changes into a single agent. The results show that this can achieve performance close to DGM and HGM on SWE-bench Pro, at a significantly lower cost.

Critical strengths
  1. The problem itself is interesting, especially given how expensive the evaluation stage is in existing self-improving agents.
  2. The method achieving comparable performance without task evaluations is quite interesting.
  3. I appreciate the ablations and the code-level examples, which help explain what the integration step is actually doing.
Critical weaknesses
  1. The contributions are not clearly separated. The paper proposes both removing task evaluation and integrating modifications from multiple agents. However, I don't think the experiments sufficiently distinguish the benefits of these two components. In particular, it would be interesting to compare against approaches that use actual evaluation scores for selecting or integrating modifications.

  2. I am not entirely convinced about how generalizable the approach is. The initial coding agent seems relatively simple, and many of the improvements are functionalities already available in existing coding agents. I wonder whether the method would still provide meaningful improvements when starting from a stronger agent, or after more generations.

  3. The contribution of the integration step is not entirely clear to me. Looking at Table 3, even single exploration achieves 81.7% compared to 85.6% for ELITE. I think the paper could better investigate how much integration actually contributes, especially compared to selecting or combining modifications using task evaluation scores.

  4. I think the claims about better generalization need stronger evidence. The transfer benchmarks are relatively small, and the baselines are only run once. Given that some of the improvements are rather small, I am not sure the current results are sufficient to support this conclusion.

Questions for authors
  1. Does the method still work when initialized from a reasonably strong coding agent?

  2. How much improvement can be attributed to integration itself, compared to selecting children using actual evaluation scores?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Agpf (1/5)

Summary of our response

We thank the reviewer for the constructive feedback on separating the contributions, testing a stronger starting agent, and strengthening the evidence for transfer performance. Our revisions address three main points. First, Section 5.4 (Table 4) adds Evaluation-based Select-one and Direct joint implementation and organizes the ablations into three groups to clarify the roles of child selection, executable exploration and integration, and other components. The score-based selection comparison achieves higher Pro performance, while ELITE obtains higher transfer scores and a higher three-benchmark average. Second, Section 5.3 (Table 3) adds mini-swe-agent results for the initial agent and accepted children 1, 3, and 6, showing further improvement across accepted updates and final gains on all three benchmarks. Third, Sections 5.1-5.2 and the main Table 1 now reflect three independent DeepSeek runs per method; Appendix A.1.2 (Table 7) reports the baseline runs and sample standard deviations. ELITE retains higher mean transfer scores and a higher overall average. Appendix A.5.2 details the added experiments, and Appendix A.7 discusses their scope.

AuthorsOpenReview ↗

Response to Reviewer Agpf (2/5)

W1: The contributions are not clearly separated. The paper proposes both removing task evaluation and integrating modifications from multiple agents. However, I don't think the experiments sufficiently distinguish the benefits of these two components. In particular, it would be interesting to compare against approaches that use actual evaluation scores for selecting or integrating modifications.

Q2: How much improvement can be attributed to integration itself, compared to selecting children using actual evaluation scores?

Response to W1 and Q2 about Separating Contributions and Selection Using Task Scores

We thank the reviewer for suggesting that we further distinguish the roles of downstream task feedback and mechanism integration.

We have added two comparisons, Evaluation-based Select-one and Direct joint implementation, and reorganized the revised Table 4 into three groups to make the contributions examined by each comparison clearer: (1) Alternative selection strategies retain seven independently implemented children and compare integration with child selection, using either LLM judgments or actual task scores; (2) Exploration and integration ablations examine single-branch exploration and joint implementation without separate executable children; and (3) Other component ablations examine structured hypothesis generation and deterministic validity gates. The complete Table 4 is reproduced below.

Table 4: Design ablations of ELITE. All methods use DeepSeek. Scores are success rates (%); Avg. equally weights the three benchmarks. Full ELITE and the no-gates ablation report three-run means; all other variants use one run. Bold numbers indicate column maxima.

MethodProVerifiedPolyglotAvg.
Full ELITE (K=7K=7)85.673.369.376.1
Alternative selection strategies
Select-one (LLM-based)83.368.064.071.8
SIFT-style Select-one85.072.046.067.7
Evaluation-based Select-one91.770.062.074.6
Exploration and integration ablations
Single exploration (K=1K=1)81.764.064.069.9
Direct joint implementation78.372.066.072.1
Other component ablations
w/o hypothesis generation85.068.060.071.0
w/o deterministic gates59.451.345.352.0

To address the distinction between integration and downstream task feedback, we examine two comparisons separately.

(1) Without downstream task feedback, does integration outperform selecting a single exploratory child? Select-one (LLM-based) and SIFT-style Select-one both retain K=7K=7 same-parent exploration and replace integration with selecting one child. Select-one uses a single LLM decision, while SIFT-style Select-one aggregates pairwise LLM judgments with a Bradley-Terry model. Neither uses downstream candidate task scores. Their three-benchmark averages are 71.8% and 67.7%, compared with 76.1% for full ELITE, a difference of 4.3 and 8.4 percentage points, respectively. With exploration breadth and the absence of task feedback held constant, these comparisons examine integration relative to the two LLM-based selection strategies.

(2) With the same seven exploratory children, how does integration compare with selection using actual task scores? The added Evaluation-based Select-one comparison uses the original DeepSeek backbone and initial scaffold. Within each round, it evaluates each of the seven independently implemented children on the same 30 Pro tasks, requiring 7×30=2107\times30=210 child-task evaluations, and takes the highest-scoring child as the next agent without integration. This comparison uses one complete independent evolution run.

Evaluation-based Select-one scores 91.7% on Pro, compared with ELITE's 85.6%, but obtains lower scores on Verified (70.0% versus 73.3%) and Polyglot (62.0% versus 69.3%); its three-benchmark average is 74.6%, compared with 76.1%. Thus, in this comparison, selection using actual task scores improves target-benchmark performance, while integration without candidate task feedback yields better transfer scores and a higher overall average. This comparison changes the next-agent construction rule from integration to score-based selection; it does not add feedback to an otherwise unchanged integration pipeline.

The added Direct joint implementation comparison addresses a complementary aspect: generating seven proposals and implementing them jointly, without separate executable children, yields an average of 72.1%. Together, these comparisons clarify the roles of selection, executable exploration, and integration. We have revised the analysis in Section 5.4 (Table 4) and provided the corresponding settings and budgets in Appendix A.5.2.

AuthorsOpenReview ↗

Response to Reviewer Agpf (3/5)

W2: I am not entirely convinced about how generalizable the approach is. The initial coding agent seems relatively simple, and many of the improvements are functionalities already available in existing coding agents. I wonder whether the method would still provide meaningful improvements when starting from a stronger agent, or after more generations.

Q1: Does the method still work when initialized from a reasonably strong coding agent?

Response to W2 and Q1 about a Stronger Starting Agent and Further Evolution

We thank the reviewer for raising concerns about the capabilities of the initial scaffold and the benefits of subsequent recursive improvement.

The initial scaffold in our original experiments reused the DGM implementation with the necessary adaptations. To examine ELITE from another established starting point, we added an experiment using mini-swe-agent v2.4.6. It is an open-source coding-agent scaffold maintained by the SWE-agent team and adopted in several agent evolution studies: SIFT uses it as the initial harness, while Live-SWE-agent builds a self-evolution mechanism on top of it.

We held the Claude-opus-5.5 backbone fixed and ran ten evolution rounds, yielding six accepted children. Child indices count accepted updates, rather than evolution rounds. We evaluated the initial agent and accepted children 1, 3, and 6 on Pro, and the initial and final agents on Verified and Polyglot. These evaluations were performed post hoc and did not guide evolution. Table 3 of the revised manuscript is reproduced below.

Table 3: ELITE evolution from mini-swe-agent. Scores are success rates (%); gains are in percentage points. Child indices follow acceptance order; child 6 is the final agent after ten evolution rounds. Dashes indicate unreported scores.

AgentProVerifiedPolyglot
Initial agent71.770.072.0
Accepted child 173.3----
Accepted child 380.0----
Accepted child 6 (final)83.378.078.0
Initial-to-final gain+11.7+8.0+6.0

Pro increases from 71.7% initially to 73.3%, 80.0%, and 83.3% for accepted children 1, 3, and 6, respectively. The final agent also improves Verified from 70.0% to 78.0% and Polyglot from 72.0% to 78.0%. These results extend the evidence beyond the original DGM scaffold and show further gains across multiple accepted updates when starting from an established agent.

We have added these results and their analysis to Section 5.3 (Table 3), the experimental settings to Appendix A.5.2, and discussion of evolution dynamics and evaluation scope to Appendix A.7.

Reviewer_vNbTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ELITE, a framework for coding-agent self-improvement without downstream task evaluation during evolution. It generates multiple modifications from the same agent, integrates compatible mechanisms through code-level reasoning, and applies deterministic validity checks before accepting updates. Experiments show improved task performance and cross-benchmark transfer with reduced evolution costs compared with evaluation-driven baselines.

Critical strengths

The motivation is compelling: reducing costly downstream evaluations is a practical challenge in agent self-improvement, and using implementation-level reasoning to guide evolution offers a promising alternative.

Critical weaknesses

1In one run, performance rises from 60.0% to 81.7% after the first accepted update, reaching 86.7% eventually. This suggests that much of the gain may come from fixing basic scaffold limitations, leaving the benefit of sustained recursive improvement unclear. 2A missing ablation is to generate seven improvement proposals and implement them jointly, without constructing seven separate candidates. This would clarify whether executable exploration provides benefits beyond proposal generation and synthesis.

Questions for authors

See weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer vNbT (1/3)

Summary of our response

We thank the reviewer for the constructive suggestions on the initial scaffold and the role of executable exploration. Our revisions address two main points. First, Section 5.3 (Figure 3 and Table 3) clarifies the original trajectory's further 5.0-percentage-point gain after the first accepted update and adds mini-swe-agent results for the initial agent and accepted children 1, 3, and 6. After ten evolution rounds, the final agent improves Pro, Verified, and Polyglot by 11.7, 8.0, and 6.0 percentage points, respectively, with the intermediate results showing continued gains across accepted updates. Second, Section 5.4 (Table 4) adds the requested Direct joint implementation ablation: jointly implementing seven proposals without separate executable children yields an average of 72.1%, compared with 76.1% for full ELITE. Appendix A.5.2 provides the experimental settings, and Appendix A.7 discusses the mechanism and scope of continued improvement.

AuthorsOpenReview ↗

Response to Reviewer vNbT (2/3)

W1: In one run, performance rises from 60.0% to 81.7% after the first accepted update, reaching 86.7% eventually. This suggests that much of the gain may come from fixing basic scaffold limitations, leaving the benefit of sustained recursive improvement unclear.

Response to W1 about the Initial Scaffold Limitations and Subsequent Improvement

We thank the reviewer for raising concerns about the capabilities of the initial scaffold and the benefits of subsequent recursive improvement.

The initial scaffold in our original experiments reused the DGM implementation with the necessary adaptations. The first accepted update highlighted by the reviewer did indeed yield a substantial improvement. Meanwhile, performance in that run continued to improve from 81.7% to 86.7% during subsequent evolution, a further gain of 5.0 percentage points. Thus, the original results include additional gains beyond the first update. To further address the concern about limitations of the initial scaffold, we added an experiment starting from the established mini-swe-agent scaffold.

We used mini-swe-agent v2.4.6, an open-source coding-agent scaffold maintained by the SWE-agent team and adopted in several agent evolution studies. For example, SIFT uses it as the initial harness, while Live-SWE-agent builds a self-evolution mechanism on top of it. It provides a representative starting point for testing ELITE on another existing scaffold. We held the Claude-opus-5.5 backbone fixed and ran ten evolution rounds, yielding six accepted children. Child indices count accepted updates, rather than evolution rounds. We evaluated the initial agent and accepted children 1, 3, and 6 on Pro, and the initial and final agents on Verified and Polyglot. These evaluations were performed post hoc and did not guide evolution. Table 3 of the revised manuscript is reproduced below.

Table 3: ELITE evolution from mini-swe-agent. Scores are success rates (%); gains are in percentage points. Child indices follow acceptance order; child 6 is the final agent after ten evolution rounds. Dashes indicate unreported scores.

AgentProVerifiedPolyglot
Initial agent71.770.072.0
Accepted child 173.3----
Accepted child 380.0----
Accepted child 6 (final)83.378.078.0
Initial-to-final gain+11.7+8.0+6.0

The intermediate results directly address whether improvement continues beyond the first accepted update: Pro increases from 73.3% at accepted child 1 to 80.0% at child 3 and 83.3% at child 6, a further gain of 10.0 percentage points after child 1. The final agent also improves on both transfer benchmarks. These results provide evidence that ELITE's benefits extend beyond the original DGM scaffold and include gains across multiple accepted updates when starting from an established agent.

We have added these results and their analysis to Section 5.3 (Table 3), the experimental settings to Appendix A.5.2, and discussion of the mechanism and scope of continued improvement to Appendix A.7.

AuthorsOpenReview ↗

Response to Reviewer vNbT (3/3)

W2: A missing ablation is to generate seven improvement proposals and implement them jointly, without constructing seven separate candidates. This would clarify whether executable exploration provides benefits beyond proposal generation and synthesis.

Response to W2 about the Joint-Implementation Ablation

We thank the reviewer for suggesting this targeted ablation. To test whether independent executable exploration provides benefits beyond proposal generation and synthesis, we added an ablation that directly implements seven proposals jointly.

This experiment uses the DeepSeek configuration from the original paper, with the same initial scaffold, evaluation tasks, and number of evolution rounds as full ELITE. Each round still generates seven improvement proposals, but produces a single child directly from them, skipping the seven independent candidate implementations and the subsequent integration based on those implementations. Thus, this variant retains multiple-proposal generation and joint implementation while testing the role of the intermediate executable exploration stage.

For consistency with the revised manuscript, we reproduce the complete Table 4 below. The table now groups the comparisons into alternative selection strategies, exploration and integration ablations, and other component ablations. The Direct joint implementation row in the second group is the experiment added specifically in response to W2; the other rows provide the broader ablation context.

Table 4: Design ablations of ELITE. All methods use DeepSeek. Scores are success rates (%); Avg. equally weights the three benchmarks. Full ELITE and the no-gates ablation report three-run means; all other variants use one run. Bold numbers indicate column maxima.

MethodProVerifiedPolyglotAvg.
Full ELITE (K=7K=7)85.673.369.376.1
Alternative selection strategies
Select-one (LLM-based)83.368.064.071.8
SIFT-style Select-one85.072.046.067.7
Evaluation-based Select-one91.770.062.074.6
Exploration and integration ablations
Single exploration (K=1K=1)81.764.064.069.9
Direct joint implementation78.372.066.072.1
Other component ablations
w/o hypothesis generation85.068.060.071.0
w/o deterministic gates59.451.345.352.0

Removing independent executable exploration reduces performance on all three benchmarks, with the average success rate decreasing from 76.1% to 72.1%. This result supports the additional value of independently implementing improvement proposals before integration, beyond generating and jointly implementing multiple proposals. The key distinction lies in the implementation-level evidence available for integration. While both approaches begin with seven improvement proposals, direct joint implementation must resolve their interactions within a single implementation process. In contrast, ELITE independently realizes each proposal as an executable modification of the same parent, making the resulting code changes, interfaces, dependencies, and potential conflicts explicit. This allows the integrator to identify complementary mechanisms and reconcile their interactions based on concrete implementations rather than proposal descriptions alone.

This distinction is further illustrated by the code-level example in Section 5.3, where two exploratory children independently introduce exit-status feedback and continuous output reading. The integrated implementation preserves both mechanisms, demonstrating how executable exploration can provide a concrete basis for combining complementary functionality. We have added the ablation and corresponding analysis to Section 5.4 (Table 4), with the experimental settings in Appendix A.5.2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response to All Reviewers

We thank all three reviewers for their time, careful reading, and constructive feedback. We appreciate Reviewer vNbT's recognition of the practical motivation for reducing costly downstream evaluations. We thank Reviewer Agpf for recognizing the value of our ablations and code-level examples in explaining the integration process. We also appreciate Reviewer 8Kiq's recognition of the novelty of the approach and interest in the main and transfer results.

We have revised the Introduction and Related Work to clarify ELITE's positioning relative to prior approaches. The experimental sections now include mini-swe-agent evolution results, additional design ablations, and repeated DeepSeek runs for DGM and HGM, with corresponding updates to the main results. The appendix provides the supporting experimental settings, budgets, statistics, validity-gate details, and prompt templates. The revised discussion and Limitations further clarify the mechanism and scope of improvement and identify archive-based search as future work.

Below, we respond to each reviewer in turn and address their comments point by point, presenting the relevant evidence, clarifications, and revisions. In these responses, we also indicate where the corresponding changes appear in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.