acceptodds
Under review as a conference paper at ICLR 2027

Mendel Gödel Machine: Recursive Self-Improving Coding Agents via Comparative Evolution

14 views

Abstract

Self-improving coding agents that iteratively rewrite their own source code have demonstrated impressive performance on coding tasks. However, existing solutions generally derive self-modification from a single failure trajectory at a time, overlooking rich comparative signals available in the agent's expanding archive of past attempts. According to Mendelian principles of controlled inheritance, we introduce Mendel Gödel Machine (MGM). In addition to the general single-trajectory clonal mutation, MGM includes two new types of self-modification that better utilizes evidences accumulated: the reaction-norm mutation edits an agent based on its trajectories on multiple tasks simultaneously, and the cross-lineage hybridization edits an agent using the trajectory of a reference agent from another lineage on the same task. Under an additive fitness landscape model, we prove theoretically and demonstrate via controlled surrogate simulation that the new strategies facilitate a faster and better convergence over single-trajectory baselines. Experiments on SWE-bench and Polyglot confirm MGM's consistent improvement in performance, efficiency, and generalizability. Our code is publicly available at https://anonymous.4open.science/r/MendelGM/

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Recursive Self-Improvement, Gödel Machine, Agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_T8E5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper does the Darwin Gödel machine with some new, more fancy mutation operators that improve performance on a task and transfer results.

Critical strengths
  1. The paper offers what seem like nice improvements over the Darwin-Gödel machine in terms of having some new mutation operators that are interesting and seem to perform well.

  2. The results about transfer, if they hold up to statistical tests and replication, are interesting and a bit surprising.

  3. Overall this research area (DGMs, ADAS, HGM, HyperAgents are very promising and important), so it is great to improve up on them.

Critical weaknesses
  1. The main weakness in the paper is a lack of replicated runs and statistical analysis. As far as I can tell from the text, there's only one run of each experiment. That's unacceptable in science. We have to be able to differentiate signal from noise, and small sample sizes can be deceiving. We need multiple runs of each treatment and proper statistics to tell us whether the differences reflect signal or chance. I'm tempted to say this is a major, unfixable problem with the paper because I don't know whether you can address it in the short time we have for review, unless you've been doing other replicates in the background in anticipation of this concern and can report updated results and statistics. Unfortunately, what seems like an overall strong paper has this rather fatal flaw, and unless it can be addressed, it's hard to argue for publication. Please look into the problems associated with the replication crisis in science, specifically in reinforcement learning and machine learning more broadly, including the work and presentations by Joelle Pineau on this topic. If you can provide replications and stats, and the results hold, I would definitely increase my score.

  2. The paper relies too much on mathematical equations to explain its ideas. A reader should not have to slog through lots of math in place of a clean, crisp explanation, especially when little or no math is needed to explain the concept. It's fine to include the math too, although in many cases it's probably unnecessary or could go in an appendix. What you can't do is leave some of the paper's key concepts unexplained or underexplained and just rely on a lot of math.

Equation 13 is a great example of why it seems like you're including a ton of math just for the sake of it. It's unnecessary for such a simple concept and takes up space and demands mental energy from the reader without any real payoff.

  1. I think the paper is inaccurate in implying that the Darwin-Gödel machine and HyperAgents only look at one individual and aren't able to combine features produced by different individuals across a lineage or an entire archive. The Darwin-Gödel machine allows agents to look back through the entire lineage and could mix and match features developed at different times, bringing them together through a crossover-like operator if it chooses too. HyperAgents condition on the entire archive and therefore can definitely do things like the crossover operator and other operators proposed here right?. Thus, the paper is overstating its novelty. It is true that you are specifically encouraging those mutations to happen frequently, and you have some results suggesting that this can work well, although we need proper replication and statistics to establish that. But it's not fair to say that these sorts of mechanisms are new in your paper and could not occur with some probability in systems like HyperAgents and even DGM.

  2. The paper mentions its additive fitness model but doesn't cite much literature on that sort of model. Have you done a deep dive into the literature to determine whether this model and the results you claim to show and prove have been presented before? There's a huge amount of theory in the literature on genetic algorithms and biological crossover/sexual reproduction studying when recombination can be beneficial, both empirically and provably. There are all sorts of toy domains, such as bit strings. That literature needs to be scoured and properly cited, and you need to discuss whether what you're doing is new in that context.

  3. We're not told why we should expect transfer to other domains. See the questions section.

Minor comments

  1. There are some typos.

  2. What you call “cross-lineage hybridization” seems to be the same thing as crossover, a term that researchers in genetic algorithms, as well as biologists, have used for decades. Is there any reason to use different terminology? If so, at least mention crossover.

Questions for authors

One perplexing thing is you never explain why we would expect transfer to other domains and to other backbones. You also don't explain whether that's a good thing. Is there something about what your algorithm is doing that makes it more likely to transfer than prior algorithms?

One could imagine that a better optimization algorithm is better at adapting to the task it is given, and we want the algorithm to solve the task it is given. Thus, in some sense, you could argue that transferring better than another algorithm is a bad sign for an algorithm, because it means it's not properly adapting to the task it's asked to solve (unless you're specifically regularizing things in some way that's not explained). It also seems a bit fishy without an explanation of why we would we expect your algorithm to transfer better than prior algorithms, especially since we don't have proper replication and statistics. It would be good to address all of these issues.

5 comments
AuthorsOpenReview ↗

Response to W1: Statistical Significance and Reproducibility

We thank the reviewer for raising concerns about statistical significance and reproducibility. To address these concerns, we conducted expanded benchmark evaluations, repeated evaluations of the selected agents, and independent evolution runs.

1. Statistical Significance Across Two Benchmarks

All experiments are conducted after solving the problem of disabled test cases.

We evaluated the agents on SWE-bench Verified instances(so far 393 tasks have been finished) and conducted three repeated evaluations on Polyglot-225. All methods were evaluated on the same task instances within each benchmark.

The results are summarized below.

BenchmarkInitialHGMMGMMGM − HGMStatistical Significance
SWE-bench Verified-393221/393 (56.23%)242/393 (61.58%)262/393 (66.67%)+5.09 ppp = 0.01045
Polyglot-22537.33 ± 1.54%49.33 ± 1.78%70.22 ± 3.80%+20.89 ppp ≤ 0.00315

Polyglot results are reported as mean ± standard deviation across three independenrt and repeated evaluations. Statistical significance is assessed using the exact two-sided McNemar test on paired instance-level outcomes. For Polyglot-225, the reported p-value is a conservative upper bound across the three evaluations.

These results demonstrate that MGM consistently outperforms HGM on both benchmarks, with statistically significant improvements in resolve rate.

For Polyglot-225, the detailed results of the three repeated evaluations are:

EvaluationInitialHGMMGMMGM − HGMConservative McNemar p-value
r188/225115/225166/225+22.67 pp≤0.000108
r282/225111/225159/225+21.33 pp≤0.000429
r382/225107/225149/225+18.67 pp≤0.003143
Mean ± SD37.33 ± 1.54%49.33 ± 1.78%70.22 ± 3.80%+20.89 ppAll p < 0.01

Thus, the observed performance advantage is not restricted to a single benchmark.

2. Reproducibility Across Independent Evolution Runs

To further assess reproducibility, we conducted two independent SWE-bench evolution runs using the same Qwen3.6-35B-A3B(To run more quickly, here we use fp8 version) backbone and experimental configuration.

Each evolution run independently selected its best agents. We subsequently evaluated each selected agent three times on the same 60 SWE-bench instances.

Evolution RunInitial (3 evaluations)HGM (3 evaluations)MGM (3 evaluations)HGM MeanMGM MeanMGM − HGM
Run 135, 35, 3735, 37, 3740, 40, 4160.56%67.22%+6.67 pp
Run 235, 35, 3735, 38, 3839, 41, 4161.67%67.22%+5.56 pp
Overall Mean59.44%——61.11%67.22%+6.11 pp

All individual evaluation scores are resolved instances out of 60. Run 1 and Run 2 are independent evolution runs; the three evaluations within each run measure the performance of its selected best agents.

MGM consistently outperforms HGM in both independent evolution runs, achieving improvements of 6.67 and 5.56 percentage points, respectively. Moreover, MGM achieves higher scores in all six repeated evaluation comparisons.

These results demonstrate that MGM's advantage is reproducible across independent evolution runs under the same backbone and evaluation conditions.

AuthorsOpenReview ↗

Response to W2 and W4:

W2: Clarity of Mathematical Presentation

We thank the reviewer for this constructive feedback. Our original intention in providing detailed mathematical formulations was to precisely specify the proposed operators, including their eligibility conditions, the diagnostic evidence they receive, and the agents they modify. We aimed to avoid ambiguities between the conceptual description and the actual implementation, thereby ensuring methodological rigor and reproducibility.

However, we agree that formal precision should not come at the expense of readability. Some conditions can indeed be explained more clearly and concisely in natural language. Equation (13), as the reviewer points out, is a good example: it simply specifies that Cross-lineage Hybridization is applicable when two agents have been evaluated on a shared task and at least one of them fails.

To address this concern, we will keep clear, concise natural language descriptions of CM,RM and CH, focusing on their core intuitions, applicability conditions, and operational differences. We will replace the operator-specific mathematical formulations, including Eq. (13), with straightforward textual explanations in the main body.

The complete formal definitions, detailed mathematical specifications, and implementation pseudocode will be moved to a dedicated Reproducibility section in the appendix. This will allow readers who require precise specifications for rigorous reproduction or future reference to access all necessary technical details, while keeping the main text accessible and easy to follow. We appreciate the reviewer's suggestion and believe this revision will substantially improve the clarity of our presentation.

W4: Additive Fitness Landscape and Related Literature

We thank the reviewer for pointing out the extensive literature on fitness landscapes and genetic recombination. We fully acknowledge prior work on fitness landscape analysis, including the foundational study of Royal Road functions [1], as well as theoretical investigations demonstrating the benefits of crossover in evolutionary optimization [2–4].

We would like to clarify that we do not claim the additive fitness landscape model itself as a novel contribution. Rather, we deliberately adopt a simplified and well-established modeling paradigm to provide a theoretical analysis of our proposed self-modification operators.

Specifically, our analysis in §4 and Appendix D.2 investigates how comparative evidence can improve the effectiveness of self-modification. Under the stated assumptions, Eqs. (35–37) establish sufficient conditions under which comparative evidence preserves the true causal defects while eliminating irrelevant candidates, thereby increasing the effective probability of repairing a genuine defect.

The purpose of this analysis is therefore to provide theoretical insight into why our proposed operators can be beneficial, rather than to introduce a new fitness landscape model or claim the first theoretical demonstration that recombination can improve evolutionary search.

We appreciate the reviewer's suggestion and will strengthen the related-work discussion by incorporating the relevant classical literature [1–4] and explicitly clarifying the role of the additive fitness landscape as an analytical model rather than a claimed contribution.

References

[1] Mitchell, M., Forrest, S., & Holland, J. H. (1992). The Royal Road for Genetic Algorithms: Fitness Landscapes and GA Performance. In F. J. Varela & P. Bourgine (Eds.), Toward a Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life (pp. 245–254). MIT Press.

[2] Jansen, T., & Wegener, I. (2002). The Analysis of Evolutionary Algorithms: A Proof That Crossover Really Can Help. Algorithmica, 34(1), 47–66.

[3] Doerr, B., Doerr, C., & Ebel, F. (2015). From Black-Box Complexity to Designing New Genetic Algorithms. Theoretical Computer Science, 567, 87–104.

[4] Sudholt, D. (2017). How Crossover Speeds Up Building-Block Assembly in Genetic Algorithms. Evolutionary Computation, 25(2), 237–274.

AuthorsOpenReview ↗

Response to W3

We thank the reviewer for this important comment. We agree that sufficiently general self-modifying agents could, in principle, implement similar operations. We would like to clarify the mechanisms implemented in DGM/HyperAgent and explain how MGM differs from sequentially accumulating modifications.

1. DGM does not implement archive-conditioned self-modification.

In the official DGM implementation (DGM_outer.py and self_improve_step.py), the archive is used to select a parent, after which self-improvement operates on that parent's reconstructed codebase and its own evaluation feedback. Other archived agents' trajectories and modifications are not explicitly provided to the editor. Thus, although DGM can accumulate improvements sequentially along a lineage, it does not implement the cross-agent comparison or multi-trajectory diagnosis proposed in MGM. Selecting any node from the archive is fundamentally different from using comparative evidence across archive nodes during a single self-modification. This distinction is also discussed in our Appendix B.1 (p. 24).

Thus, we guess that reviewer's suggestion as potentially referring to that RM is simply accumulations of multi-turn CM. We appreciate this perspective, but would like to clarify an important distinction.

2. Our operators aim to discover generalizable improvements, rather than merely accumulate modifications.

The fundamental difference is that RM is designed to identify underlying weaknesses that generalize across tasks, rather than simply combining the modifications produced by multiple CM steps.

Importantly, we attempt to empirically investigate this intended generality in Appendix A.3 (Are Skills Evolved from MGM Really More General?). There, we analyze the characteristics of evolved skills and compare whether MGM and HGM discover reusable workflow-level improvements or more fragmented, task-specific modifications. Our analysis suggests that MGM more consistently discovers general procedures, that can be applied across different tasks and repositories.

Furthermore, §4 and Appendix D.2 provide a theoretical motivation for this distinction. Under informative comparative evidence, RM and CH can narrow the set of plausible failure causes through diagnostic compression, increasing the probability of identifying and repairing underlying weaknesses.

3. HyperAgents has the potential capability, but our empirical analysis reveals a substantial difference in realized behavior.

We acknowledge that HyperAgents, whose implementations are slightly different from DGM, can access historical artifacts and evolve its meta-agent, potentially discovering analogous operations. However, theoretical expressivity does not establish that such operations are systematically performed.

To investigate this, we audited the actual tool-use trajectories of DGM, HGM, and HyperAgents, excluding directory listings and score-only inspections.

MethodCH-like behaviorRM-like behavior
DGM0%0%
HGM0%0%
HyperAgents24.2%2.5%
MGM67.0%29.0%

For HyperAgents, we analyzed the first 24 (To ensure fair comparisons) generations across five independent runs (total of 120 modifications). We additionally inspected 958 modifications from the DGM runs released by the HyperAgents authors, finding no corresponding behaviors.

Importantly, our criteria for HyperAgents are permissive proxies: they count reading another lineage's task outputs and evaluations (CH-like), or multiple trajectories of the same parent (RM-like), without requiring the full diagnostic procedure implemented by MGM. These frequencies are descriptive comparisons across different experimental settings, rather than controlled estimates of causal performance improvement.

4. These explicit mechanisms also provide measurable practical benefits.

Our ablation study further supports the practical effectiveness of the proposed operators. Importantly, these conclusions remain supported after correcting the previously identified test-execution issue.

In summary, more importantly, the core novelty of MGM lies in its two explicitly designed evolutionary operators: RM and CH. These operators introduce systematic mechanisms for leveraging comparative evidence across tasks and lineages to guide more generalizable self-modification. Our contribution is not to claim that similar behaviors are theoretically impossible in existing self-evolving frameworks, but to explicitly formulate, implement, and investigate these two diagnostic operators.

MGM and HyperAgents are complementary instead of competing approaches. Consequently, our proposed operators are fully compatible with HyperAgents and could be naturally incorporated into its evolvable meta-agent.

Reviewer_b1XkOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Mendel Gödel Machine (MGM), an archive-based framework for self-improving coding agents that extends HGM’s tree search. The core observation is that DGM/HGM use only a single failure trajectory for each self-modification, even though the archive already contains the same agent’s performance across tasks and different agents’ performance on the same task. This comparative information is largely left unused. The authors therefore introduce three self-modification operators: clonal mutation, the original single-trajectory approach; reaction-norm mutation, which examines the same agent’s trajectories across tasks; and cross-lineage hybridization, which compares agents from different lineages on the same task, allowing a failing agent to “learn” a generalizable skill from a successful one. A failed-task pool further biases evaluation toward tasks where failures have already occurred, while creating task overlap across lineages.

On the theoretical side, the authors use a surrogate model with an additive fitness landscape to show that, assuming comparative evidence is sound and informative, the comparative operators have higher one-step repair probabilities than the single-trajectory operator. Monte Carlo simulations then illustrate faster convergence.

In experiments using Qwen3.6-35B-A3B and the same budget of 200 evaluations, MGM improves accuracy from 50.8% to 93.2% on Polyglot-60 and from 68.3% to 78.3% on SWE-bench Verified-60, while HGM reaches 77.9% and 73.3%, respectively. The evolved scaffolds also undergo zero-shot transfer: MGM shows positive transfer to SWE-bench Pro and Multilingual, whereas HGM shows little, and MGM retains its lead when the backbone is switched to DeepSeek-V4, reaching 96.9% on Polyglot-225. Ablations show that removing either RM or CH reduces performance.

However, the Polyglot results should be treated with caution: the inherited DGM harness has publicly discussed leakage issues, and the paper does not explicitly report safeguards or an audit.

Critical strengths
  1. The core idea is novel, simple, and well motivated. Existing archive-based methods mainly treat the archive as a leaderboard for sampling. MGM instead turns existing cross-task and cross-lineage trajectories into diagnostic evidence for self-modification. This directly addresses a concrete limitation of HGM without requiring additional evaluations.

  2. The method is clearly presented. The paper explains its relationship to DGM and HGM clearly.

  3. The ablations clearly establish the contribution of both operators. Removing either RM or CH substantially reduces final accuracy, showing that both contribute to the end-to-end results.

Critical weaknesses
  1. The paper shows that MGM outperforms HGM, but not that the gains come from better diagnosis through comparative evidence. This is the paper’s central claim, yet none of its three layers of evidence directly establishes it. The theoretical analysis in Appendix D.2 assumes that comparison only eliminates non-causal candidates (Eq. 35), from which the result follows; for CH, soundness is assumed by construction. The simulations in Appendix D.3 directly specify ρ\rho without modeling candidate-set construction, and MGM and HGM coincide exactly when ρ=1\rho = 1. The real-world experiments report only end-to-end accuracy and ablations, without comparing the edit quality of RM/CH offspring against CM offspring at the operator level, even though the authors already have the relevant data (Figures 10–11). Readers therefore cannot tell whether the gains come from better diagnosis, a change in the evaluation distribution induced by the failed-task pool, or chance differences in search trajectories. The ablation (Table 4) only shows that removing an operator changes the final result; it cannot distinguish whether this is due to degraded diagnostic quality or to altered search dynamics.

  2. The experimental evidence is not yet reliable enough to support the strength of the conclusions. The main Polyglot result—93.3%, described as “surpasses GPT-5”—uses a harness inherited from DGM, for which reference-answer leakage through git history and .meta files has been publicly confirmed. This does not establish that leakage occurred in MGM’s experiments, but it does warrant careful scrutiny of the reported results, as explicitly requested in the Questions section.

  3. The wording repeatedly goes beyond what the evidence supports. The abstract still states, “we prove theoretically … faster and better convergence,” although the proof establishes only a one-step fix-probability inequality under specific assumptions. Figure 2 places scores obtained under different protocols in the same plot for cross-system comparison. Appendix A.4 acknowledges uncertainty from seeds and subset selection without quantifying it. None of these issues is fatal on its own, but together with W1 and W2, they risk giving readers an inflated impression of how thoroughly the method has been validated.

Questions for authors

I would be happy to raise my scores if the questions below are satisfactorily addressed:

  1. Clarification — Implications for the main Polyglot results and the comparison with GPT-5. The evaluation code is inherited from DGM. Roesner & Kohno (https://arxiv.org/abs/2609.17817) observed agents in DGM and HyperAgents recovering hidden tests and reference solutions through this harness, and the HyperAgents authors also confirmed the leakage. Please clarify: (i) whether your harness strips git history and removes .meta/ before agents run; (ii) whether web search can be disabled, ideally blocking all network access except the connections required to reach the LLM server, to prevent agents from searching for solutions externally; and (iii) how frequently git-history commands and accesses to .meta/, example, and test paths occur across all Polyglot trajectories. Please also rerun the Polyglot experiments for both HGM and MGM after stripping git history and removing reference solutions. Since cross-lineage hybridization is itself a mechanism for transferring skills, MGM’s advantage over HGM could partly reflect faster propagation of this exploit. The relative performance claim therefore also depends on resolving this concern.

  2. Clarification — Whether the proposed mechanism is supported. Appendix D.2 now explicitly states the assumption that comparative evidence is sound and informative (Eq. 35), but the derivation essentially establishes that repair probability increases if comparison eliminates only non-causal candidates. For CH, soundness is assumed by construction rather than derived. The data needed to examine this assumption are already available in Figures 10–11. For MGM and both ablation runs, please report offspring utility grouped by generating operator (CM / RM / CH), including the mean, median, and number of evaluations, together with parent-to-child utility changes. If RM/CH offspring do not show greater improvements than CM offspring, the experimental gains would need to be explained by other factors—such as a shift in the evaluation distribution caused by the failed-task pool—rather than attributed to better diagnosis.

  3. Clarification — How the theoretical claim should be interpreted. Appendix D.2 proves a one-step inequality under stated assumptions, pfRM,pfCH>pfCMp_f^{\mathrm{RM}}, p_f^{\mathrm{CH}} > p_f^{\mathrm{CM}}, but does not derive “faster expected convergence” from it. Appendix D.3 also acknowledges that the simulations directly specify ρ\rho rather than model candidate-set construction. Yet the abstract still states, “we prove theoretically … faster and better convergence.” Do the authors intend to add a convergence result—for example, an expected reduction in Hamming distance per unit budget under the surrogate model, accounting for pbp_b and the conditions under which each operator is available—or revise the abstract and conclusions to claim only the fix-probability result together with simulation evidence? Either approach would be acceptable, but the current wording is not.

4 comments
AuthorsOpenReview ↗

Response to RQ1 and W2

We thank the reviewer for identifying this issue. We acknowledge that our original Polyglot evaluation inherited a test-execution issue from DGM, causing some required Java test cases to be skipped and consequently overestimating the reported resolve rates.

After investigating the issue, we corrected the grading procedure and regraded all archived predictions using the complete required test suites.

Corrected results. The original and corrected evaluation results are summarized below.

BenchmarkAgentOriginal GradingCorrected GradingChange
SWE-60Initial41/60 (68.33%)41/60 (68.33%)0
SWE-60HGM44/60 (73.33%)44/60 (73.33%)0
SWE-60MGM47/60 (78.33%)47/60 (78.33%)0
Polyglot-59Initial30/59 (50.85%)25/59 (42.37%)−8.47 pp
Polyglot-59HGM46/59 (77.97%)41/59 (69.49%)−8.47 pp
Polyglot-59MGM55/59 (93.22%)51/59 (86.44%)−6.78 pp
Polyglot-59MGM without RM47/59 (79.66%)45/59 (76.27%)−3.39 pp
Polyglot-59MGM without CH44/59 (74.58%)40/59 (67.80%)−6.78 pp

Root cause and verification. All score reductions occurred on Java instances. For example, the bowling task contains 31 tests, of which 30 were disabled by default. When the complete test suite was executed, the original MGM prediction failed 15 tests.

To validate the correction, we:

  1. Verified the hashes of all 475 archived predictions, ensuring that regrading used the original outputs.
  2. Checked the required test-case names for every instance reported as resolved.
  3. Confirmed that the official reference implementations for all 12 Java instances passed their complete test suites.

The audit found no remaining grading errors in the verified evaluation records.

Impact on the conclusions. Importantly, the grading correction does not change the relative ranking of the methods. MGM continues to outperform HGM on both benchmarks.

BenchmarkOriginal MGM − HGMCorrected MGM − HGM
SWE-60+5.00 pp+5.00 pp
Polyglot-59+15.25 pp+16.95 pp

Furthermore, our new Polyglot-225 evaluation, reported separately, includes three repeated evaluations under the corrected grading procedure:

AgentResolve Rate (Mean ± SD)
Initial37.33 ± 1.54%
HGM49.33 ± 1.78%
MGM70.22 ± 3.80%

MGM maintains an average advantage of 20.89 percentage points over HGM in this expanded evaluation.

For Polyglot-225, the detailed results of the three repeated evaluations are:

EvaluationInitialHGMMGMMGM − HGMConservative McNemar p-value
r188/225115/225166/225+22.67 pp≤0.000108
r282/225111/225159/225+21.33 pp≤0.000429
r382/225107/225149/225+18.67 pp≤0.003143
Mean ± SD37.33 ± 1.54%49.33 ± 1.78%70.22 ± 3.80%+20.89 ppAll p < 0.01

We agree that the original test-execution issue required correction and appreciate the reviewer for bringing it to our attention. We have corrected the grading procedure, comprehensively audited the archived predictions, and updated the affected results. The SWE-bench grading results remain unchanged, and the corrected Polyglot evaluations continue to support MGM's advantage over HGM.

Concerning potential answer leakage through internet access or Git history, we had implemented explicit safeguards to prevent agents from retrieving benchmark solutions through external network access, Git metadata, repository history, or other unintended sources. We also carefully inspected the evaluation environments and agent execution trajectories for potential leakage. Our checks found no evidence of answer leakage or benchmark contamination, supporting the validity of our experimental results.

AuthorsOpenReview ↗

Response to RQ2 and W1

Empirical Effectiveness of RM and CH.

We thank the reviewer for highlighting this distinction. We agree that Eq. (35) establishes a sufficient condition under which comparative evidence can improve repair probability, rather than guaranteeing that the implemented operators always satisfy this condition.

To further examine the effectiveness of our proposed operators, we conducted an additional independent run and analyzed the saved evaluation histories from both the original SWE June experiments and this new run.

Specifically, we compare RM and CH against the corresponding independent HGM runs, using parent–offspring performance differences on matched evaluation tasks. For each operator, we calculate its relative advantage over HGM within each experimental run and aggregate these differences across the two runs, weighted by the number of matched task pairs.

OperatorEvaluated offspringMatched task pairsRelative gain over CM (%)
RM (B)31259+1.15
CH (C)1477+1.94

The results show that both proposed operators exhibit more favorable evolutionary outcomes than the corresponding HGM baseline, with relative advantages of 1.15 and 1.94 percentage points for RM and CH, respectively.

These additional results provide empirical support for the effectiveness of comparative evolutionary guidance. We will clarify that our theoretical analysis characterizes the conditions under which such guidance can be beneficial, while the empirical comparisons evaluate the relative effectiveness of the implemented operators.

AuthorsOpenReview ↗

Response to RQ3 and W3

Clarifying the Scope of Our Theoretical Claims

We thank the reviewer for carefully distinguishing our theoretical result from the broader convergence claim. We agree that the original wording, particularly in the abstract, overstated what our formal analysis establishes. We will therefore revise the abstract and conclusions to accurately reflect the scope of our theoretical results, as suggested by the reviewer.

1. Clarification of the theoretical result.

We acknowledge that an improvement in single-step repair probability does not, by itself, establish faster expected convergence of the complete evolutionary process.

2. Concrete revisions to the manuscript.

Following the reviewer's suggestion, we will replace the original claim in the abstract:

"We prove theoretically and demonstrate via controlled surrogate simulation that the new strategies facilitate a faster and better convergence over single-trajectory baselines."

with:

"Under an additive fitness landscape model, we derive sufficient conditions under which comparative evidence improves single-step defect-repair probability. Controlled surrogate simulations further illustrate the potential benefits of improved repair probabilities for evolutionary search."

We will make corresponding revisions to the Introduction and Conclusion, explicitly distinguishing:

  • Theoretical analysis: Conditional improvements in single-step defect-repair probability.
  • Simulation evidence: Potential improvements in evolutionary search under specified repair probabilities.
  • Empirical evidence: Observed performance improvements of the implemented MGM system.

This distinction will ensure that our theoretical contribution is presented accurately without implying a general convergence guarantee.

3. Other concerns regarding the presentation of evidence.

We also acknowledge the reviewer's concerns regarding Figure 2 and the experimental uncertainty discussed in Appendix A.4.

Figure 2. To ensure fair comparisons, we have evaluated Initial, HGM, and MGM on the complete Polyglot-225 benchmark under the same evaluation protocol, with three repeated evaluations. The results are 37.33% ± 1.54% for Initial, 49.33% ± 1.78% for HGM, and 70.22% ± 3.80% for MGM. We will revise Figure 2 to present these matched results and clearly distinguish externally reported scores obtained under different evaluation protocols.

Experimental uncertainty. We have substantially expanded our empirical evaluation to quantify the uncertainty arising from task selection and stochasticity:

  • Expanded SWE-bench evaluation: Due to limited GPU resources, evaluation of the full 500-instance SWE-bench Verified benchmark is still in progress. We currently report results on 393 validated instances, while the remaining unevaluated instances are still queued. On these 393 instances, Initial achieves 221/393 (56.23%), HGM achieves 242/393 (61.58%), and MGM achieves 262/393 (66.67%). MGM outperforms HGM by 5.09 percentage points, with a statistically significant paired difference.
  • Independent evolution runs: We additionally conducted two independent SWE-bench evolution runs using the same Qwen backbone and experimental configuration. MGM consistently outperforms HGM in both runs, achieving improvements of 6.67 and 5.56 percentage points, respectively.
  • Repeated evaluations: On Polyglot-225, three repeated evaluations of the selected agents yield an average improvement of 20.89 percentage points for MGM over HGM, with the standard deviations reported above.

These additional experiments provide stronger evidence for the consistency and statistical reliability of MGM's improvements. We will explicitly report the current evaluation coverage and distinguish uncertainty across repeated evaluations from variability across independent evolution runs, for which the available number of runs remains limited.

Reviewer_b1XkOpenReview ↗

Additional review comment

I appreciate the clarification regarding the Java issue. However, my main concern is a separate and more fundamental one: whether the hidden tests and reference solutions were accessible to the agent during Polyglot during evolution or evaluation.

Based on my experience, even the corrected HGM result of 69.49% on Polyglot looks surprisingly high. To be clear, I am not saying that it is impossible; better configurations by the authors could account for it. But I would like the soundness of these results to be verified.

Since the discussion period is nearly over and there is likely not enough time to rerun experiments, I instead ask the authors to upload the following existing files, unmodified and as written by the harness, together with a SHA-256 manifest:

(A) Per-task evaluation outputs for the initial agent and the best MGM agent (node #16), on both Polyglot-60 and Polyglot-225:

<task>.md: the full agent trajectory;
<task>_docker.log: including the pre-run workspace listing and any workspace-sanitization log lines;
<task>_eval.md: the grading-time test output;
<task>.json: the submitted model_patch and the resolved flag.

(B) Self-improvement artifacts for every node on the lineage of the best agent (#2 → #9 → #16):

self_evo.md: the trajectory of the self-modifying agent;
self_improve.log: including the diagnosed problem statement;
model_patch.diff: the code modification at each step;
metadata.json: the parent node and evaluation results.

I am happy to run the checks from my side as soon as these files are available. I like the idea of this submission and would be glad to raise my rating once this soundness concern is addressed.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to RQ1/W1 and RQ2/W2

RQ1 and W1: Fair Presentation of Benchmark Results

We thank the reviewer for pointing out this issue. We agree that placing results from the Polyglot-60 subset and the full Polyglot-225 benchmark on the same axes may create a misleading impression of direct comparability.

To address this concern, we have now evaluated Initial, HGM, and MGM on the complete Polyglot-225 benchmark, rather than relying on the original subset results.

We repeated the evaluation three times using the corrected grading procedure(as reviewer 2 pointed out). The results are summarized below.

AgentEvaluation 1Evaluation 2Evaluation 3Mean ± SD
Initial88/22582/22582/22537.33 ± 1.54%
HGM115/225111/225107/22549.33 ± 1.78%
MGM166/225159/225149/22570.22 ± 3.80%

On the complete benchmark, MGM achieves an average improvement of 20.89 percentage points over HGM, consistently outperforming HGM across all three evaluations.

We will revise Figure 2(which will be pushed to Image/) to replace the original subset-based MGM/HGM points with these full Polyglot-225 results. The historical subset results will be reported separately, with their task coverage explicitly identified.

We will also clearly distinguish externally reported results whenever their evaluation protocols differ, and avoid interpreting such cross-protocol comparisons as controlled evidence of superiority.

We appreciate the reviewer's suggestion, which prompted us to improve both the empirical coverage and the fairness of the visualization.

RQ2 and W2

To enhance transparency and facilitate the review process, we have made the evaluation trajectories available in our repository's GitHub Release for independent inspection.

We thank the reviewer for highlighting the limited size of our original SWE-bench Verified evaluation. We agree that evaluating only 60 instances may not provide sufficiently robust evidence.

To address this concern, we have expanded our evaluation to whole 500 SWE-bench Verified instances. Due to limited computational resources, these are all the instances we have completed as of the rebuttal deadline; the remaining 107 instances are still queued for evaluation.

AgentResolvedResolve Rate
Initial221/39356.23%
HGM242/39361.58%
MGM262/39366.67%

MGM outperforms HGM by 5.09 percentage points and the initial agent by 10.44 percentage points.

Furthermore, a paired instance-level comparison shows that MGM resolves 38 instances that HGM fails, whereas HGM resolves only 18 instances that MGM fails. This advantage remains statistically significant under the exact two-sided McNemar test (p = 0.01045).

While we acknowledge that the remaining 107 instances have not yet been evaluated, we believe the results on 393 instances already provide substantially stronger evidence for the effectiveness of MGM than the original 60-instance evaluation. In particular, the consistent performance advantage over HGM across a much larger portion of SWE-bench Verified supports our main empirical conclusions.

We will explicitly report the current evaluation coverage and its limitations in the revised manuscript. We appreciate the reviewer's suggestion, which has helped us substantially strengthen the empirical validation of our work.

AuthorsOpenReview ↗

Response to W3

W3: Limitations of the Theoretical Model

We thank the reviewer for highlighting this important limitation. We agree that the additive fitness landscape model in Section 4 simplifies the complexity of real-world LLM-based code generation. Specifically, representing the genotype as a binary vector and assuming independent, additive contributions from individual defects does not fully capture the nonlinear interactions and complex dependencies present in actual codebases.

We adopted these simplifying assumptions to make the effect of comparative evidence on defect identification and repair probability analytically tractable. However, in practical software engineering tasks, defects may interact with one another, and modifying one component may affect the behavior of others.

We acknowledge this gap between the theoretical abstraction and real-world coding environments. As suggested by the reviewer, we will expand the Limitations section to explicitly discuss these assumptions and their implications.

Specifically, we propose adding the following paragraph:

Limitations of the Theoretical Model. Our theoretical analysis adopts a simplified additive fitness landscape, representing defects as binary variables with independent contributions to overall fitness. While this abstraction enables a tractable analysis of comparative evidence and defect-repair probability, real-world software engineering tasks involve complex dependencies and nonlinear interactions between code components. A modification that resolves one defect may introduce new failures or affect other parts of the system, and the effectiveness of a repair may depend on contextual factors not captured by our model. Consequently, our theoretical results characterize the benefits of comparative evidence under the stated assumptions. However, we don't establish a general convergence guarantee for arbitrary coding agents. Extending the analysis to non-additive fitness landscapes and interacting defects remains an important direction for future work.

We appreciate the reviewer's suggestion and will incorporate this discussion into the revised manuscript to clarify the scope and limitations of our theoretical findings.

AuthorsOpenReview ↗

Summary of authors

Final Summary of Authors

We sincerely thank all reviewers for their constructive feedback and valuable suggestions.

During the rebuttal period, we conducted additional experiments, expanded our benchmark evaluations, corrected the grading issues, and clarified our methodological contributions and theoretical claims.

Importantly, across two independent evolutionary runs, each with three repeated evaluations, MGM consistently outperformed HGM, maintaining a substantial performance advantage. The expanded SWE-bench Verified and corrected Polyglot evaluations further support the effectiveness of our approach.

Finally, we have uploaded a revised manuscript to OpenReview, incorporating the revisions discussed throughout the rebuttal.

We sincerely appreciate the reviewers' time and feedback, which have helped us substantially strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.