- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper does the Darwin Gödel machine with some new, more fancy mutation operators that improve performance on a task and transfer results.
Critical strengths
-
The paper offers what seem like nice improvements over the Darwin-Gödel machine in terms of having some new mutation operators that are interesting and seem to perform well.
-
The results about transfer, if they hold up to statistical tests and replication, are interesting and a bit surprising.
-
Overall this research area (DGMs, ADAS, HGM, HyperAgents are very promising and important), so it is great to improve up on them.
Critical weaknesses
-
The main weakness in the paper is a lack of replicated runs and statistical analysis. As far as I can tell from the text, there's only one run of each experiment. That's unacceptable in science. We have to be able to differentiate signal from noise, and small sample sizes can be deceiving. We need multiple runs of each treatment and proper statistics to tell us whether the differences reflect signal or chance. I'm tempted to say this is a major, unfixable problem with the paper because I don't know whether you can address it in the short time we have for review, unless you've been doing other replicates in the background in anticipation of this concern and can report updated results and statistics. Unfortunately, what seems like an overall strong paper has this rather fatal flaw, and unless it can be addressed, it's hard to argue for publication. Please look into the problems associated with the replication crisis in science, specifically in reinforcement learning and machine learning more broadly, including the work and presentations by Joelle Pineau on this topic. If you can provide replications and stats, and the results hold, I would definitely increase my score.
-
The paper relies too much on mathematical equations to explain its ideas. A reader should not have to slog through lots of math in place of a clean, crisp explanation, especially when little or no math is needed to explain the concept. It's fine to include the math too, although in many cases it's probably unnecessary or could go in an appendix. What you can't do is leave some of the paper's key concepts unexplained or underexplained and just rely on a lot of math.
Equation 13 is a great example of why it seems like you're including a ton of math just for the sake of it. It's unnecessary for such a simple concept and takes up space and demands mental energy from the reader without any real payoff.
-
I think the paper is inaccurate in implying that the Darwin-Gödel machine and HyperAgents only look at one individual and aren't able to combine features produced by different individuals across a lineage or an entire archive. The Darwin-Gödel machine allows agents to look back through the entire lineage and could mix and match features developed at different times, bringing them together through a crossover-like operator if it chooses too. HyperAgents condition on the entire archive and therefore can definitely do things like the crossover operator and other operators proposed here right?. Thus, the paper is overstating its novelty. It is true that you are specifically encouraging those mutations to happen frequently, and you have some results suggesting that this can work well, although we need proper replication and statistics to establish that. But it's not fair to say that these sorts of mechanisms are new in your paper and could not occur with some probability in systems like HyperAgents and even DGM.
-
The paper mentions its additive fitness model but doesn't cite much literature on that sort of model. Have you done a deep dive into the literature to determine whether this model and the results you claim to show and prove have been presented before? There's a huge amount of theory in the literature on genetic algorithms and biological crossover/sexual reproduction studying when recombination can be beneficial, both empirically and provably. There are all sorts of toy domains, such as bit strings. That literature needs to be scoured and properly cited, and you need to discuss whether what you're doing is new in that context.
-
We're not told why we should expect transfer to other domains. See the questions section.
Minor comments
-
There are some typos.
-
What you call “cross-lineage hybridization” seems to be the same thing as crossover, a term that researchers in genetic algorithms, as well as biologists, have used for decades. Is there any reason to use different terminology? If so, at least mention crossover.
Questions for authors
One perplexing thing is you never explain why we would expect transfer to other domains and to other backbones. You also don't explain whether that's a good thing. Is there something about what your algorithm is doing that makes it more likely to transfer than prior algorithms?
One could imagine that a better optimization algorithm is better at adapting to the task it is given, and we want the algorithm to solve the task it is given. Thus, in some sense, you could argue that transferring better than another algorithm is a bad sign for an algorithm, because it means it's not properly adapting to the task it's asked to solve (unless you're specifically regularizing things in some way that's not explained). It also seems a bit fishy without an explanation of why we would we expect your algorithm to transfer better than prior algorithms, especially since we don't have proper replication and statistics. It would be good to address all of these issues.
5 comments
Response to W2 and W4:
W2: Clarity of Mathematical Presentation
We thank the reviewer for this constructive feedback. Our original intention in providing detailed mathematical formulations was to precisely specify the proposed operators, including their eligibility conditions, the diagnostic evidence they receive, and the agents they modify. We aimed to avoid ambiguities between the conceptual description and the actual implementation, thereby ensuring methodological rigor and reproducibility.
However, we agree that formal precision should not come at the expense of readability. Some conditions can indeed be explained more clearly and concisely in natural language. Equation (13), as the reviewer points out, is a good example: it simply specifies that Cross-lineage Hybridization is applicable when two agents have been evaluated on a shared task and at least one of them fails.
To address this concern, we will keep clear, concise natural language descriptions of CM,RM and CH, focusing on their core intuitions, applicability conditions, and operational differences. We will replace the operator-specific mathematical formulations, including Eq. (13), with straightforward textual explanations in the main body.
The complete formal definitions, detailed mathematical specifications, and implementation pseudocode will be moved to a dedicated Reproducibility section in the appendix. This will allow readers who require precise specifications for rigorous reproduction or future reference to access all necessary technical details, while keeping the main text accessible and easy to follow. We appreciate the reviewer's suggestion and believe this revision will substantially improve the clarity of our presentation.
W4: Additive Fitness Landscape and Related Literature
We thank the reviewer for pointing out the extensive literature on fitness landscapes and genetic recombination. We fully acknowledge prior work on fitness landscape analysis, including the foundational study of Royal Road functions [1], as well as theoretical investigations demonstrating the benefits of crossover in evolutionary optimization [2–4].
We would like to clarify that we do not claim the additive fitness landscape model itself as a novel contribution. Rather, we deliberately adopt a simplified and well-established modeling paradigm to provide a theoretical analysis of our proposed self-modification operators.
Specifically, our analysis in §4 and Appendix D.2 investigates how comparative evidence can improve the effectiveness of self-modification. Under the stated assumptions, Eqs. (35–37) establish sufficient conditions under which comparative evidence preserves the true causal defects while eliminating irrelevant candidates, thereby increasing the effective probability of repairing a genuine defect.
The purpose of this analysis is therefore to provide theoretical insight into why our proposed operators can be beneficial, rather than to introduce a new fitness landscape model or claim the first theoretical demonstration that recombination can improve evolutionary search.
We appreciate the reviewer's suggestion and will strengthen the related-work discussion by incorporating the relevant classical literature [1–4] and explicitly clarifying the role of the additive fitness landscape as an analytical model rather than a claimed contribution.
References
[1] Mitchell, M., Forrest, S., & Holland, J. H. (1992). The Royal Road for Genetic Algorithms: Fitness Landscapes and GA Performance. In F. J. Varela & P. Bourgine (Eds.), Toward a Practice of Autonomous Systems: Proceedings of the First European Conference on Artificial Life (pp. 245–254). MIT Press.
[2] Jansen, T., & Wegener, I. (2002). The Analysis of Evolutionary Algorithms: A Proof That Crossover Really Can Help. Algorithmica, 34(1), 47–66.
[3] Doerr, B., Doerr, C., & Ebel, F. (2015). From Black-Box Complexity to Designing New Genetic Algorithms. Theoretical Computer Science, 567, 87–104.
[4] Sudholt, D. (2017). How Crossover Speeds Up Building-Block Assembly in Genetic Algorithms. Evolutionary Computation, 25(2), 237–274.
Response to W3
We thank the reviewer for this important comment. We agree that sufficiently general self-modifying agents could, in principle, implement similar operations. We would like to clarify the mechanisms implemented in DGM/HyperAgent and explain how MGM differs from sequentially accumulating modifications.
1. DGM does not implement archive-conditioned self-modification.
In the official DGM implementation (DGM_outer.py and self_improve_step.py), the archive is used to select a parent, after which self-improvement operates on that parent's reconstructed codebase and its own evaluation feedback. Other archived agents' trajectories and modifications are not explicitly provided to the editor. Thus, although DGM can accumulate improvements sequentially along a lineage, it does not implement the cross-agent comparison or multi-trajectory diagnosis proposed in MGM. Selecting any node from the archive is fundamentally different from using comparative evidence across archive nodes during a single self-modification. This distinction is also discussed in our Appendix B.1 (p. 24).
Thus, we guess that reviewer's suggestion as potentially referring to that RM is simply accumulations of multi-turn CM. We appreciate this perspective, but would like to clarify an important distinction.
2. Our operators aim to discover generalizable improvements, rather than merely accumulate modifications.
The fundamental difference is that RM is designed to identify underlying weaknesses that generalize across tasks, rather than simply combining the modifications produced by multiple CM steps.
Importantly, we attempt to empirically investigate this intended generality in Appendix A.3 (Are Skills Evolved from MGM Really More General?). There, we analyze the characteristics of evolved skills and compare whether MGM and HGM discover reusable workflow-level improvements or more fragmented, task-specific modifications. Our analysis suggests that MGM more consistently discovers general procedures, that can be applied across different tasks and repositories.
Furthermore, §4 and Appendix D.2 provide a theoretical motivation for this distinction. Under informative comparative evidence, RM and CH can narrow the set of plausible failure causes through diagnostic compression, increasing the probability of identifying and repairing underlying weaknesses.
3. HyperAgents has the potential capability, but our empirical analysis reveals a substantial difference in realized behavior.
We acknowledge that HyperAgents, whose implementations are slightly different from DGM, can access historical artifacts and evolve its meta-agent, potentially discovering analogous operations. However, theoretical expressivity does not establish that such operations are systematically performed.
To investigate this, we audited the actual tool-use trajectories of DGM, HGM, and HyperAgents, excluding directory listings and score-only inspections.
| Method | CH-like behavior | RM-like behavior |
|---|---|---|
| DGM | 0% | 0% |
| HGM | 0% | 0% |
| HyperAgents | 24.2% | 2.5% |
| MGM | 67.0% | 29.0% |
For HyperAgents, we analyzed the first 24 (To ensure fair comparisons) generations across five independent runs (total of 120 modifications). We additionally inspected 958 modifications from the DGM runs released by the HyperAgents authors, finding no corresponding behaviors.
Importantly, our criteria for HyperAgents are permissive proxies: they count reading another lineage's task outputs and evaluations (CH-like), or multiple trajectories of the same parent (RM-like), without requiring the full diagnostic procedure implemented by MGM. These frequencies are descriptive comparisons across different experimental settings, rather than controlled estimates of causal performance improvement.
4. These explicit mechanisms also provide measurable practical benefits.
Our ablation study further supports the practical effectiveness of the proposed operators. Importantly, these conclusions remain supported after correcting the previously identified test-execution issue.
In summary, more importantly, the core novelty of MGM lies in its two explicitly designed evolutionary operators: RM and CH. These operators introduce systematic mechanisms for leveraging comparative evidence across tasks and lineages to guide more generalizable self-modification. Our contribution is not to claim that similar behaviors are theoretically impossible in existing self-evolving frameworks, but to explicitly formulate, implement, and investigate these two diagnostic operators.
MGM and HyperAgents are complementary instead of competing approaches. Consequently, our proposed operators are fully compatible with HyperAgents and could be naturally incorporated into its evolvable meta-agent.
Response to W1: Statistical Significance and Reproducibility
We thank the reviewer for raising concerns about statistical significance and reproducibility. To address these concerns, we conducted expanded benchmark evaluations, repeated evaluations of the selected agents, and independent evolution runs.
1. Statistical Significance Across Two Benchmarks
All experiments are conducted after solving the problem of disabled test cases.
We evaluated the agents on SWE-bench Verified instances(so far 393 tasks have been finished) and conducted three repeated evaluations on Polyglot-225. All methods were evaluated on the same task instances within each benchmark.
The results are summarized below.
Polyglot results are reported as mean ± standard deviation across three independenrt and repeated evaluations. Statistical significance is assessed using the exact two-sided McNemar test on paired instance-level outcomes. For Polyglot-225, the reported p-value is a conservative upper bound across the three evaluations.
These results demonstrate that MGM consistently outperforms HGM on both benchmarks, with statistically significant improvements in resolve rate.
For Polyglot-225, the detailed results of the three repeated evaluations are:
Thus, the observed performance advantage is not restricted to a single benchmark.
2. Reproducibility Across Independent Evolution Runs
To further assess reproducibility, we conducted two independent SWE-bench evolution runs using the same Qwen3.6-35B-A3B(To run more quickly, here we use fp8 version) backbone and experimental configuration.
Each evolution run independently selected its best agents. We subsequently evaluated each selected agent three times on the same 60 SWE-bench instances.
All individual evaluation scores are resolved instances out of 60. Run 1 and Run 2 are independent evolution runs; the three evaluations within each run measure the performance of its selected best agents.
MGM consistently outperforms HGM in both independent evolution runs, achieving improvements of 6.67 and 5.56 percentage points, respectively. Moreover, MGM achieves higher scores in all six repeated evaluation comparisons.
These results demonstrate that MGM's advantage is reproducible across independent evolution runs under the same backbone and evaluation conditions.