- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DGM/HyperAgents, but without doing any evals. Instead it just asks the agents to improve the agent without ever testing it and giving that feedback to the meta-agent.
Critical strengths
This is an interesting, novel paper. The core idea is simple, yet rather radical. It suggests we can make progress on ADAS/DGM without doing any evals. That's a new take, and the results are surprising. I find the main results and the transfer results quite interesting and shocking.
Critical weaknesses
-
The main weakness, as the authors acknowledge, is a lack of replication. There is only one (!) run of the baselines. Worse, they are high-variance methods, so we don't know if the results are signal or noise. I recommend the authors add replicates and/or harvest results from those methods from their original papers.
-
Relatedly, there are zero statistics in the paper. We thus do not know what is signal and what is noise. Even with a single replicate, there are some cases where you may be able to use some fancy stats to try to help us differentiate signal from noise, but you'd have to know enough to use those complex stats appropriately.
1 and 2 are a shame, because the paper is interesting.
-
There is not enough discussion of WHY this might work. It's rather radical to believe we can improve our performance on a task without testing on that task? So, why does this work? Does it only work up to a point? Surely we can't expect it to keep outperforming empirical tests for long, right?
-
How do we know that the models you are using did not read the DGM/ADAS/HGM/HyperAgents work and their code, and that's why it is able to do this remarkable thing?
-
I may have missed it, but as far as I saw, the critical gates are not described. What are these? Scripts? libraries? That you made or others did? Or LLMs? More minor things:
-
Line 48-49 this language seems more to describe generational population methods (e.g. classic evolutionary algorithms), but DGM and similar use archives.
-
Lines 82-84 and elsewhere. HyperAgents can see the whole archive of past solutions and can (and did) crossover/combine subcomponents of different solutions when making a new solution, so this writing is inaccurate. DGM also looked at the whole history of its lineage and all the patches that led to it. So both can study mechanisms/features and choose to combine, fix, repair, etc. irrespective of a reward score if they choose. I thus don't think it's accurate so say your method is the first to allow crossover/picking specific ideas/features and merging them/etc. Please correct this language.
-
Line 71-74: these descriptions are too dense to understand what these works did and how it is different from yours.
I like this paper and want to see it published eventually, but I cannot vote for acceptance without stats and proper rigorous science to differentiate signal from noise. If you can address most or all of my key weaknesses, I would be delighted to raise my score.
Questions for authors
See my questions in the Weaknesses section above.
Also, by only having one solution chosen as the parent of each new crop, you abandon the principles of open-endedness that are important for solving very hard problems. Why do that? Did you consider or try a more archive or population-based approach? It is not incompatible with your idea. At a minimum, I recommend pointing that out as a possible future work direction.
What do you tell the model at the start? What is it's prompt?
In figures like Fig 2, why did you not also compare to HyperAgents?
For single exploration, did you give it the same budget in terms of generated solutions?
22 comments
Response to Reviewer 8Kiq (2/11)
W1: The main weakness, as the authors acknowledge, is a lack of replication. There is only one (!) run of the baselines. Worse, they are high-variance methods, so we don't know if the results are signal or noise. I recommend the authors add replicates and/or harvest results from those methods from their original papers.
W2: Relatedly, there are zero statistics in the paper. We thus do not know what is signal and what is noise. Even with a single replicate, there are some cases where you may be able to use some fancy stats to try to help us differentiate signal from noise, but you'd have to know enough to use those complex stats appropriately.
Response to W1 and W2 about Independent Replications and Statistical Reliability
We thank the reviewer for emphasizing the importance of independent repeated runs and statistical analysis in assessing the reliability of our results.
The original submission reported three independent ELITE runs per backbone, but only one run per baseline. We have added two complete independent evolution runs each for DGM and HGM under DeepSeek, giving all three methods three runs in this configuration. Each complete evolution run is the unit of replication. The table below reproduces Table 7 from Appendix A.1.2; Run 1 contains the original baseline results, and Runs 2 and 3 are the additions. We report means and sample standard deviations, with Avg. equally weighting the three benchmarks. ELITE's individual runs for both backbones appear in Table 6 of the revised manuscript.
Table 7: DeepSeek baseline repetitions and comparison with ELITE. Success rates are percentages; sample SDs are percentage points. Run 1 reports the original baseline results.
| Method and run | Pro | Verified | Polyglot | Avg. |
|---|---|---|---|---|
| DGM Run 1 | 86.7 | 64.0 | 62.0 | 70.9 |
| DGM Run 2 | 85.0 | 64.0 | 62.0 | 70.3 |
| DGM Run 3 | 88.3 | 62.0 | 64.0 | 71.4 |
| DGM Mean SD | ||||
| HGM Run 1 | 86.7 | 68.0 | 64.0 | 72.9 |
| HGM Run 2 | 88.3 | 62.0 | 64.0 | 71.4 |
| HGM Run 3 | 90.0 | 66.0 | 66.0 | 74.0 |
| HGM Mean SD | ||||
| ELITE Mean SD |
ELITE's mean success rates on Verified and Polyglot are 73.3% and 69.3%, higher than DGM's 63.3% and 62.7% and HGM's 65.3% and 64.7%. Its three-benchmark average is , exceeding DGM and HGM by 5.2 and 3.3 percentage points, respectively. DGM and HGM retain higher mean Pro scores. The added runs therefore quantify run-to-run variability and preserve the observed advantage on the evaluated transfer benchmarks and their three-benchmark average.
We have also updated the main Table 1 and Section 5.2 to use the three-run DeepSeek means for DGM and HGM. Section 5.1 specifies the repetition counts, and Appendix A.1.2 provides the per-run results and sample SDs. Qwen baseline results remain single-run observations. These statistics characterize variation across the available independent runs; the Limitations paragraph and Appendix A.7 describe the scope of the comparisons and the value of further repetitions and broader task coverage.
Response to Reviewer 8Kiq (3/11)
W3: There is not enough discussion of WHY this might work. It's rather radical to believe we can improve our performance on a task without testing on that task? So, why does this work? Does it only work up to a point? Surely we can't expect it to keep outperforming empirical tests for long, right?
Response to W3 about Why Evaluation-Free Evolution Can Work and Its Limits
We thank the reviewer for raising this fundamental question. We have expanded the explanation of the mechanism and its limits, supported by the additional ablations, the mini-swe-agent experiment, and the existing implementation analyses.
(1) Why can improvement occur without candidate task feedback?
Downstream task scores are one source of information for improving coding agents; another is the combination of the LLM's existing programming knowledge and inspectable source code. The LLM parameters remain fixed, while the scaffold governs tool invocation, execution feedback, and the problem-solving process. Concrete limitations, such as missing exit-status feedback, blocking output reads, or incomplete parsing, can be identified from the implementation without first evaluating an evolved candidate on downstream tasks. Improving these shared mechanisms can benefit task execution across different instances.
ELITE's Explore-then-Integrate process turns proposals into inspectable implementations. Independent exploration exposes code changes, interfaces, and dependencies; integration uses this evidence to combine complementary mechanisms and reconcile conflicts; deterministic checks filter some structural and operational errors. These steps provide a route from existing programming knowledge to scaffold improvement, although operational validity alone does not guarantee downstream gains.
(2) What evidence supports this explanation?
The revised Table 4 adds Evaluation-based Select-one and Direct joint implementation and groups the comparisons by child selection, exploration and integration, and other components. We reproduce the complete table below.
Table 4: Design ablations of ELITE. All methods use DeepSeek. Scores are success rates (%); Avg. equally weights the three benchmarks. Full ELITE and the no-gates ablation report three-run means; all other variants use one run. Bold numbers indicate column maxima.
| Method | Pro | Verified | Polyglot | Avg. |
|---|---|---|---|---|
| Full ELITE () | 85.6 | 73.3 | 69.3 | 76.1 |
| Alternative selection strategies | ||||
| Select-one (LLM-based) | 83.3 | 68.0 | 64.0 | 71.8 |
| SIFT-style Select-one | 85.0 | 72.0 | 46.0 | 67.7 |
| Evaluation-based Select-one | 91.7 | 70.0 | 62.0 | 74.6 |
| Exploration and integration ablations | ||||
| Single exploration () | 81.7 | 64.0 | 64.0 | 69.9 |
| Direct joint implementation | 78.3 | 72.0 | 66.0 | 72.1 |
| Other component ablations | ||||
| w/o hypothesis generation | 85.0 | 68.0 | 60.0 | 71.0 |
| w/o deterministic gates | 59.4 | 51.3 | 45.3 | 52.0 |
The results provide three complementary observations. First, full ELITE averages 76.1%, compared with 71.8% for Select-one, 67.7% for SIFT-style Select-one, and 72.1% for Direct joint implementation. The selection variants retain seven independently implemented children, while Direct joint implementation generates seven proposals and implements them jointly as one child. These comparisons support the roles of executable exploration and integration relative to the evaluated alternatives. Second, removing the deterministic gates reduces the average to 52.0%, showing the importance of operational validity constraints in this setting.
Third, Evaluation-based Select-one evaluates each of seven children on the same 30 Pro tasks per round and selects the highest-scoring child without integration. It achieves higher Pro performance (91.7% versus 85.6%), while ELITE has higher Verified (73.3% versus 70.0%) and Polyglot (69.3% versus 62.0%) scores. This comparison changes both the selection signal and the next-agent construction rule, so it compares the two strategies rather than isolating task feedback alone. The repetition counts are stated in the table caption.
Section 5.3 and Appendix A.1.4 provide a concrete implementation example: two exploratory children separately introduce continuous output reading and exit-status feedback. Their integrated descendant returns all 100,000 output lines and the correct exit code of 7 in a local probe. This demonstrates compatible functionality in the resulting code; it does not attribute aggregate benchmark gains to either modification individually.
Response to Reviewer 8Kiq (1/11)
Summary of our response
We thank the reviewer for the thoughtful comments on statistical reliability, the mechanism of evaluation-free improvement, and implementation clarity. Our revisions address four areas. First, Sections 5.1-5.2 and the main Table 1 now reflect three independent DeepSeek runs per method; Appendix A.1.2 (Table 7) reports the added baseline runs and sample standard deviations. Second, Sections 5.3-5.4 (Tables 3-4) add the mini-swe-agent trajectory and two ablations, while Appendix A.7 connects these results to implementation-level reasoning, evolution dynamics, and evaluation scope. Third, Appendix A.5.2 (Table 12) details the deterministic gates and ablation settings; Appendix A.3 documents the prompt templates reproduced below. Finally, Sections 1-2 clarify archive-based prior work and ELITE's distinction, and the Limitations paragraph and Appendix A.7 identify archive-based search as a future direction. Below, we also clarify the role of existing model knowledge and distinguish matched exploration breadth from matched total computation.