- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents AGENT GA, a framework that uses genetic algorithms to improve solutions generated by autonomous coding agents. Instead of directly evolving code, the framework evolves the initial conditions of agent runs, including task prompts and archived solutions from previous generations. Multiple agents independently explore and improve solutions, while an evolutionary algorithm selects better results and adjusts the probabilities of different search operations using a modified HEDGE algorithm. Experiments on 16 Kaggle competitions show that AGENT GA achieves higher average leaderboard rankings than AIDE, AutoGluon, and a single-agent baseline.
Critical strengths
The paper addresses a valuable problem. Improving the effectiveness of autonomous coding agents on long-running optimization tasks is important. Allowing agents to reuse previous solutions while exploring different approaches is a practical direction.
Critical weaknesses
-
The methodological contribution is not sufficiently established. The concern is that the paper does not clearly show what specific challenges of autonomous agent search require the proposed design, or how its adaptations address these challenges beyond existing approaches. The paper identifies early termination and the difficulty of maintaining useful information across long agent runs. It addresses these issues by restarting agents, passing previous solutions through files, and applying population-based search. These are reasonable design choices, but the paper does not explain why they require the proposed evolutionary framework rather than a simpler restart-and-reuse approach. Moreover, the paper defines agent-seed optimization over stochastic agent executions, but the algorithm does not explicitly address the uncertainty in evaluating different seeds. As a result, the main methodological insight of the proposed framework remains unclear.
-
The experimental comparisons are not fair in terms of computational resources. Although AGENT GA significantly outperforms AIDE, the two methods use very different computational budgets. As acknowledged in Appendix G.1, AIDE uses approximately three orders of magnitude fewer LLM tokens. Therefore, the improvement could simply result from substantially more computation and search attempts rather than the proposed framework. The single-agent comparison has a similar problem. Although both methods are given the same nominal attempt allowance, most single agents stop early. AGENT GA actually performs 4.7 times as many scored experiments and incurs 20.7 times the LLM cost. This comparison does not establish that the proposed evolutionary framework is more effective under a comparable budget.
-
The experiments do not establish the contribution of the proposed components. The paper does not provide sufficient ablation studies to show why the proposed framework works. In particular, it does not isolate the effects of population-based search, archive inheritance, genetic operators, or HEDGE-based adaptive allocation. The reported 70.7% win rate of agents inheriting previous solutions, compared with 17.8% for agents starting from scratch, is also not convincing evidence for the proposed inheritance mechanism. Agents with inherited solutions start from previously optimized results, while the others must start over. This difference is expected and does not demonstrate the value of the specific design. More importantly, the paper does not compare AGENT GA against a simple approach that repeatedly restarts an agent with the best previous solution. Such a baseline would help determine whether the proposed genetic algorithm is necessary at all.
Questions for authors
-
What specific challenge of autonomous agent search does the proposed evolutionary framework solve that cannot be addressed by simpler approaches? In particular, why are population-based evolution and the proposed agent-seed formulation necessary beyond repeatedly restarting agents with previous solutions?
-
How does AGENT GA compare with AIDE and other baselines under the same computational budget? Can the authors provide comparisons with matched LLM token costs or numbers of scored experiments?