- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
An LLM-based evolutionary method is introduced for improving LLM harnesses. The approach includes behavioral diversity and generalization considerations and is shown to be superior to 'naive' competition-based evolution.
Critical strengths
Overall the algorithm is creative and the paper shows the progression from naive competition, identifying its failure points, and addressing them in the new algorithm. The ablations show the introduced modifications to the naive approach seem to be the right ones. The inclusion of failure points alongside the other three lineages is novel.
Critical weaknesses
None
Questions for authors
There is no mention of the type and size of LLM used for the various steps of the algorithm. Authors also do not mention the LLMs tested with the various harnesses. What prompt was used for the LLM judge? Have the authors applied this method on agentic benchmarks such as SWE bench or terminal bench?