- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoSpace, a method that evolves both an LLM agent harness and the set of edits available to modify it. The harness consists of Memory, Retrieval, and Prompt components. Based on failures from previous rounds, EvoSpace decides whether to expand the current edit space and then applies edits to the harness. The method is evaluated on LoCoMo and MemBench with two LLM backbones and improves over the compared memory systems.
Critical strengths
The paper studies an interesting problem in agent self-improvement: the set of allowed harness edits may itself need to change as new failure modes are identified. The experiments cover two LLM backbones, evolution trajectories, ablations, case studies, and transfer experiments.
Critical weaknesses
- I am concerned that the main evaluation does not separate adaptation from evaluation. The same query pool is used to provide feedback during harness evolution and to report the final performance. This is particularly concerning for Memory repair, since repaired memories can be query-specific. The transfer results in Appendix B.4 are helpful, but they do not show performance on unseen queries from the same task after evolution. It is therefore difficult to tell how much of the reported improvement would generalize to new queries.
- The edit-space ablation does not fully isolate the effect of dynamic edit-space adaptation. In Table 3, the Scheduled setting opens Memory repair at a fixed round without a priority hint, while Dynamic both chooses when to expand the space and passes the newly admitted capability to the editor as a priority hint. Therefore, the gap between Scheduled and Dynamic may come from either adaptive admission or the priority hint.
Questions for authors
1.Have you evaluated the evolved harness on held-out queries that were not used during the evolution process?
2.Could you separate the effect of dynamic capability admission from the priority hint given to EDIT, for example by evaluating Dynamic without the hint or Scheduled with the hint?
3.Which LLM is used for DIAGNOSE, META-EVOLVE, and EDIT? I could not find this clearly specified in the paper.