- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper investigates why self-evolving memory systems often achieve substantial improvements in early iterations but struggle to make further progress. The authors analyze three stages of the evolution loop, and through experiments on EvolveMem, AutoResearchClaw, MemPro, and GEPA, and find that evaluation outcomes can be unstable, edits frequently introduce regressions, and previously repaired questions may fail again in later iterations.
Critical strengths
1: The paper studies an important and timely problem. Most existing work on self-evolving agents focuses on improving final performance, while paying relatively little attention to what happens during the evolution process. Understanding why performance improvements plateau is valuable for developing more reliable self-improving systems.
2: The experiments cover multiple evolution methods and include several useful diagnostic analyses. The repeated evaluations, model substitutions, and comparisons of aligned and misaligned edits provide valuable empirical evidence.
3: The question-level analysis provides useful insights beyond aggregate performance. Rather than reporting only the final accuracy, the authors examine which questions are repaired, which previously correct answers become incorrect, and whether repairs persist across iterations.
Critical weaknesses
1: Abstract is too abstract and hard to follow, as it lacks a clear logical flow. Also the paper is unnecessarily difficult to follow. Although the underlying motivation is relatively straightforward, the presentation introduces a large number of configurations, metrics, and terminology before clearly establishing the main research questions. The experiments often read as a collection of experiment logs rather than a coherent investigation, making it difficult to understand which findings are central and how they support the paper's main conclusions.
2: The paper identifies several failure patterns, but their causal contributions to the performance plateau remain unclear. The central claim is that performance gains are lost across three stages of the self-evolution loop. However, most experiments establish correlations between these failure patterns and performance changes, rather than isolating how much each factor contributes to the plateau.
3: Limited evaluation setting. Although the paper considers several evolution methods, the experiments are restricted to two conversational memory benchmarks. LoCoMo contains only ten conversations, and the Mem-Gallery experiments use caption-only scenarios. For an analysis paper, I'd like to see a comprehensive inclusion of different methods.
Questions for authors
1: Can the authors better distinguish the contributions of evaluation noise, edit-induced regressions, and bottleneck exhaustion to the observed plateau?
2: How do the authors distinguish false repairs from genuine improvements on questions with unstable baseline outcomes?
While AI-assisted research is becoming increasingly common, authors remain responsible for ensuring the scientific rigor and quality of their submissions. I encourage the authors to conduct a more thorough evaluation, carefully validate their claims and experiments, and ensure the presentation is sound and well-organized before submitting to a venue such as ICLR.
Thank you for the detailed response and additional experiments, which address most of my concerns. I have updated my rating in support of the paper.
I sincerely encourage the authors to pay more attention to the clarity, coherence, and scientific rigor of the manuscript before future submissions. First impressions can be important in peer review, and a well-written paper makes it easier for reviewers to appreciate its contributions. I hope the authors will continue refining the manuscript to make it more coherent and easier to follow. Good luck!