- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies instability issue in memory-bsaed self-evolving agent. It probes two aspects: 1) cross-run variance; 2) sample ordering - both can lead to significantly degradation of agent performance, raising questions about previously reported results from Agent Workflow Memory and Reasoning Bank. It identified two specific issues that cause the instablity: task and environment underspecification. By specifying the tasks and getting environmental feedback, the baseline results can be further improved.
However, I think the overall contribution is slightly below the bar of a ICLR paper. To improve the paper, the authors can consider 1) propose a more comprehensive solutions for the observed issues; 2) expand the study to more benchmarks or agent harness.
Critical strengths
- The paper is clearly motivated, rooted in the stability of self-evolving agents. This is serious issue in my work that the reported numbers might be exaggerated.
- The two identified issue: cross-run variance and sample order are both intuitive and well investigated with clear evidence.
- The additional analysis that helped proposed effective solutions (task and environment specifications) are good findings and effective backed by the experimental results.
Critical weaknesses
- The first weakness is the novelty - since self-evolving a process of optimization, we can expect the two identified issues (not saying these are trivial), but solutions based on these findings are relatively weak based on some manual observations.
- I also have concerns that these solutions themselve (despite useful in the tested data and agents), may be overfitting to these small range of benchmarks and agents. Hence, a more comprehensive needed to improve the soundness of the work.
- Memory-based agent now is a relatively narrow scope of harness based agent. The authors should consider extending the work to the entire harness evolution.
Questions for authors
- Have you tried the proposed solution on other benchmarks?
- What is the reason to focus only on Agent Workflow Memory and Reasoning Bank?
3 comments
Weakness 2: I also have concerns that these solutions themselve (despite useful in the tested data and agents), may be overfitting to these small range of benchmarks and agents. Hence, a more comprehensive needed to improve the soundness of the work.
- +Rub and +Env are benchmark- and method-agnostic by design: they pass information that the evaluator or environment already produces to the memory construction step, without benchmark-specific tuning. Our response to Q1 above also shows that +Rub helps on another benchmark and another memory method. We therefore do not believe these two interventions are overfit to the settings in this paper.
- +PMod is different: its content is specific to our experiments (e.g., no API access, no user confirmation). However, we believe the procedure behind it is transferable: when unintended strategies are observed in memory, the corresponding environment constraints can be stated explicitly in the prompt to steer self-improvement back on track.
Weakness 3: Memory-based agent now is a relatively narrow scope of harness based agent. The authors should consider extending the work to the entire harness evolution.
- Thank you for the suggestion. We agree this is an important direction, and Appendix E already takes a small step toward it. The letta code based method is not limited to a memory bank: its reflection agent can edit the system prompt and create or edit skill files. The same reliability issue appears: enabling it does not improve overall success and increases variance in the majority of cases.
- More broadly, we expect the causes we identify to extend beyond memory. Any agent that persistently modifies itself based on task outcomes is stateful, so early randomness can compound, and results can depend on task order and on how well the feedback is specified. Harness evolution is a recent and fast-moving direction that gives the agent a larger action space, so these issues may matter at least as much there. Our recommendations still apply: stress-test the system and build mechanisms for human intervention. We will make this connection explicit in the revised paper.
Question 2: What is the reason to focus only on Agent Workflow Memory and Reasoning Bank?
- We chose these two because both are established methods with public implementations, and both were validated on web browsing tasks, the focus of this paper. This lets us use their reported results as a reference point (Table 2).
- Our conclusions do not rest on these two alone. In Appendix E we evaluated a third, more recent filesystem-based method, where enabling memory still fails to give consistent gains.
Thank you again and please let us know if you have follow-up questions!
Follow up on our response
Dear Reviewer P5p1,
Thank you again for your review and suggestions! We hope our earlier response has helped address your concerns.
Since then, we have posted two additional sets of results in our responses to other reviewers, which improved the strength of our work, and may also be relevant to your concerns: (1) experiments with controlled easy-to-hard and hard-to-easy task orders, which show that difficulty direction affects the learning outcome; and (2) a leakage-controlled analysis showing that the gain from rubrics is essentially unchanged after excluding tasks with shared answers, and thus not explained by answer leakage.
Please let us know if you have any remaining concerns. Thank you!
Thank you for the detailed and constructive review! We are glad that you found the paper written clearly, and the two identified issues well supported by evidence. We respond to your concerns and questions below.