acceptodds
Withdrawn from ICLR 2027

Rethinking Self-Evolving Agent Memory: Content, Writer, and Source

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model agents can improve through interaction experience by storing and reusing external textual memory. However, there is little understanding of how such memory enables self-evolution and which aspects of its design actually drive downstream performance. Existing methods often simultaneously change what is stored, which model writes it, which experiences are used, and how the memory is maintained, making their contributions difficult to isolate. In this paper, we decompose the memory pipeline into controlled design dimensions and systematically study four common representations—raw trajectories, reflections, rules, and procedural skills—under a shared agent framework across six interactive benchmarks. We find that there is no universally best memory representation. Instead, performance depends on whether the abstraction of the stored experience matches the reusable structure of the environment. We further find that diversity does not consistently outperform repeated experience, whereas success-derived memory consistently outperforms failure-derived memory, with gaps of up to 38 percentage points. Together, our results shift the design of self-evolving agent memory from accumulating more experience and stacking additional mechanisms toward selecting reliable experience and matching its representation to the structure of the task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving Agents, Agent Memory, Memory Representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nNomOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies experience-derived textual memory for LLM agents. It compares raw trajectories, reflections, rules, and procedural skills across six benchmarks, and examines memory writers, experience sources, evolution budgets, and maintenance policies on benchmark subsets. The acting model is held fixed, and memory is frozen during held-out evaluation. The reported findings include benchmark-dependent representation rankings, strong performance from raw trajectories, non-uniform benefits from different writers and maintenance policies, and advantages of success-derived memory. The intended contribution is a set of empirical design principles for agent memory.

Critical strengths
  1. The paper addresses a consequential design question through a useful comparative framework. Organizing the study around memory content, writer, and experience source makes the design space easier to analyze. The six-benchmark representation comparison, shared acting model, and frozen-memory evaluation provide practical information for researchers choosing memory designs.

  2. Several observed effects are substantial. Skill memory improves ALFWorld success from 58% to 79%, Rule memory improves AppWorld from 20% to 34%, and success-only versus failure-only Rule memory differs by 38 percentage points on ALFWorld. These results show that memory design and source selection can materially affect downstream performance and motivate more controlled investigation.

Critical weaknesses
  1. The protocol does not fully isolate the factors that the paper aims to disentangle. Section 3.1 describes an online loop in which memory influences subsequent rollouts, which then update memory. Changing the representation or writer can therefore also change the experience used to construct later memory. This supports comparison of complete online configurations, while isolating the transformation of experience requires holding the underlying trajectories fixed. Similarly, collecting 100 qualifying episodes for success-only and failure-only conditions does not ensure equal total interaction budgets or matched source-task distributions. These distinctions matter because the intended conclusions concern the effects of representation, writer, and outcome source themselves.

  2. Some comparative conclusions exceed the numerical and statistical evidence. Lines 375–377 state that writer changes produce a smaller spread than representation changes on all three benchmarks. In Figure 4, however, ALFWorld Rule memory ranges from 57% to 83% across writers, a 26-point spread, while the largest representation spread within an individual writer is 13 points. The stated comparison therefore needs correction or a different, explicit definition. More broadly, repeated runs are reported but the main figures and tables omit uncertainty estimates, making smaller rankings, maintenance effects, and saturation around 50–100 episodes difficult to assess. My weak-rejection recommendation is based on these limitations in the reliability and interpretation of the proposed design principles.

Questions for authors
  1. Were the same rollout traces supplied to different representations and writers, or were traces regenerated under each evolving memory? Please clarify what is fixed in the evolution-data protocol and what constitutes an episode. For Table 3, can the existing logs provide total rollout attempts and source-task distributions for the success-only and failure-only conditions? A comparison using successful and failed trajectories from matched tasks, if available, would help assess the outcome-source interpretation without requiring a complete rerun.

  2. How was the spread comparison in lines 375–377 calculated, and how does it reconcile with Figure 4's 26-point writer spread for ALFWorld Rule memory? Please also provide existing per-run results or uncertainty estimates for the principal comparisons, preferably using the shared evaluation tasks. Clarifying which rankings, maintenance gains, and saturation claims remain supported would be useful.

Reviewer_6ynrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically investigates the effects of memory content, memory writers, experience sources, and memory management on self-evolving LLM agents through a unified framework and evaluations across six benchmarks.

Critical strengths

The paper addresses an important research problem and provides useful empirical insights into how different memory design choices affect agent performance.

Critical weaknesses

Limited representativeness of the canonical memory implementations. Although the unified framework enables controlled comparisons, the four canonical memory representations may not fully capture the capabilities of existing memory paradigms. In particular, mechanisms such as iterative reflection and hierarchical skill composition are not fully represented. This raises concerns about whether the conclusions generalize beyond the specific implementations considered in this paper. Limited actionable guidance for memory design in new environments. The paper aims to provide general principles for designing agent memory, but its findings mainly identify which representations perform better on specific benchmarks. It remains unclear how these observations can guide memory selection in new environments, especially when tasks require multiple types of memory simultaneously. Insufficient analysis of the memory evolution process. Although the framework supports memory revision, merging, and deletion, the analysis mainly focuses on downstream performance. The paper provides limited insight into how memory quality, abstraction, and correctness change during evolution, making it difficult to understand the mechanisms underlying memory improvement. Insufficient validation of the relationship between task structure and memory representation. A central conclusion is that memory representations should match task structure. However, explanations for the effectiveness of different representations are largely based on observed benchmark results. Without explicitly characterizing task properties or validating their relationship with memory effectiveness, this conclusion remains primarily descriptive rather than predictive. Potential confounding factors in the controlled comparisons. Although the authors standardize the agent and memory pipeline, different memory representations use different retrieval keys, which may affect their relative performance. Additionally, the success-only and failure-only settings may require different numbers of environment interactions and produce different task distributions. These factors complicate the attribution of performance differences solely to memory content or experience outcomes.

Questions for authors

Please refer to the questions I raised in the "weakness" section.

Reviewer_Ao6LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how different design factors in experience-driven memory systems affect the performance of self-evolving LLM agents. To this end, the authors conduct controlled comparisons within a unified framework, examining various memory representations, writer models, and experience sources through self-evolution experiments. Experiments across six interactive benchmarks show that no single memory representation consistently performs best, and stronger memory writers do not necessarily produce better memories. The analysis of experience sources reveals a more consistent pattern: success-derived memories outperform failure-derived ones across all tested settings. The authors further analyze memory scaling, task diversity, and memory-management operations to provide empirical insights into the design of self-evolving agent memory systems.

Critical strengths
  1. The motivation is clear and meaningful, and the paper is well organized, clearly written, and easy to follow.

  2. This paper provides a clear and systematic empirical analysis of memory design choices within a unified framework. The authors compare different memory representations, writer models, and experience sources under consistent experimental settings. These findings provide useful empirical guidance for designing memory systems for self-evolving agents.

Critical weaknesses
  1. Limited Incremental Insights Beyond Existing Empirical Studies. While the paper provides a systematic analysis of memory design choices within a unified framework, several of its main findings overlap with observations from prior studies. For instance, EvoMemBench has already shown that no single memory form consistently performs best across different settings, while SkillEvolBench has reported that raw trajectory reuse can outperform distilled procedural skills. I appreciate the value of conducting controlled comparisons across these factors. However, as a “rethinking” paper, it does not sufficiently clarify what new insights its unified analysis provides beyond these existing observations. The authors are encouraged to better distinguish their novel findings from previously reported results and explain what additional understanding can be gained specifically from their controlled experiments under a unified framework.

  2. Insufficient Analysis Explaining the Main Findings. A central conclusion of the paper is that the effectiveness of different memory representations depends on how well they match the structure of the task. However, this conclusion is primarily supported by performance comparisons and qualitative interpretations of benchmark characteristics, with limited direct evidence explaining the underlying mechanisms. For example, Section 4.1 attributes the effectiveness of Skill memory on ALFWorld to recurring workflows and the advantage of Rule memory on AppWorld to local action preconditions. While these explanations are plausible, the current experiments do not directly verify them. Although Section 5.1 provides some memory retrieval statistics, they do not establish why different representations work better for particular tasks. I would appreciate more targeted analyses, such as examining Skill reuse across different ALFWorld task families or whether Rule memory actually reduces precondition-related failures in AppWorld. Such analyses would better support the paper's conclusions and provide more actionable insights into memory design.

Questions for authors

My main concerns are discussed in the Weaknesses section. I have one additional question:

  1. Comparability of Success-Only and Failure-Only Experience Sources. In Table 3, the All-outcomes condition uses the first 100 evolving episodes, while the Success-only and Failure-only conditions continue until 100 episodes with the selected outcome are collected. This may introduce differences in the total number of interactions and the task difficulty distribution. Could the authors clarify the total number of episodes generated and whether the source task distributions are comparable across these conditions? A comparison under matched interaction budgets or controlled task difficulty would help isolate the effect of experience outcome.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.