acceptodds
Under review as a conference paper at ICLR 2027

When More Experience Hurts: Diagnosing Experience Growth and Reader Alignment in Self-Evolving Language Agents

Abstract

Experience-based self-evolution allows large language models (LLMs) to adapt after deployment without updating their parameters, while keeping accumulated experience readable and auditable. However, whether models can turn a growing memory bank into sustained improvement across model scales remains unclear. In this paper, we study these dynamics across four tasks and three representative strategies, evaluating successive memory checkpoints and varying the models that generate trajectories, edit memory, and use the resulting experience. We find that accumulating experience creates new successes without reliably preserving earlier ones, and that stronger experience sources do not consistently improve weaker readers. Controlled interventions show that the same experience can cause different losses across readers, while experience that helps alone can become ineffective alongside other entries. Guided by these findings, we explore explicit protection of experience applicability boundaries during reuse, with preliminary improvements suggesting a promising direction for more reliable self-evolution.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Experience-based Learning, External Memory, LLM Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_n7aMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a systematic diagnostic analysis of language agents that self-evolve through textual experience. It distinguishes three core roles—trajectory builder, memory editor, and experience reader—and conducts experiments across four representative tasks, three memory-management strategies, and three model scales. The results show that new gains coexist with regressions on previously solved questions, stronger experience sources do not necessarily help weaker readers, and experience utility depends on the reader and surrounding context. These findings provide valuable evidence for understanding the limitations of experience-based self-evolution. Guided by this diagnosis, the authors explore Scope Protection and report preliminary performance improvements on the 4B model.

Critical strengths
  1. Beyond aggregate performance gains, the paper tracks gains and regressions at the question level. It shows that net gains can conceal failures on previously solved questions alongside successes on new ones, highlighting that more experience does not necessarily lead to sustained improvement. This is a meaningful and practically relevant problem.
  2. The authors develop a clear diagnostic framework that distinguishes memory construction and updating from memory reading and use, and examines reader-side differences through matched-input interventions. The appendices also document several attempts that did not yield consistent gains, providing useful lessons for future research.
Critical weaknesses
  1. Evaluation is limited to one model family, leaving generalization across model families unclear. All experiments use the 4B, 35B, and 122B models from the Qwen-3.5 family. However, model families may differ in long-context processing, instruction following, and robustness to interference. It remains unclear whether the observed degree of reader misalignment and scale-related trends generalize to other families.
  2. Differences in experimental protocols and run cohorts limit the interpretability of some cross-condition comparisons. Early HotpotQA construction curves use token F1, later interventions use binary correctness, and Table 4 uses EM. The main 4B/35B SuperGPQA construction curves use an answer-only protocol, whereas the 122B reference and some later analyses use a brief-explanation protocol. In the core refinement experiment in Table 4, Scope Protection uses an earlier complete cohort, native MemRL and Top-1 share another replay cohort, and Compact Examples rebuilds experience. Although these differences are disclosed, they make it difficult to disentangle method gains from protocol and run differences, while also making the main text and appendices harder to follow together.
  3. Scope Protection remains preliminary and does not establish reliable harm reduction. The main text provides limited detail on a reproducible implementation, and evaluation is restricted to the 4B model with MemRL. Average accuracy increases from 44.2% to 45.6% in Table 4, but this difference is subject to the limitations of cross-cohort comparison. More importantly, Appendix F explicitly states that, in its within-cohort comparison, additional corrections outweigh new errors while harm itself increases. The evaluation sets were also reused during exploration, making independent held-out evaluation necessary to establish the robustness of the improvement.
Questions for authors
  1. Could the paper provide its complete procedure or prompt template more clearly? Is the strategy also effective on the 35B model and with the ExpeL and TF-GRPO memory mechanisms?
  2. In regards of Table 4, could the authors conduct repeated paired replays for variants that can share a memory bank, using matched questions, banks, prompts, scoring rules, and budgets, and report construction variability for variants that require rebuilding experience? Given that Appendix F notes reuse of evaluation sets during exploration, could Scope Protection’s performance and help/harm outcomes be validated on an independent held-out set?
  3. Table 9 shows that, even with the same sample order, the final HotpotQA F1 gain of 4B TF-GRPO changes from -0.99 pp to -36.51 pp. However, Appendix C.2 also states that the API seed was unset and some historical code and content were not fully fixed. Could the authors fix the implementation and scoring protocol, vary generation randomness and the ordering of the 150 construction examples separately, and report multiple runs to distinguish sampling variability, ordering effects, and cumulative degradation from memory updates?
Reviewer_ZcSUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the reliability of experience-based self-evolution in LLM agents. The authors evaluate ExpeL, TF-GRPO, and MemRL across four benchmarks and multiple Qwen3.5 model scales. They find that accumulating experience can introduce new successes while causing regressions on previously solved tasks. Controlled experiments further reveal reader-dependent memory utility and interference between experience entries. The paper also explores Scope Protection as a lightweight mitigation strategy.

Critical strengths

1. Fine-grained analysis of experience accumulation.

The paper tracks question-level outcomes across memory checkpoints rather than relying solely on aggregate accuracy. This reveals how newly acquired successes can coexist with previously correct answers becoming incorrect. Evaluating three memory strategies across four benchmarks provides useful empirical coverage.

2. Useful controlled experiments on experience utilization.

The shared-context experiments show that identical experience can affect readers differently. Table 2(a) reports 111 failures unique to the 4B reader compared with 22 unique to the 35B reader. The single-entry, padded, and combined-context comparisons also provide evidence of interference between experience entries.

Critical weaknesses

1. Limited novelty and insufficient discussion of closely related work.

Several central findings substantially overlap with prior studies that are neither cited nor discussed in the Related Work section. Xiong et al. [1] already investigate how memory addition and deletion affect agent performance. They identify harmful experience reuse and cases where seemingly useful experiences lead to incorrect behavior. Hu et al. [2] explicitly study continual learning through external memory. They demonstrate that accumulating experience can cause forgetting and negative transfer, with memory organization affecting the preservation of previously acquired capabilities. Zhang et al. [3] further show that repeated memory consolidation can turn useful memories into harmful ones. Their experiments directly demonstrate performance degradation on previously solved tasks. These studies closely overlap with the paper's main findings in Q1–Q3, particularly the non-monotonic benefits of memory growth and the loss of earlier successes.

The paper acknowledges cross-model memory transfer failures through MemCollab [4], but its reader-dependent findings require clearer differentiation from this work. Context interference beyond length effects has also been studied in [5]. More importantly, the proposed Scope Protection shares its central motivation with Boundary-Aware Skill Memory [6], which explicitly models experience applicability conditions and risk cues to prevent inappropriate reuse. MemHarness [7] similarly addresses the risks of directly replaying past experience without adapting it to the current task. Apart from [4], none of these studies are cited or discussed. While the paper offers additional question-level diagnostics and controlled comparisons, it remains unclear what fundamentally new insights or actionable principles emerge beyond these existing findings. The omission of such closely related work further weakens the paper's novelty and research positioning.

2. Limited mechanistic insights from the diagnostic analysis.

Most experiments establish behavioral differences without identifying their underlying causes. The reader-alignment analysis shows different outcomes under identical contexts but does not explain why readers use the same experience differently. Similarly, context-removal experiments demonstrate recoverability without isolating the sources of interference. Consequently, the analysis offers limited mechanistic understanding or actionable guidance for improving memory systems.

References

[1] Xiong et al. (2026). How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior. ACL 2026. https://aclanthology.org/2026.acl-long.27/

[2] Hu et al. (2026). When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents. https://arxiv.org/abs/2604.27003

[3] Zhang et al. (2026). Useful Memories Become Faulty When Continuously Updated by LLMs. https://arxiv.org/abs/2605.12978

[4] Chang et al. (2026). MemCollab: Cross-Model Memory Collaboration via Contrastive Trajectory Distillation. https://arxiv.org/abs/2603.23234

[5] Levy et al. (2025). More Documents, Same Length: Isolating the Challenge of Multiple Documents in RAG. EMNLP 2025 Findings. https://aclanthology.org/2025.findings-emnlp.1064/

[6] Lin et al. (2026). When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents. https://arxiv.org/abs/2608.22339

[7] Wu et al. (2026). MemHarness: Memory Is Reconstructed, Not Replayed. https://arxiv.org/abs/2607.28272

Questions for authors

1.Given the existing findings on harmful memory accumulation, forgetting, and negative transfer, what fundamentally new insights does this paper establish beyond prior diagnostic studies?

2.How does the proposed reader–experience alignment analysis differ from existing cross-model memory transfer and context-interference findings? Can the authors isolate experience-specific effects from general context sensitivity?

3.Can the authors quantify how much of the reported recovery exceeds unchanged-input replay variability? Do the interference findings remain consistent on representative evaluation samples rather than selected harm cases?

4.How does Scope Protection differ from existing applicability-aware memory methods? Can its improvements be validated using matched evaluation conditions and an independent held-out test set?

5.Can the authors provide additional experiments to establish the robustness and generalizability of the findings? For example, evaluating Scope Protection against native MemRL on the same independent test set and comparing it with stronger applicability-aware methods such as Boundary-Aware Skill Memory or MemHarness. Extending the analysis to other model families and longer memory trajectories (e.g., 500–1,000 examples) would also help determine whether the observed patterns hold beyond the current setting.

Reviewer_F476OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how accumulating experience affects the performance of self-evolving language agents. Through experiments across four reasoning benchmarks and multiple memory-management approaches, the authors show that additional experience does not always improve performance and can cause previously correct answers to become incorrect. They further examine how memory quality and reader behavior contribute to these failures and explore lightweight interventions to improve experience utilization.

Critical strengths
  1. The paper addresses an important practical problem for agents that continuously learn from past interactions.

  2. The checkpoint-level analysis in Figures 3 and 4 goes beyond aggregate accuracy by tracking newly gained and lost successes, providing useful insights into experience accumulation.

  3. The controlled experiments in Figure 5 and Table 2 offer valuable evidence about how memory composition and reader capabilities influence the effectiveness of previously acquired experience.

Critical weaknesses
  1. The central observation that accumulated experience can harm performance overlaps with prior work, particularly Xiong et al. (ACL 2026), which investigates error propagation and misaligned experience replay. Although the diagnostic analysis is useful, its distinct contribution relative to existing studies could be articulated more clearly.

  2. The experiments primarily use models from one family and relatively short experience histories. While Figure 4 illustrates answer turnover, stronger controls for decoding variability would help establish whether these regressions are consistent rather than stochastic.

  3. The proposed interventions yield relatively modest improvements. In Table 4, Scope Protection improves average accuracy by only 1.4 percentage points, while the other variants provide no overall improvement. The statistical reliability and practical significance of these results remain unclear.

Relevant references missing from the manuscript:

  • Xiong et al. (ACL 2026). How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior.

  • Wang et al. (Findings of ACL 2026). SAMem: State-Aware Memory as a Fine-Grained Memory for LLM Agents in Decision-Making.

  • Liang et al. (Findings of ACL 2026). Learning How to Remember: A Meta-Cognitive Management Method for Structured and Transferable Agent Memory.

Questions for authors
  1. How does the diagnostic framework differ from existing studies of harmful experience replay and memory misalignment? Which findings do the authors consider particularly distinctive?

  2. How much of the answer turnover observed in Figure 4 can be attributed to decoding variability? Could repeated evaluation on a representative subset help clarify this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.