acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Experience Utilization in Self-Evolving Language Model Agents

Abstract

Self-evolving agents improve by accumulating and reusing experience from past interactions. Existing work has largely focused on how experience is constructed, represented, and updated, while paying less attention to how experience should be used during runtime decision-making. As a result, most agents rely on rigid usage strategies, either injecting experience once at initialization or at every step, without considering whether it is needed for the current decision. This paper studies experience utilization as a critical design dimension of self-evolving agents. We ask whether agents benefit from interweaving experience use with decision-making, so that experience is invoked only when additional guidance is needed. To examine this question, we introduce ExpWeaver, a lightweight instantiation that leaves experience construction unchanged and modifies only runtime utilization by exposing experience as an optional resource during reasoning. Across four representative frameworks, seven LLM backbones, and three types of environments, ExpWeaver consistently achieves the best performance among different utilization strategies. Reinforcement learning experiments further show that this behavior can be amplified through training. Usage-pattern, causal ablation, and entropy-based analyses reveal that ExpWeaver enables agents to invoke experience selectively, at beneficial decision points, and under higher reasoning uncertainty. Overall, our findings call for a shift from merely studying what experience to store toward understanding how and when experience should enter decision-making.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Self-Evolving Agent, Experience Utilization, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5AeLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies when self-evolving language-model agents should use experience from previous interactions. The authors observe that most existing methods either inject experience once at the beginning of a task or retrieve it at every decision step, without adapting experience use to the agent's current decision state. To address this limitation, the paper proposes ExpWeaver, which allows an agent to emit [RETRIEVE] during ReAct reasoning, after which the system retrieves relevant experience before the agent continues with action generation. The authors integrate this mechanism into ReasoningBank, AWM, SkillRL, and G-Memory, and evaluate it on ALFWorld, WebShop, and several question-answering datasets under both prompting and reinforcement-learning settings. The main conclusion is that allowing an agent to decide when to retrieve experience generally improves task performance over fixed Init-only and Always-on strategies.

Critical strengths

The paper studies a practical question that has received relatively little systematic investigation. Experience-driven agents need to decide not only what to store, but also when to use what they have stored. ExpWeaver is lightweight, and its explicit trigger makes the timing of retrieval observable and amenable to analysis. The experiments cover multiple experience representations, models, and types of task environments, as well as both prompting and reinforcement-learning settings. This breadth is a clear strength of the paper. In addition to reporting task performance, the authors analyze retrieval frequency, temporal patterns within trajectories, Empty/Random ablations, and output entropy. These are all relevant directions of analysis.

Critical weaknesses
  1. The lack of a standard ReAct tool-use baseline leaves the novelty and advantage of the method insufficiently established

A more natural ReAct baseline would expose the paper's retriever as an ordinary tool that the model can call autonomously, for example, RetrieveSkills(query, k=6). During reasoning, the model would decide whether and when to invoke this tool. A tool call would return only the IDs, names, and short descriptions of the top-6 candidate skills. Based on this information, the model could then call ReadSkill(skill_id) to read the full content of one or a small number of selected skills. For a strictly controlled comparison, this baseline could use the same experience repository, retrieval backend, and query construction as ExpWeaver. Such a two-stage baseline would likewise use experience only when needed, without injecting the full content of every retrieved candidate at once. Because the paper compares ExpWeaver only against no experience, initialization-time injection, and injection at every step, the current results show at most that adaptive access can outperform fixed injection. They do not establish that SkillRL's bulk injection of all retrieved skills after a trigger is superior to staged, selective reading by a ReAct agent, nor do they establish ExpWeaver as a sufficiently distinct methodological contribution.

  1. Injecting multiple experience items in bulk may be less efficient and more susceptible to interference than selective reading by a ReAct agent

ReasoningBank returns the top-3 memory items at each retrieval. SkillRL returns up to six task-specific skills, subject to a similarity threshold of 0.4, together with all general skills. G-Memory first uses MiniLM to retrieve the top-kk historical queries, then uses an LLM to select the top-MM queries and compress the corresponding interaction graphs, after which the retrieved insights and interaction subgraphs are injected into the prompt. AWM likewise refers to multiple “relevant workflows.” Across these four frameworks, the model therefore largely decides only whether to trigger retrieval; it cannot first inspect short descriptions of the retrieved candidates and then selectively read the full content of one or a few items.

Bulk injection increases context length and may expose the model to irrelevant, redundant, or conflicting experience. A more appropriate ReAct comparison would provide a dynamically callable retrieval tool. When the model needs experience, it would call this tool and receive only the IDs, names, and short descriptions of the top-6 candidates, after which it could use ReadSkill(skill_id) to read the full content of one or a few candidates as needed. This would avoid conflating retrieval quality with the reading policy, while potentially reducing context use and interference. ExpWeaver is not compared against this natural baseline and may perform worse than staged ReAct reading when the experience repository is large or highly confusable. The paper also does not report the number of experience items or tokens actually injected per retrieval, nor does it include stress tests that vary repository size, candidate similarity, or the prevalence of conflicting experience. SkillRL's injection of up to six task-specific skills together with all general skills is the clearest and potentially most severe instance of this issue.

  1. The retrieval-query implementation is underspecified, and the causal attribution to retrieval timing remains incomplete

The paper states that ExpWeaver uses the current context ctc_t as the retrieval query when retrieval is triggered, but it does not define the precise composition, serialization, or truncation of ctc_t. For example, it remains unclear whether the query includes the full interaction history, which observations, actions, and reasoning traces are included, and whether previously retrieved experience is retained.

Moreover, Init-only retrieves experience using information available at the beginning of a task, whereas ExpWeaver retrieves during interaction using a richer current context. The two conditions may therefore differ not only in when experience is presented, but also in which experience is retrieved. The Random Utilization ablation similarly constructs queries at randomly selected states without holding either the query or the retrieved content fixed. The current evidence supports the effectiveness of the complete adaptive experience-utilization pipeline, but it does not strictly attribute the improvement to when retrieval occurs. The authors should specify the exact query construction and introduce controlled experiments that separate the contribution of retrieval timing from that of retrieval relevance.

Questions for authors

Could the authors compare against a two-stage ReAct baseline using exactly the same fixed experience repository and retrieval backend? In this baseline, the retriever would be exposed as a tool that the model can autonomously decide whether and when to call during reasoning. A call would return only the IDs, names, and short descriptions of the top-6 candidate skills, after which the model could use ReadSkill(skill_id) to read the full content of one or a few selected skills. To control for retrieval differences, the tool could use the same query construction as ExpWeaver. Under this setting, what advantage does ExpWeaver's bulk injection of full experience items provide?

Please specify the complete retrieval-query construction used by ReasoningBank, SkillRL, and G-Memory. What task information, observations, actions, reasoning traces, and previously retrieved experience are included in ctc_t, and how is the query serialized and truncated? Is the current reasoning trace hth_t that triggers [RETRIEVE] included in the query? For AWM, which does not have an explicit retrieval mechanism, please explain how workflows are selected and injected when experience use is triggered. In addition, could the authors provide controlled experiments that hold the query or retrieved content fixed, to better separate the contribution of retrieval timing from retrieval relevance?

Reviewer_XCCwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that work on self-evolving agents has focused on how experience is built and stored, and has neglected when it should be used. Existing methods either inject experience once at the start or retrieve it at every step. ExpWeaver is a prompting change to a ReAct agent: the agent can emit a [Retrieve] token while reasoning, and only then is experience retrieved, using the current context as the query. It is evaluated inside two experience frameworks (ReasoningBank and SkillRL) on ALFWorld and WebShop with five backbones, and with GRPO training of Qwen3-4B on ALFWorld. The paper reports ExpWeaver as best in most settings and says RL training widens the gap. Analyses show that stronger models rarely retrieve, that retrieval clusters at the first step, and that ablations with empty or randomly timed retrieval do worse.

Critical strengths
  • The setup asks a clean question. Each framework's experience construction is held fixed and only the usage strategy changes, across five backbones and two environments in Figs. 2-3. In SkillRL all strategies share the same pre-constructed repository, which makes that comparison well controlled.
  • The paper shows that injecting more experience is not always better. Always-on falls below Init-only after GRPO training (67.9 vs. 70.6, Fig. 4) and in some prompting cells (79.88 vs. 83.04 for Qwen3.5 on SkillRL ALFWorld, Fig. 3). Random retrieval scores below Empty in 10 of the 12 cells of Table 2. The explicit trigger also makes usage measurable (Table 1, Fig. 5).
Critical weaknesses
  • The main text does not show that the gains come from retrieved experience. Section 3.2 says that without a trigger the agent "proceeds without additional experience incorporated", yet ExpWeaver still gains where it almost never retrieves. This points to a prompt or scaffold effect. A prompt-only control (ExpWeaver prompt, retrieval disabled) is missing.
  • "Consistently achieves the best performance" is stronger than Figs. 2-3 show, and no variance is reported. The main text gives no run counts or error bars. ExpWeaver is not the best strategy in 3 of the 20 prompting cells, and leads by under 1 point in 6 others (e.g. 84.58 vs. 84.33 for Qwen3.5 on ReasoningBank ALFWorld).
Questions for authors
  • Do Init-only and ExpWeaver use the same retriever, top-k and memory, and differ only in the query (x vs. c_t) and timing? For AWM, which "does not include an explicit retrieval mechanism" (line 258), what does [Retrieve] return?
  • How many runs and test items are behind each bar in Figs. 2-4 and each cell of Table 2, and what are the standard deviations? Why does the w/o experience baseline differ between Figs. 2 and 3 for the same model and task?
  • For Fig. 4, what does Qwen3-4B score under the four strategies before training, and how often does the trained agent retrieve? Without these I cannot tell whether training strengthened selective use.
  • Could you add a prompt-only control, a simple heuristic trigger (for example, retrieve after a failed search, which Section 5.1 says is when re-invocation happens), and an aggregate entropy test for Fig. 6 over all trajectories?
Reviewer_WqzbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how self-evolving LLM agents should use accumulated experience during task execution. The authors propose ExpWeaver, which allows agents to retrieve experience when needed during reasoning, instead of relying on fixed retrieval strategies. Experiments across four agent frameworks, multiple LLM backbones, and different task environments show that ExpWeaver generally improves task performance. The authors also conduct RL experiments and behavioral analyses to further investigate experience utilization.

Critical strengths
  1. The paper studies a meaningful problem in self-evolving agents. The focus on experience utilization is interesting, as most existing research has focused on how to construct and update experience.

  2. The proposed method is simple and can be integrated into different agent frameworks without changing their experience construction pipelines. The experiments cover different experience representations and task environments, and the additional ablation studies help explain the observed performance improvements.

Critical weaknesses
  1. As the authors acknowledge, adaptive experience utilization has already been explored in works such as ExpSeek and ProactAgent. I think the contribution of ExpWeaver needs to be clarified. I am also not fully convinced by the statement in Section 2 that "The impact of experience utilization is intertwined with changes in how experience is formed." Both ExpSeek [1] and ProactAgent [2] have already examined the effects of different experience retrieval strategies to some extent. The authors should explain more clearly what new insights this paper provides beyond these studies.

  2. The experimental baselines are missing comparisons with closely related methods, particularly ExpSeek and ProactAgent. Comparing ExpWeaver with initialization-only, always-on, and random retrieval is useful, but it is not enough to establish its advantage over existing adaptive retrieval methods. I would like to see comparisons with these methods under consistent experimental settings.

  3. For online self-evolving frameworks such as ReasoningBank, using the same experience construction pipeline does not necessarily mean that the experience repositories are identical. Differences in task execution order, randomness during inference, or agent behavior could lead to different accumulated experiences. How do the authors control for these factors? Were the experience repositories kept fixed or synchronized across different utilization strategies?

  4. In Figure 2, under ReasoningBank on WebShop, Always-on outperforms ExpWeaver with Kimi-K2.5. Could the authors provide further analysis of these cases? They seem to contradict the paper's claim that ExpWeaver consistently achieves the best performance.

  5. The improved success rate after RL training does not necessarily show that the model has learned a better retrieval policy. Although the paper includes analyses of experience utilization patterns, it remains unclear how the retrieval behavior changes through RL training.

[1] ExpSeek: Self-Triggered Experience Seeking for Web Agents

[2] Ask Only When Needed: Proactive Retrieval from Memory and Skills for Experience-Driven Lifelong Agents

Questions for authors

See Critical Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.