- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies when self-evolving language-model agents should use experience from previous interactions. The authors observe that most existing methods either inject experience once at the beginning of a task or retrieve it at every decision step, without adapting experience use to the agent's current decision state. To address this limitation, the paper proposes ExpWeaver, which allows an agent to emit [RETRIEVE] during ReAct reasoning, after which the system retrieves relevant experience before the agent continues with action generation. The authors integrate this mechanism into ReasoningBank, AWM, SkillRL, and G-Memory, and evaluate it on ALFWorld, WebShop, and several question-answering datasets under both prompting and reinforcement-learning settings. The main conclusion is that allowing an agent to decide when to retrieve experience generally improves task performance over fixed Init-only and Always-on strategies.
Critical strengths
The paper studies a practical question that has received relatively little systematic investigation. Experience-driven agents need to decide not only what to store, but also when to use what they have stored. ExpWeaver is lightweight, and its explicit trigger makes the timing of retrieval observable and amenable to analysis. The experiments cover multiple experience representations, models, and types of task environments, as well as both prompting and reinforcement-learning settings. This breadth is a clear strength of the paper. In addition to reporting task performance, the authors analyze retrieval frequency, temporal patterns within trajectories, Empty/Random ablations, and output entropy. These are all relevant directions of analysis.
Critical weaknesses
- The lack of a standard ReAct tool-use baseline leaves the novelty and advantage of the method insufficiently established
A more natural ReAct baseline would expose the paper's retriever as an ordinary tool that the model can call autonomously, for example, RetrieveSkills(query, k=6). During reasoning, the model would decide whether and when to invoke this tool. A tool call would return only the IDs, names, and short descriptions of the top-6 candidate skills. Based on this information, the model could then call ReadSkill(skill_id) to read the full content of one or a small number of selected skills. For a strictly controlled comparison, this baseline could use the same experience repository, retrieval backend, and query construction as ExpWeaver. Such a two-stage baseline would likewise use experience only when needed, without injecting the full content of every retrieved candidate at once. Because the paper compares ExpWeaver only against no experience, initialization-time injection, and injection at every step, the current results show at most that adaptive access can outperform fixed injection. They do not establish that SkillRL's bulk injection of all retrieved skills after a trigger is superior to staged, selective reading by a ReAct agent, nor do they establish ExpWeaver as a sufficiently distinct methodological contribution.
- Injecting multiple experience items in bulk may be less efficient and more susceptible to interference than selective reading by a ReAct agent
ReasoningBank returns the top-3 memory items at each retrieval. SkillRL returns up to six task-specific skills, subject to a similarity threshold of 0.4, together with all general skills. G-Memory first uses MiniLM to retrieve the top- historical queries, then uses an LLM to select the top- queries and compress the corresponding interaction graphs, after which the retrieved insights and interaction subgraphs are injected into the prompt. AWM likewise refers to multiple “relevant workflows.” Across these four frameworks, the model therefore largely decides only whether to trigger retrieval; it cannot first inspect short descriptions of the retrieved candidates and then selectively read the full content of one or a few items.
Bulk injection increases context length and may expose the model to irrelevant, redundant, or conflicting experience. A more appropriate ReAct comparison would provide a dynamically callable retrieval tool. When the model needs experience, it would call this tool and receive only the IDs, names, and short descriptions of the top-6 candidates, after which it could use ReadSkill(skill_id) to read the full content of one or a few candidates as needed. This would avoid conflating retrieval quality with the reading policy, while potentially reducing context use and interference. ExpWeaver is not compared against this natural baseline and may perform worse than staged ReAct reading when the experience repository is large or highly confusable. The paper also does not report the number of experience items or tokens actually injected per retrieval, nor does it include stress tests that vary repository size, candidate similarity, or the prevalence of conflicting experience. SkillRL's injection of up to six task-specific skills together with all general skills is the clearest and potentially most severe instance of this issue.
- The retrieval-query implementation is underspecified, and the causal attribution to retrieval timing remains incomplete
The paper states that ExpWeaver uses the current context as the retrieval query when retrieval is triggered, but it does not define the precise composition, serialization, or truncation of . For example, it remains unclear whether the query includes the full interaction history, which observations, actions, and reasoning traces are included, and whether previously retrieved experience is retained.
Moreover, Init-only retrieves experience using information available at the beginning of a task, whereas ExpWeaver retrieves during interaction using a richer current context. The two conditions may therefore differ not only in when experience is presented, but also in which experience is retrieved. The Random Utilization ablation similarly constructs queries at randomly selected states without holding either the query or the retrieved content fixed. The current evidence supports the effectiveness of the complete adaptive experience-utilization pipeline, but it does not strictly attribute the improvement to when retrieval occurs. The authors should specify the exact query construction and introduce controlled experiments that separate the contribution of retrieval timing from that of retrieval relevance.
Questions for authors
Could the authors compare against a two-stage ReAct baseline using exactly the same fixed experience repository and retrieval backend? In this baseline, the retriever would be exposed as a tool that the model can autonomously decide whether and when to call during reasoning. A call would return only the IDs, names, and short descriptions of the top-6 candidate skills, after which the model could use ReadSkill(skill_id) to read the full content of one or a few selected skills. To control for retrieval differences, the tool could use the same query construction as ExpWeaver. Under this setting, what advantage does ExpWeaver's bulk injection of full experience items provide?
Please specify the complete retrieval-query construction used by ReasoningBank, SkillRL, and G-Memory. What task information, observations, actions, reasoning traces, and previously retrieved experience are included in , and how is the query serialized and truncated? Is the current reasoning trace that triggers [RETRIEVE] included in the query? For AWM, which does not have an explicit retrieval mechanism, please explain how workflows are selected and injected when experience use is triggered. In addition, could the authors provide controlled experiments that hold the query or retrieved content fixed, to better separate the contribution of retrieval timing from retrieval relevance?