- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the selection bottleneck in external experience memory for LLM agents: as the memory bank grows, memories selected by semantic retrieval may become ineffective or even cause negative transfer when decision conditions such as phase, object, tool, and precondition do not match—i.e., “relevance is not the same as utility.” The authors propose MEM-EVOLVER, a memory-level selection-policy framework that does not update model parameters. It consists of three modules: (1) Divergence-Guided Memory Construction, which contrasts successful and failed rollouts produced by the same reasoning strategy on the same task and extracts compact experience cards containing trigger conditions and transfer boundaries; (2) Record-Aware Memory Selection, which semantically recalls top-k candidates at the beginning of an episode, matches each candidate with historical usage records under a similar episode context, and asks an LLM to perform listwise reranking conditioned on the current decision state, falling back to the original retrieval ranking as a conservative prior when evidence is sparse; and (3) Feedback-Driven Evidence Update, which writes each rollout outcome into structured usage records with usage context, thereby continuously enriching the evidence available for subsequent selection. On ALFWorld and ScienceWorld with three backbones (Qwen3-8B/GPT-4o-mini/GPT-4o), MEM-EVOLVER achieves an average success rate of 61.08%, outperforming the strongest memory baseline MemoryBank (55.67%) and no-memory REACT (47.78%). Moreover, the memory bank constructed with Qwen3-8B can be transferred to stronger backbones without regeneration.
Critical strengths
-
The problem is real, and the motivation is quantitatively supported. The gap between semantic top-1 retrieval and candidate usability is a genuine and measurable bottleneck as the memory bank grows: on ALFWorld, top-10 coverage reaches 75.00% while semantic top-1 is only 28.85% (Table 3), indicating that most useful memories can be recalled but are overlooked by semantic ranking. The angle of “improving the selection policy rather than expanding memory storage” is valuable.
-
The abstraction is clean and compatible with existing systems. The paper decouples “what can be reused” (experience cards) from “when reuse is effective, ineffective, or causes negative transfer” (usage records) into two banks, and uses context-matched usage records to support listwise reranking. The approach is entirely no-gradient and leaves the backbone unchanged; it preserves the same retrieve-and-inject interface as Memento/ExpeL/MemoryBank while replacing only the selection stage, which facilitates attribution and makes it easy to layer onto other memory systems.
-
The ablations and diagnostics localize the source of gains. The progressive ablation (Table 2) rules out the alternative explanation that “generic LLM reranking alone explains the gains”—an LLM judge without records recovers only part of the gap. The positive-only variant loses most of the gains on ALFWorld, validating the necessity of negative/boundary evidence. The scale analysis (Fig. 3), which fixes the backbone, memory bank, and retriever and varies only the record budget, shows that the gains come from accumulated usage feedback rather than model changes. The paired analysis (Fig. 4) shows that recoveries significantly outnumber regressions (ALFWorld +21.15% vs. −1.92%).
Critical weaknesses
-
There is no empirical comparison with the most related method. MemRL (Zhang et al., 2026) is closest in spirit to this work—it likewise uses environment feedback for utility-aware memory retrieval. The paper’s core distinction is that “its per-entry Q-values aggregate historical outcomes and do not re-estimate utility with respect to usage context,” but Table 1 does not include MemRL, so this distinction is currently supported only by textual argument. The authors should add this comparison or explain why it is infeasible.
-
The benchmarks are relatively old and limited in scale. Evaluation is restricted to ALFWorld (2021) and ScienceWorld (2022). The motivation emphasizes the selection bottleneck caused by memory-bank growth, but on earlier environments with relatively fixed task distributions, it is questionable whether the memory-bank scale and task diversity are sufficient to expose this bottleneck. It is recommended to validate scalability on more recent interactive agent environments.
-
A simple non-LLM selection baseline is missing. Usage records themselves contain outcome labels and reward differences; a count-based or UCB-style selector that aggregates historical success rates by task family/condition (similar to MemRL’s Q-value aggregation) could exploit these signals. Without such a control, it is impossible to determine the net gain of LLM listwise reranking over simple statistics, and thus impossible to answer whether LLM reranking is truly necessary.
-
Inference overhead is not reported. According to Algorithm 1, each decision step requires one LLM listwise rerank, and the prompt length grows with k cards × m records. Even under the paper’s setting of “using memory only when a failure signal appears,” the additional number of LLM calls and token overhead are not quantified, making it difficult to assess the deployment cost relative to semantic top-1 retrieval.
Questions for authors
-
Please report the per-episode token overhead of MEM-EVOLVER relative to each baseline (the number of reranking calls and prompt length), as well as a sensitivity analysis of the hyperparameters k (number of candidates) and m (number of matched records).
-
Please correct two details: (1) a residual [cite] placeholder remains at Line 182; and (2) Sec. 3.1 states “top-3 matched usage records,” whereas Appendix B (Implementation Details) states “top m = 5.” The experimental setup is inconsistent; please unify it according to the actual experiments.