acceptodds
Desk rejected at ICLR 2027

MEM-EVOLVER: Memory-Level Policy Improvement for LLM Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Experience memory enables LLM agents to reuse past interactions without updating model parameters, but growing memory banks introduce a selection bottleneck: semantically similar memories may fail or cause negative transfer when the current phase, object, tool, destination, or precondition differs. We propose MemEvolver, a no-parameter-update framework that improves the external memory-selection policy rather than simply expanding the memory bank. MemEvolver builds compact experience cards from success–failure divergences, records historical outcomes of memory use as structured usage records, and reranks retrieved candidates using evidence of utility, negative transfer, and transfer boundaries. This converts environmental feedback into reusable evidence about which memories should be trusted under compatible decision states. Experiments on ALFWorld and ScienceWorld show improvements over no-memory agents, representative memory baselines, and semantic retrieval across multiple backbones. Ablations verify the roles of usage records and boundary evidence, while record-accumulation and retrieval-quality analyses show that usage-aware reranking recovers useful memories missed by semantic top-1 retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Experience Memory, Memory Selection, Self-Evolving Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w5yzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the selection bottleneck in external experience memory for LLM agents: as the memory bank grows, memories selected by semantic retrieval may become ineffective or even cause negative transfer when decision conditions such as phase, object, tool, and precondition do not match—i.e., “relevance is not the same as utility.” The authors propose MEM-EVOLVER, a memory-level selection-policy framework that does not update model parameters. It consists of three modules: (1) Divergence-Guided Memory Construction, which contrasts successful and failed rollouts produced by the same reasoning strategy on the same task and extracts compact experience cards containing trigger conditions and transfer boundaries; (2) Record-Aware Memory Selection, which semantically recalls top-k candidates at the beginning of an episode, matches each candidate with historical usage records under a similar episode context, and asks an LLM to perform listwise reranking conditioned on the current decision state, falling back to the original retrieval ranking as a conservative prior when evidence is sparse; and (3) Feedback-Driven Evidence Update, which writes each rollout outcome into structured usage records with usage context, thereby continuously enriching the evidence available for subsequent selection. On ALFWorld and ScienceWorld with three backbones (Qwen3-8B/GPT-4o-mini/GPT-4o), MEM-EVOLVER achieves an average success rate of 61.08%, outperforming the strongest memory baseline MemoryBank (55.67%) and no-memory REACT (47.78%). Moreover, the memory bank constructed with Qwen3-8B can be transferred to stronger backbones without regeneration.

Critical strengths
  1. The problem is real, and the motivation is quantitatively supported. The gap between semantic top-1 retrieval and candidate usability is a genuine and measurable bottleneck as the memory bank grows: on ALFWorld, top-10 coverage reaches 75.00% while semantic top-1 is only 28.85% (Table 3), indicating that most useful memories can be recalled but are overlooked by semantic ranking. The angle of “improving the selection policy rather than expanding memory storage” is valuable.

  2. The abstraction is clean and compatible with existing systems. The paper decouples “what can be reused” (experience cards) from “when reuse is effective, ineffective, or causes negative transfer” (usage records) into two banks, and uses context-matched usage records to support listwise reranking. The approach is entirely no-gradient and leaves the backbone unchanged; it preserves the same retrieve-and-inject interface as Memento/ExpeL/MemoryBank while replacing only the selection stage, which facilitates attribution and makes it easy to layer onto other memory systems.

  3. The ablations and diagnostics localize the source of gains. The progressive ablation (Table 2) rules out the alternative explanation that “generic LLM reranking alone explains the gains”—an LLM judge without records recovers only part of the gap. The positive-only variant loses most of the gains on ALFWorld, validating the necessity of negative/boundary evidence. The scale analysis (Fig. 3), which fixes the backbone, memory bank, and retriever and varies only the record budget, shows that the gains come from accumulated usage feedback rather than model changes. The paired analysis (Fig. 4) shows that recoveries significantly outnumber regressions (ALFWorld +21.15% vs. −1.92%).

Critical weaknesses
  1. There is no empirical comparison with the most related method. MemRL (Zhang et al., 2026) is closest in spirit to this work—it likewise uses environment feedback for utility-aware memory retrieval. The paper’s core distinction is that “its per-entry Q-values aggregate historical outcomes and do not re-estimate utility with respect to usage context,” but Table 1 does not include MemRL, so this distinction is currently supported only by textual argument. The authors should add this comparison or explain why it is infeasible.

  2. The benchmarks are relatively old and limited in scale. Evaluation is restricted to ALFWorld (2021) and ScienceWorld (2022). The motivation emphasizes the selection bottleneck caused by memory-bank growth, but on earlier environments with relatively fixed task distributions, it is questionable whether the memory-bank scale and task diversity are sufficient to expose this bottleneck. It is recommended to validate scalability on more recent interactive agent environments.

  3. A simple non-LLM selection baseline is missing. Usage records themselves contain outcome labels and reward differences; a count-based or UCB-style selector that aggregates historical success rates by task family/condition (similar to MemRL’s Q-value aggregation) could exploit these signals. Without such a control, it is impossible to determine the net gain of LLM listwise reranking over simple statistics, and thus impossible to answer whether LLM reranking is truly necessary.

  4. Inference overhead is not reported. According to Algorithm 1, each decision step requires one LLM listwise rerank, and the prompt length grows with k cards × m records. Even under the paper’s setting of “using memory only when a failure signal appears,” the additional number of LLM calls and token overhead are not quantified, making it difficult to assess the deployment cost relative to semantic top-1 retrieval.

Questions for authors
  1. Please report the per-episode token overhead of MEM-EVOLVER relative to each baseline (the number of reranking calls and prompt length), as well as a sensitivity analysis of the hyperparameters k (number of candidates) and m (number of matched records).

  2. Please correct two details: (1) a residual [cite] placeholder remains at Line 182; and (2) Sec. 3.1 states “top-3 matched usage records,” whereas Appendix B (Implementation Details) states “top m = 5.” The experimental setup is inconsistent; please unify it according to the actual experiments.

Reviewer_uNWaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MEM-EVOLVER, a framework for improving LLM agents through experience-based memory without updating model parameters. The method compares successful and failed trajectories to extract reusable strategies describing what actions to take, under which conditions, and what mistakes to avoid. It then tracks how these strategies perform when reused, recording the task context, outcomes, and reasons for success or failure.

At inference time, the agent retrieves potentially relevant strategies using vector search and reranks them with an LLM based on their historical effectiveness in similar situations.

Experiments on ALFWorld and ScienceWorld show improvements over semantic retrieval and existing memory-based baselines.

Critical strengths
  1. Simple and intuitive method

The approach combines standard components (vector search, LLM-based reranking, and historical feedback) without requiring model fine-tuning. Its simplicity makes it relatively easy to implement and integrate into existing agentic systems, increasing its practical relevance.

  1. Promising empirical results

Consistent improvements over the selected memory-based baselines across two benchmarks (ALFWorld and ScienceWorld) and three LLM backbones. These improvements suggest that the proposed approach might be effective and generalize across different models and environments, strengthening the paper's empirical contribution (however more experiments are required to make this point convincing).

Critical weaknesses

I find this paper stronger than many other submissions I have reviewed during this cycle, and I appreciate the simplicity of the proposed approach and the promising results. However, I believe substantial additional experimental validation is needed to establish its central contribution relative to existing work.

  1. Insufficient comparisons with recent memory methods

The paper compares against Memento, ExpeL, and MemoryBank but omits several relevant recent approaches, particularly MemRL, ReasoningBank, and Mem0. MemRL is especially important, as it also combines semantic retrieval with feedback-driven utility estimation, making it a closely related alternative to the proposed method.

Moreover, MemRL reports 94.9% success on ALFWorld using GPT-5-mini, compared with the best reported MEM-EVOLVER result of 79.85% using Qwen3-8B. Although these numbers are not directly comparable because of different backbones and potentially different evaluation settings, they suggest that the reported performance is not competitive with stronger contemporary systems. A controlled comparison using the same backbones and evaluation protocol is therefore essential to establish the proposed method's empirical contribution.

  1. Limited experimental validation and insufficient analysis of cross-model transfer.

The experimental evaluation is relatively narrow. All experience memories and usage records are constructed using Qwen3-8B, while inference is evaluated only on Qwen3-8B, GPT-4o-mini, and GPT-4o. These backbones are insufficient to establish the effectiveness of the approach with more recent and stronger LLMs.

In particular, the paper lacks experiments disentangling the effects of the model used to construct memories from the model that consumes them. I would like to see memory banks generated by several different models (ideally 2–3 additional backbones) and a cross-model transfer matrix evaluating how each bank performs with different inference backbones. This is important for establishing whether the proposed method captures generalizable experience rather than model-specific behaviors.

The evaluation should also extend beyond ALFWorld and ScienceWorld to more diverse and challenging benchmarks, such as BigCodeBench, Lifelong Agent Bench, and Humanity's Last Exam (HLE), which have already been used to evaluate related memory methods.

  1. Unclear efficiency trade-offs.

Furthermore, MEM-EVOLVER introduces additional LLM-based reranking and memory-management operations, but the paper does not provide a clear analysis of their computational cost, latency, or token overhead. Since the method primarily combines existing components, demonstrating that its performance improvements justify these additional costs is important for assessing its practical contribution.

Minor:

There is a literal [cite] placeholder in Section 2.3, page 4.

Questions for authors
  1. What are the confidence intervals of the results?

  2. Could the authors provide independent ablations for the individual components of the full selector, including robust judging, parse-valid filtering, transfer-boundary checks, and the semantic-rank prior? Currently, it is difficult to determine which component accounts for the final improvement beyond the single-judge variant.

  3. In the positive-only ablation, are all other components, prompts, and inference budgets kept identical to the full method? Does removing negative records also change the total number of retrieved records?

  4. Could the authors clarify how Recall@10 coverage is computed? Does measuring whether a candidate memory is useful require executing an additional rollout with each of the top-10 memories, and if so, how are those rollouts controlled for stochasticity?

  5. How many exploratory rollouts are generated per training task, and what fraction of tasks yield a valid success–failure pair? How are the successful and failed trajectories selected when multiple candidates are available?

  6. Could the authors compare the current contrastive memory construction approach against memories extracted from successful trajectories only and failed trajectories only, keeping the memory budget and inference procedure fixed? This would help establish whether success–failure contrast is necessary for the reported improvements.

  7. Equations (8)–(9) suggest that reliable feedback may require both memory-conditioned and baseline continuation rollouts. Could the authors clarify when these paired rollouts are actually executed, how frequently they are required, and their computational overhead? In particular, can the proposed method accumulate reliable usage records from ordinary agent interactions without additional counterfactual rollouts?

  8. The framework relies on several complex, manually designed prompts for experience extraction, query generation, reranking, feedback recording, and ReAct-style reasoning. Could the authors provide prompt-sensitivity experiments or comparisons against simpler prompt variants? In particular, how much of the improvement comes from the proposed memory mechanism versus the extensive hand-crafted instructions and heuristics in the reranking prompt?

  9. The method largely combines established components, including experience extraction, vector retrieval, LLM reranking, and historical feedback. The main distinguishing feature appears to be the use of context-matched historical records rather than aggregated per-memory utility scores. Could the authors provide a controlled comparison against MemRL and simpler alternatives, such as reranking based on aggregated success rates or scalar utility scores, to establish the necessity of the proposed structured-record design?

  10. The experience bank and usage records are constructed exclusively with Qwen3-8B. Could the authors evaluate memory banks generated by several different backbones and report a cross-model transfer matrix? This would help disentangle the quality of generated memories from the ability of the inference model to use them.

  11. The evaluation covers only Qwen3-8B, GPT-4o-mini, and GPT-4o. Could the authors evaluate more recent and stronger LLMs, including models with substantially better baseline reasoning performance? It is currently unclear whether external memory provides comparable benefits for stronger agents or whether its utility diminishes as backbone capabilities improve.

  12. Could the authors provide an end-to-end cost analysis covering experience-bank construction, additional rollouts, query generation, LLM reranking, and feedback-record generation? Comparisons at matched inference budgets would help determine whether the proposed memory mechanism offers an advantage over allocating equivalent compute to stronger reasoning or additional agent rollouts.

Reviewer_Jv4qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the memory selection bottleneck in LLM agents where semantically similar past experiences may cause negative transfer due to mismatches in task phase, object, tool, or preconditions. The authors propose MEM-EVOLVER, a no-gradient framework that improves the external memory selection policy with a frozen backbone. It consists of three modules: first, divergence-guided memory construction which extracts compact experience cards by contrasting successful and failed trajectories; second, record-aware memory selection which combines episode-start semantic recall, historical usage record matching, and LLM listwise reranking to select contextually useful experiences; and third, feedback-driven evidence update which logs rollout outcomes and transfer boundaries into structured usage records for future selections. Experiments on ALFWorld and ScienceWorld across multiple backbones show consistent gains over no-memory and semantic-retrieval memory baselines.

Critical strengths

First, well-motivated decoupled design. The paper clearly identifies that semantic relevance does not equal behavioral utility. Decoupling static reusable guidance from dynamic contextual evidence and specifically tracking negative transfer and transfer boundaries provides a simple yet effective non-parametric mechanism to prevent agents from over-following mismatched memories. Second, strong empirical gains and cross-backbone generalization. The method achieves consistent improvements across two interactive environments and three backbones. Notably, experience banks and usage records constructed solely by a smaller open-weight model transfer effectively to boost stronger proprietary models at inference time. Ablations also convincingly prove the necessity of negative and boundary records.

Critical weaknesses

Listwise reranking requires feeding top-10 candidate cards and multiple matched usage records into long LLM prompts, yet the paper reports no token costs, latency overhead, or the total rollout budget needed to collect thousands of usage records. Moreover, generating the query and candidate set only once at episode initialization is restrictive for long-horizon, partially observable tasks where crucial objects or states only appear mid-episode.

Questions for authors

(1) Could you kindly clarify how the failure signals that trigger memory lookup at test time are formally detected? Additionally, we observed a substantial performance gain in Table 2 when moving from the single judge variant to the full model, so could you elaborate on the exact implementation of robust frozen judgments and parse-valid gating, such as whether it involves multi-sample voting? (2) Regarding the computational cost, what is the average per-episode token cost and latency overhead at inference time when compared to standard semantic retrieval baselines? Also, what is the total rollout budget required during training to build the experience bank and collect usage records? (3) Concerning hyperparameters and static recall, was top-3 or top-5 used for matched usage records in the main experiments? Moreover, in partially observable tasks, does relying solely on episode-initial queries for candidate recall ever lead to missing necessary mid-episode experiences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.