acceptodds
Desk rejected at ICLR 2027

MemEff: Enhancing Self-Evolving Agents through Behavioral Efficacy-Guided Memory Optimization

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Learning from past experiences to optimize the memory of large language models (LLMs) has emerged as a key paradigm for autonomous agents handling interactive real-world tasks. However, existing studies focus on leveraging external feedback to improve memory utility, overlooking the influence of memory on LLM agent behavior. This oversight makes it difficult to measure memory's contribution to task performance, leading to suboptimal memory guidance and hindering the further improvement of LLM agents. To bridge this gap, we propose , a runtime learning framework that explicitly integrates the behavioral efficacy of accumulated memory into the self-evolution process of LLM-based agents. MemEff quantifies behavioral efficacy by measuring the difference in action generation probabilities between agents with and without memory guidance. By combining this measure with environmental feedback, MemEff estimates the potential performance gains of memories from historical interactions and identifies high-utility memories to guide LLM agents. Extensive experiments demonstrate that MemEff enhances the utility of retrieved memories, outperforming state-of-the-art baselines in task completion across multiple benchmarks. These results highlight the importance of behavioral efficacy in memory optimization and continual self-evolution of LLM agents. Codes and data are available at https://anonymous.4open.science/r/MemEff-75CD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent Memory, Self-Evolving Agent, Behavioral Efficacy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ro3kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MemEff improves LLM agents with frozen parameters through memory optimization. It measures behavioral efficacy using the difference in action log probabilities with and without memory, then combines this signal with historical reward gains to select and retrieve memories. Across five online rounds on three interactive benchmarks with two execution models, MemEff outperforms the evaluated baselines on average.

Critical strengths
  1. The method combines changes in action probabilities with historical reward gains for memory selection and retrieval without updating the execution model's parameters.

  2. Table 1 reports average performance gains across three task domains and two execution models. With GPT-5.1, MemEff exceeds the strongest reported baseline on WebArena-Lite by 7.50 percentage points.

Critical weaknesses
  1. Changing behavior does not necessarily correct errors. After a failure, Equation 7 selects the memory that most reduces the probability of the original trajectory, while Section 5.5 states that only the first three steps are scored. If the error occurs later, this rule may favor changing a correct prefix. Table 2 measures trajectory deviation but does not verify whether the errors are corrected.

  2. The effect of the behavioral efficacy kernel on task performance lacks an ablation. Table 3 compares the full retrieval strategy with semantic retrieval and MemRL but does not isolate the contribution of the behavioral efficacy kernel. Task success rates should be compared with and without this kernel while keeping the candidate pool and other components fixed.

Questions for authors
  1. Do the patterns in Figures 1 and 2 hold after controlling for task difficulty and comparing candidate memories for the same task? Specifically, do memories with high rewards but little behavioral impact yield smaller task gains than other candidates?

  2. When a failure occurs after the first three steps, how does Equation 7 distinguish changing a correct prefix from correcting the actual error? Can the selection rule be evaluated using subsequent execution outcomes rather than only trajectory similarity?

  3. How much does task performance decline when the predictor using both kernels is replaced with the instruction similarity kernel alone, keeping the candidate pool, memory generation, reward gain objective, and fallback rules fixed?

Reviewer_YN3AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes MemEff, a memory-based LLM policy, the memory is measured and selected based on its behavioral efficacy.

Critical strengths

This work introduces the behavioral efficacy of memory, creating an explicit measurement of memory quality and informativeness.

Critical weaknesses
  1. This work measures behavioral efficacy as the difference in action log-probabilities between memory-free and memory-augmented agents, and uses reward differences to estimate memory utility. However, in long-term settings, reward feedback may be delayed, and LLM agents introduce significant uncertainty during multi-step reasoning. Moreover, an agent may receive multiple memory items as input, raising the question of how to measure their individual contributions and mutual influence on a single action decision.

Therefore, the behavior efficacy concept is good for measuring its impact on performance, but the metric design and practical computation are worth careful consideration.

  1. The preliminaries assume there exists a function mapping between reward difference and behavioral efficacy (log action probability gap). Why does this map exist? Is this assumption too strong? And in Eq. 6, behavioral efficacy is the log action probability gap with and without memory, but for the same action, the reward will be the same. How, then, can the map (f) in Eq. 1 be established?

  2. When calculating Eq. 6, how is the action selected to measure behavioral efficacy? Is it obtained with or without memory? The LLM is sensitive to input, and the output is free-form, so the probability of an action in its exact form in ALFWorld and ScienceWorld may be quite small both with and without memory. Therefore, I think the setting for behavioral efficacy is inflexible and not generally practical.

Questions for authors

In abs, the author says the behavioral efficacy "measuring the difference in action generation log-probabilities between agents with and without memory guidance", but in preliminary, the behavioral efficacy is the reward difference between with/without memory. It looks like a mismatch?

Fig 1,2 is difficult for me to understand. What is relative behavioral efficacy, in Fig 1, it is a negative percentage, but fig 2 shows percentage up to 175%.

Reviewer_s1vBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a memory framework named MemEff. MemEff measures the “behavioral efficacy” of memory through changes in action-generation log-probabilities with and without memory guidance, and combines this measure with historical reward gains to optimize memory generation and selection. Experiments on three interactive benchmarks demonstrate strong memory capabilities, with the method achieving the best performance in most settings.

Critical strengths
  1. The paper has a clear motivation, as memory construction is an important direction for improving LLM agents. The proposed optimization strategy is also novel.

  2. The paper is well structured, and the method is clearly described and easy to follow. The experimental section also includes well-designed ablation studies.

Critical weaknesses
  1. MemEff expands its memory mainly by adding new entries and updates memory scoring based on usage history. While this provides a mechanism for memory management and utilization, it lacks an explicit mechanism for revising or repairing existing memories, which may lead to performance degradation over prolonged interactions.

  2. For failed trajectories, the authors select the candidate memory with the lowest behavioral efficacy. However, the method treats each trajectory as a whole and does not explore finer-grained optimization, such as analyzing and optimizing different stages separately.

Questions for authors

Please refer to the weaknesses part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.