acceptodds
Under review as a conference paper at ICLR 2027

Mem- : Adaptive Memory through Learning When and What to Generate

Abstract

We present Mem-, a framework for adaptive memory in large language model (LLM) agents, where useful guidance is generated on demand rather than retrieved from external memory stores. Existing memory-augmented agents typically rely on similarity-based retrieval from episodic memory banks or skill libraries, returning static entries that often misalign with the current context. In contrast, Mem- uses a dedicated language or vision-language model with its own parameters, separate from the downstream agent, to generate context-specific guidance for complex tasks. Conditioned on the current agent context, the model jointly decides when to produce guidance and what guidance to produce. We train it with a decision-content decoupled reinforcement learning (RL) objective, enabling it to abstain when generation would not help and otherwise produce concise, useful guidance. Across diverse agentic benchmarks spanning web navigation, terminal-based tool use, and text-based embodied interaction, Mem- consistently outperforms retrieval-based and prior RL-optimized memory baselines, achieving over 30% relative improvement on web navigation tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
generative memory, agent memory, llm memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_iyyEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Mem-π represents reusable experience as a separate guidance-generating model for a frozen downstream agent. Supervised experience distillation first learns from training-split hints, followed by reinforcement learning with one forced abstention branch and three generated-guidance branches. Decision tokens receive a cross-branch utility comparison, while content tokens receive within-generation advantages gated by whether generation is beneficial. The memory policy is called once before the agent’s first action. Experiments cover WebArena, WorkArena, ALFWorld and LifelongAgentBench, with the main evaluation transferring from a training-time Qwen agent to GPT-5.4-mini. The reported average SR increases from 49.4 for MemRL to 55.2 for Mem-π.

Critical strengths
  1. The work provides a concrete mechanism for deciding whether guidance should be delivered, rather than assuming that more memory always helps. The separation between a trainable memory policy and a frozen receiving agent is also practically useful for transfer to a different agent.

  2. The experimental support is substantial: the paper reports held-out task evaluation across four benchmarks, three-seed means and standard deviations, and verified environment outcomes rather than relying on an LLM judge for the primary success metric. The gains are present on every benchmark, not just in an aggregate dominated by one setting.

  3. The study includes unusually useful controls and diagnostics: a zero-shot hinter, a no-Stage-1 variant, learned abstention for retrieved hints, forced generation, leave-one-domain-out evaluation, matched bank perturbations and cross-agent routing evaluation. The latter uses separate rollouts for selecting and evaluating a per-task branch, which avoids evaluating that selector on its selection samples.

Critical weaknesses
  1. The specific benefit of the decision-content objective is not cleanly isolated. The “without structured rollout” comparison reverts to vanilla GRPO, changing both branch coverage and advantage assignment. The “without delta-gating” variant uses a naive sum of decision and content advantages, rather than only removing the gate while preserving token routing. These comparisons support the full recipe but do not identify which part of the proposed decomposition matters. Moreover, Equation (7) retains per-sequence length normalization: a one-token abstention decision and the decision prefix of a long generated sequence can receive very different effective weights. The exact masking/normalization and old-policy probabilities for forced decision tokens should be specified to make the implemented surrogate reproducible.

  2. The strongest evidence is against the specified shared-bank baselines, not yet against query-adaptive guidance alternatives. The RAG baseline uses a verbatim BM25 top-1 hint, and the examples in Figure 5 emphasize copying an incorrect count or overly narrow source constraint. A same-budget query-conditioned rewriter of retrieved hints would directly test whether parametric internalization is needed to correct those failures. Table 5 also identifies learned generative guides as close related approaches, but their comparison remains descriptive. The zero-shot hinter and RAG-with-abstention controls are valuable, yet neither is equivalent to retrieval followed by adaptive rewriting.

  3. The actual input to the main text-only memory policy is unclear. Section 2 defines its context as task plus observation, and Appendix C.8 includes accessibility-tree tokenization/prefill in the memory-call cost. However, the text-only memory prompt in Appendix C.12 contains only the task description, without an observation placeholder. This distinction matters for the claim of context-specific guidance and for reproducibility. The policy input, observation truncation and available information should be made consistent across the method, prompt and cost descriptions.

Questions for authors
  1. How does the method compare with the same structured branches and a single sequence-level advantage, and with the same decoupled objective but only the content gate removed? Please specify the exact decision-token normalization and likelihood treatment under forced branching.

  2. Can a matched retrieval-plus-rewriting control, using the same hint corpus and guidance budget, close the gap? This is especially relevant to the count/constraint correction examples used to motivate generated memory.

  3. Does the main text-only memory policy receive the initial environment observation or only the task? Please provide the exact input template and truncation settings that produced Table 1 and the deployment-cost measurements.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer iyyE (Part 1/3)

We thank the reviewer for recognizing the concrete mechanism for deciding whether guidance is delivered and the verified held-out evaluation, and address below the isolation of the decision-content objective and the matched query-adaptive baselines.

W1a and Q1a. Separating branch coverage, token credit assignment, and content gating.

The "w/o structured rollout" ablation removes only the forced [ABSTAIN] branch and keeps separate advantages for decision and guidance tokens. Calling it "vanilla GRPO" in Section 3.2 was imprecise.

Under Eqs. 5 and 6, the decision token learns from Δ=Vabs−Vgen\Delta=V_{\mathrm{abs}}-V_{\mathrm{gen}}, while guidance tokens learn from reward differences among generated hints only when Δ<0\Delta<0. Stage 2 starts from a policy that favors generation. Without the forced branch, most independently sampled groups contain only [GENERATE] outputs and have no cross-branch decision signal. Training on those groups effectively reduces to GRPO over generated hints, which explains the "vanilla GRPO" description in Section 3.2. Token routing remains in place, but independent sampling cannot ensure that both decisions appear in every group.

To further demonstrate how coverage and routing each contribute, following the reviewer's suggestion, we increased the unstructured rollout budget and evaluated the two requested controls. All variants used the same training-split hint bank, frozen training-time agent, and held-out evaluations. Control (a) uses structured branches with a single sequence-level GRPO advantage for all tokens. Control (b) retains structured branches and token routing but removes only the content gate by setting it to one. Mixed groups (%) is the fraction of training groups containing both decisions, averaged over the final 50 steps.

VariantRollouts per taskMixed groups (%)WEBARENA SR (%)ALFWORLD SR (%)
w/o structured rollout (Table 2)42438.387.1
w/o structured rollout84139.688.0
w/o structured rollout166340.488.5
(a) Structured, sequence-level advantage410039.788.2
(b) Structured, routed, no content gate410042.390.6
Original "w/o Δ\Delta-gating" (naive sum)410041.389.6
Mem-π\pi (full)410043.191.4

Both branch coverage and token routing contribute to the gain, while the content gate adds little beyond them. Even with 16 unstructured rollouts, WEBARENA success remains 2.7 points below full Mem-π\pi, whereas four structured rollouts cover both decisions in every group with one quarter as many rollouts. With coverage fixed, control (b) exceeds control (a) by 2.6 points on WEBARENA, isolating the benefit of routing the two advantages to their respective tokens.

A likely explanation is that a single sequence-level advantage penalizes the decision to generate when an individual hint fails, even when generation helps on average. Restoring the content gate adds a further 0.8 points on each benchmark.

We will correct the "w/o structured rollout" description in Section 3.2, add these controls to Table 2, and report mixed-group and gate-activation rates over training in Appendix C.

AuthorsOpenReview ↗

Author Response to Reviewer iyyE (Part 2/3)

W1b and Q1b. Normalization and likelihood of the decision token.

We would clarify that the decision token is a special token and is not counted in the length normalization, which applies only to the ordinary guidance tokens of the [GENERATE] rollouts.

Each rollout starts with [ABSTAIN] or [GENERATE], receiving only the decision advantage +Δ+\Delta or −Δ-\Delta, where Δ=Vabs−Vgen\Delta=V_{\mathrm{abs}}-V_{\mathrm{gen}} (Eqs. 5 and 6), while guidance tokens receive only the content advantage. Eq. 7 uses the compact notation of Eq. 2 without showing this separation. Written explicitly, rollout jj contributes ℓ1j+1∣mj∣∑t=21+∣mj∣ℓtj\ell^j_1+\frac{1}{\lvert m^j\rvert}\sum_{t=2}^{1+\lvert m^j\rvert}\ell^j_t inside the 1/G1/G sum, where ℓtj\ell^j_t is the per-token term of Eq. 7 and the second term is absent for [ABSTAIN]. Each decision token thus has weight 11, whether it stands alone or precedes a long hint, while guidance tokens are averaged over hint length.

Eq. 4 forces one [ABSTAIN] token and three [GENERATE] tokens, but none is masked from the loss. Their likelihood ratios use the probability of the forced token, ρ1j=πθ(dj∣x)/πold(dj∣x)\rho^j_1=\pi_\theta(d^j\mid x)/\pi_{\mathrm{old}}(d^j\mid x), where πold\pi_{\mathrm{old}} is the frozen policy snapshot in Eq. 2, so forcing a branch does not set its old-policy probability to one. Clipping and the token-level KL term apply to both decision and guidance tokens.

As a worked example, consider Eqs. 3 to 6 with λlen=0.1\lambda_{\mathrm{len}}=0.1 and Lmax⁡=256L_{\max}=256. Suppose the agent fails without guidance, succeeds with hints of 120 and 80 tokens, and fails with a 200-token hint.

Rollout∣mj∣\lvert m^j\rvertTaskRewardRmR_mRRDecision adv.Decision-token weightContent adv.Weight per guidance token
[ABSTAIN], j=1j=100none0−0.615-0.61510none
[GENERATE], j=2j=21201−0.047-0.0470.953+0.615+0.6151+0.56+0.561/120
[GENERATE], j=3j=3801−0.031-0.0310.969+0.615+0.6151+0.59+0.591/80
[GENERATE], j=4j=42000−0.078-0.078−0.078-0.078+0.615+0.6151−1.15-1.151/200

Here Vgen=0.615V_{\mathrm{gen}}=0.615 and Δ=−0.615\Delta=-0.615, so the content gate opens. The content advantages use the sample standard deviation of the three [GENERATE] rewards. The table shows advantages before the common 1/G1/G factor, clipping, and the KL term. All four decision tokens have weight 11, while only guidance-token weights depend on hint length. The negative [ABSTAIN] advantage and positive [GENERATE] advantages both favor generation.

W2a and Q2. Query-time rewriting and bank updates.

The SkillRL [1] baseline we already compare against is the retrieve-and-rewrite control the reviewer describes, with a bank that keeps evolving. It retrieves from the same shared bank and rewrites stored entries from the frozen agent's training outcomes as training proceeds. On top of it, we added a reranker and a per-query rewriter after SkillRL's retrieval step.

Both controls started from the same training-split JEF-Hinter corpus as the main comparison and used the same 256-token guidance budget and three-seed held-out evaluation with GPT-5.4-mini. The added variant reranked the retrieved entries and used a 7B model to rewrite them for the current query.

MethodWEBARENA SR (%)Four-benchmark avg. SR (%)
Base Agent27.145.1
RAG31.446.9
Mem031.947.8
Memory-R133.248.8
MemRL34.049.4
SkillRL33.749.8
SkillRL + rerank and rewrite34.449.6
Mem-π\pi (Stage 1)35.051.0
Mem-π\pi43.155.2

Both controls improved on RAG but scored below Stage 1 of Mem-π\pi. With or without query-time rewriting, SkillRL can draw on only the entries it retrieves from the bank for the current task. Mem-π\pi generates guidance from experience distilled across the bank and learns to withhold it when it would not help. Both SkillRL variants supply a hint for every task.

Table 1 also compares Mem-π\pi with retrieval methods beyond plain BM25. Memory-R1 [2] learns to manage stored entries, MemRL [3] learns which entries are useful for the current task, and Mem0 [4] uses rules to manage its bank. All three score below Mem-π\pi. In Table 3, RAG with the same learned abstention objective reached 35.5% on WEBARENA, compared with 43.1% for Mem-π\pi.

AuthorsOpenReview ↗

Author Response to Reviewer iyyE (Part 3/3)

W2b. Comparison with learned generative guides.

Unlike Advisor Models [5] and SEAM [6], Mem-π\pi distills domain-specific workflow experience into its generator before RL on task outcomes and, more importantly, learns when to abstain.

Both guides start GRPO from the backbone without prior guidance distillation, as described in their papers, and supply guidance for every task, even when the agent can solve it alone. The submission identifies this lack of abstention as a gap in generative guidance. Mem-π\pi uses supervised fine-tuning on training-split JEF-Hinter hints in Stage 1 and reaches 35.0% on held-out WEBARENA tasks, ahead of Memory-R1 and MemRL in Table 1. Stage 2 adds 8.1 points on WEBARENA with GPT-5.4-mini at evaluation, even though its training rewards came from a frozen Qwen2.5-7B-Instruct agent. Comparing that agent's outcomes with and without guidance on the same task teaches Mem-π\pi when to withhold a hint.

The submitted ablations separate these two choices. Starting Stage 2 directly from Qwen2.5-7B-Instruct without Stage 1 lowers success by 5.2 points on WEBARENA and 4.5 on ALFWORLD. Forcing the trained policy to generate on every test task lowers WEBARENA success from 43.1% to 39.8%.

We also evaluated Advisor Models and SEAM directly. Following their respective training procedures, we trained each 7B guidance generator with GRPO on the frozen agent's outcomes. Both used the shared bank, the same 256-token guidance budget, and the same downstream agent and evaluation as in W2a.

MethodWEBARENA SR (%)Four-benchmark avg. SR (%)
Advisor Models33.849.1
SEAM34.850.2
Mem-π\pi (Stage 1)35.051.0
Mem-π\pi43.155.2

Starting from the backbone without Stage 1, Mem-π\pi with learned abstention already outperforms both guides on WEBARENA. It reaches 37.9%, 3.1 points above SEAM at 34.8%. The trained policy also beats SEAM by 5.0 points when forced to generate. SEAM performs about as well as Stage 1 distillation alone, and the full policy scores highest by combining distillation and learned abstention.

W3 and Q3. Correcting the text-only memory input template and observation budget.

Thank you for catching the missing observation field in Appendix C.12. The text-only Mem-π\pi receives both the task and the initial observation, which together form the input x=(q,o)x=(q,o) in Section 2.

For WEBARENA and WORKARENA, the observation is the initial accessibility tree, truncated to its first 1,024 tokens. For ALFWORLD, it is the initial room description. For LAB, it is the initial shell or database state. The vision-language variant in Appendix C.9 receives the initial screenshot and screen description instead. The corrected text-only template adds the missing field.

[system]
You are a hinter. Given a web task, write one concise, actionable hint that helps the agent take its next step, or abstain when you cannot help.
[user]
Task:
{task_description}
Initial observation: {initial_observation}
First decide: output [ABSTAIN] if you cannot provide useful guidance, or [GENERATE] followed by a concise, actionable hint.

The latency measurement in Appendix C.8 used the same input, and the tokenization and prefill of the task and initial accessibility tree account for 0.7 s of the reported 1.8 s memory call. Table 1 used this input as well.

We will correct the prompt in Appendix C.12 and state the 1,024-token truncation and the single-call timing consistently across the paper.

[1] P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, et al. SkillRL: Evolving agents via recursive skill-augmented reinforcement learning. arXiv:2602.08234, 2026.

[2] S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, et al. Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv:2508.19828, 2025.

[3] S. Zhang, J. Wang, R. Zhou, J. Liao, Y. Feng, Z. Li, Y. Zheng, W. Zhang, Y. Wen, Z. Li, et al. MemRL: Self-evolving agents via runtime reinforcement learning on episodic memory. arXiv:2601.03192, 2026.

[4] P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv:2504.19413, 2025.

[5] P. Asawa, A. Zhu, A. O'Neill, M. Zaharia, A. G. Dimakis, and J. E. Gonzalez. How to train your advisor: Steering black-box LLMs with advisor models. arXiv:2510.02453, 2025.

[6] X. Li, H. Li, Y. Zhou, Y. Liu, and Q. Ai. Beyond experience retrieval: Learning to generate utility-optimized structured experience for frozen LLMs. arXiv:2602.02556, 2026.

Reviewer_7Hu6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Mem-π\pi, a separate generative memory policy for frozen or closed downstream agents. It first distills procedural hints through supervised learning, then uses outcome-based reinforcement learning to learn when to generate or abstain and what guidance to provide. Experiments on WebArena, WorkArena, ALFWorld, and LifelongAgentBench show consistent improvements over the considered baselines, supported by ablations and cross-agent transfer analyses.

Critical strengths
  1. The paper studies a clear and practically relevant limitation of retrieval-based memory: a retrieved entry may be similar to the current task but contain mismatched details, and memory can be harmful when the base agent does not need it. The separation between learning when to intervene and what to generate is intuitive and compatible with frozen or closed agents.

  2. The empirical study is broad and generally well organized. Mem-π\pi improves performance across four interactive benchmarks, and the ablations support the contributions of experience distillation, structured abstain/generate rollouts, and decision-content credit assignment. The transfer from the training-time Qwen2.5-7B agent to GPT-5.4-mini is also encouraging.

Critical weaknesses
  1. The evaluation gives convincing evidence on the selected benchmarks, but stronger evidence for generalization would be helpful. Most main evaluations use held-out instances within related task families or skill sets. The WebArena leave-one-domain-out result is encouraging, but an evaluation with a larger train-test distribution shift, or a more realistic multi-session setting in which experience from earlier episodes is useful in later episodes, would better establish the intended scope of the method.

  2. The baseline suite does not include an empirical comparison with the closest generative/advisor methods discussed in Table 5. In addition, the main RAG baseline uses BM25 top-1 retrieval. A stronger retrieval-generation baseline that retrieves, reranks, and rewrites or aggregates memories with a comparable generator budget would make it easier to determine whether the gains specifically come from parametric memory rather than contextual rewriting.

  3. The memory-policy backbone is fixed to Qwen2.5-7B-Instruct. The cross-agent experiments vary the downstream agent, but do not test sensitivity to the model used as the memory generator. It is therefore unclear how much the results depend on this particular initialization and model capacity.

Questions for authors
  1. Could the authors provide results on a benchmark or split with a larger train-test distribution shift, or on a more realistic multi-session agent setting? If this is outside the intended scope, could the authors clarify that Mem-π\pi is primarily designed for offline procedural transfer within related task families?

  2. Could the authors compare against at least one close generative/advisor baseline from Table 5, or a stronger retrieve-rerank-rewrite baseline using a comparable generator budget? How would such a hybrid compare with Mem-π\pi?

  3. How sensitive is Mem-π\pi to the choice of memory-generator backbone? A comparison using a weaker Qwen2 model and a stronger Qwen3 model would help show whether the proposed training objective remains effective across different generator capabilities.

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer 7Hu6 (Part 1/3)

We thank the reviewer for recognizing the separation between when to intervene and what to generate for frozen or closed agents and the breadth of the evaluation, and address below the scope of generalization and the strength of the baseline comparisons.

W1 and Q1. Transfer to unseen environments and learning from earlier sessions.

ALFWorld tests a larger environmental shift than leaving out one WebArena domain. Mem-π\pi outperforms RAG in household rooms absent from training.

Across the 134 validation-unseen tasks, Mem-π\pi achieved 91.4% task success, compared with 85.5% for RAG. It is distilled only from 3,553 training tasks (Appendix C.1), with no task, trajectory, or hint from any of the validation-unseen rooms. Each room has its own layout and object placements, so a retrieved hint from a similar training task may refer to receptacles absent from the new room. Mem-π\pi generates guidance for the current room and transfers the procedure, such as where to search first or how to heat, cool, or clean an object before placing it. RAG instead returns the nearest training hint, which can retain mismatched details. Our intended scope is the transfer of procedural experience within related task families to unseen environments and episodes.

To further demonstrate learning from earlier sessions, following the reviewer's suggestion, we ran a four-session WebArena experience loop in which each session used a new batch of tasks disjoint from all earlier batches. After each session, JEF-Hinter [1] converted environment-verified successful episodes into hints. Failed episodes contributed contrastive hints only when a successful episode from the same template was available. We deduplicated hints by template and checked them against their source trajectories. Before the next session, every memory method received the expanded bank. Retrieval baselines re-indexed it, while Mem-π\pi reran Stage 1 on new hints and Stage 2 on new tasks with a fixed budget per session. The downstream agent remained frozen throughout, so any gain comes from the memory. We report task success rate (%) by session.

MethodSession 1Session 2Session 3Session 4
Base Agent27.426.827.327.0
RAG29.130.831.932.6
Mem029.631.432.733.5
Memory-R130.232.033.634.4
MemRL30.932.834.535.3
Mem-π\pi (Stage 1)32.335.036.938.2
Mem-π\pi34.738.941.843.6

Experience from earlier sessions improves Mem-π\pi faster than retrieval-based memory. From Session 1 to Session 4, Mem-π\pi gains 8.9 points, compared with 4.4 for MemRL on the same expanded bank, while the Base Agent stays nearly flat. Retrieval methods re-index new experience, whereas Mem-π\pi retrains on it. The loop therefore compares how each method would be updated in deployment, not bank growth alone.

W2a and Q2. Retrieval with query-time rewriting or bank updates.

The SkillRL [2] baseline we already compare against is the stronger retrieve-and-rewrite baseline the reviewer asks for, with a bank that keeps evolving. It retrieves from the same shared bank and rewrites stored entries from the frozen agent's training outcomes as training proceeds.

To add the reranking and query-time rewriting the reviewer also mentions, we put a reranker and a per-query rewriter after SkillRL's retrieval step. Both variants started from the same training-split JEF-Hinter bank and used a 7B generator, a 256-token guidance budget, and the same frozen downstream agent. We evaluated both over three seeds and report WebArena task success and the unweighted mean across the four benchmarks.

MethodWebArena SR (%)Four-benchmark avg. SR (%)
RAG31.446.9
Mem031.947.8
Memory-R133.248.8
MemRL34.049.4
SkillRL33.749.8
SkillRL + rerank and rewrite34.449.6
Mem-π\pi (Stage 1)35.051.0
Mem-π\pi43.155.2

Both variants improved on RAG, but neither could draw on bank entries it did not retrieve. On WebArena, SkillRL's evolving bank gained 2.3 points over RAG, and adding reranking and per-query rewriting brought the gain to 3.0 points. Mem-π\pi generates guidance from experience distilled across the whole bank and can withhold it when the agent does not need it, while both variants guide every task.

Table 1 also compares Mem-π\pi with rule-based Mem0 [3] and the RL-trained Memory-R1 [4] and MemRL [5]. MemRL is the best of those three on WebArena, and Mem-π\pi leads it by 9.1 points.

AuthorsOpenReview ↗

Author Response to Reviewer 7Hu6 (Part 2/3)

W2b. Comparison with generative advisors.

Mem-π\pi differs from Advisor Models [6] and SEAM [7] by distilling workflow experience before RL and, more importantly, learning when to abstain.

Both advisors start GRPO from a pretrained backbone without first distilling the hint bank. They guide every task, including tasks the agent can solve on its own (Section 1). Mem-π\pi uses supervised fine-tuning on training-split JEF-Hinter hints, and Stage 1 alone outperforms Memory-R1 and MemRL on held-out WebArena tasks. Stage 2 adds 8.1 points on WebArena with GPT-5.4-mini even though the training rewards came from a frozen Qwen2.5-7B-Instruct agent. During training, Mem-π\pi compares that agent's outcomes with and without guidance on the same task to learn when to withhold guidance.

The submitted ablations test these design choices. Running Stage 2 directly on Qwen2.5-7B-Instruct without Stage 1 costs 5.2 points on WebArena and 4.5 on ALFWorld. Forcing the trained policy to generate on every test task also lowers WebArena success. At the reviewer's suggestion, we evaluated both advisors directly. Each used a 7B generator trained with GRPO on the frozen agent's outcomes, as in its original method. We used the same bank, 256-token guidance budget, downstream agent, and evaluation as in W2a.

MethodWebArena SR (%)Four-benchmark avg. SR (%)
Advisor Models33.849.1
SEAM34.850.2
Mem-π\pi (Stage 1)35.051.0
Mem-π\pi43.155.2

Even from the same starting point as the advisors, without Stage 1, Mem-π\pi learns to abstain and outperforms both. It reaches 37.9% on WebArena (Table 2), 3.1 points above SEAM. With distillation but with [GENERATE] forced at test time, the trained policy still leads SEAM by 5.0 points (39.8%, Table 3). SEAM scores close to Stage 1, while the full policy combines distillation and learned abstention and achieves the highest success rate.

AuthorsOpenReview ↗

Author Response to Reviewer 7Hu6 (Part 3/3)

W3 and Q3. Sensitivity to the memory-generator backbone.

Mem-π\pi's training objective does not depend on a particular backbone, and Stage 2 improves the memory policy both with and without Stage 1 initialization.

Both stages train a language model to write guidance, and Stage 2 also teaches it when to abstain. On WebArena, Stage 2 raises success from 29.3% to 37.9% without bank supervision and from 35.0% to 43.1% after Stage 1 (Table 3). We used Qwen2.5-7B-Instruct in the submission to match Memory-R1 [4] for a fair comparison. The cross-agent experiment in Table 4 varies the downstream agent and keeps the memory generator fixed.

For the backbone comparison, we trained Qwen3.5-4B as the memory policy. We kept the JEF-Hinter bank, Stage 1 SFT budget, Stage 2 rollout and token budgets, downstream agent, prompts, and evaluation protocol unchanged. The table reports WebArena success and the unweighted four-benchmark mean after each training stage.

Memory settingWebArena SR (%)Four-benchmark avg. SR (%)
Base Agent27.145.1
RAG31.446.9
Mem-π\pi (Stage 1), Qwen2.5-7B-Instruct35.051.0
Mem-π\pi, Qwen2.5-7B-Instruct43.155.2
Mem-π\pi (Stage 1), Qwen3.5-4B33.850.6
Mem-π\pi, Qwen3.5-4B41.254.1

Stage 2 improved WebArena success by 7.4 points with Qwen3.5-4B, close to the 8.1-point gain with Qwen2.5-7B-Instruct. The same objective improved the memory policy across these two Qwen generations and model sizes.

[1] H. Nekoei, A. Jaiswal, P. Bechard, O. Shliazhko, O. Marquez Ayala, M. Reymond, M. Caccia, A. Drouin, S. Chandar, and A. Lacoste. Just-in-time episodic feedback hinter: Leveraging offline knowledge to improve LLM agents adaptation. arXiv:2510.04373, 2025.

[2] P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, et al. SkillRL: Evolving agents via recursive skill-augmented reinforcement learning. arXiv:2602.08234, 2026.

[3] P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv:2504.19413, 2025.

[4] S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, et al. Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv:2508.19828, 2025.

[5] S. Zhang, J. Wang, R. Zhou, J. Liao, Y. Feng, Z. Li, Y. Zheng, W. Zhang, Y. Wen, Z. Li, et al. MemRL: Self-evolving agents via runtime reinforcement learning on episodic memory. arXiv:2601.03192, 2026.

[6] P. Asawa, A. Zhu, A. O'Neill, M. Zaharia, A. G. Dimakis, and J. E. Gonzalez. How to train your advisor: Steering black-box LLMs with advisor models. arXiv:2510.02453, 2025.

[7] X. Li, H. Li, Y. Zhou, Y. Liu, and Q. Ai. Beyond experience retrieval: Learning to generate utility-optimized structured experience for frozen LLMs. arXiv:2602.02556, 2026.

Reviewer_teo9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mem-π, a generative memory framework that learns when to provide guidance and what to generate. It uses supervised experience distillation followed by reinforcement learning with a decision-content decoupled objective. Experiments across four agent benchmarks show consistent improvements over retrieval-based memory methods.

Critical strengths
  1. The decision-content decoupled optimization provides a practical way to learn both when and what to generate. The structured rollouts are a particularly useful design choice.

  2. The evaluation covers diverse agent benchmarks, with consistent improvements and useful ablation studies supporting the main design choices.

Critical weaknesses
  1. The paper lacks direct comparisons with closely related generative memory methods, such as SEAM and Advisor Models. This makes it difficult to assess the improvement over existing generative approaches.

  2. The shared pre-distilled experience bank may disadvantage methods such as Memory-R1, which originally rely on their own memory construction pipelines. This raises concerns about the fairness of the comparisons.

Questions for authors
  1. Could the authors compare Mem-π with recent generative memory methods under similar settings?

  2. Would the results change if Memory-R1 were evaluated using its original memory construction pipeline?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer teo9 (Part 1/2)

We thank the reviewer for recognizing the decision-content decoupled optimization and the structured rollouts, and address below the direct comparison with generative memory methods and the fairness of the shared experience bank.

W1 and Q1. Matched comparisons with SEAM and Advisor Models.

Mem-π\pi differs from SEAM [1] and Advisor Models [2] in two ways, by distilling domain-specific workflow experience before RL on downstream outcomes and, more importantly, by learning when to abstain.

Both methods start GRPO directly from the backbone, as their papers describe, and provide guidance on every task, including those the agent can solve on its own. Our introduction identifies this lack of learned abstention as a gap in generative memory. Mem-π\pi uses SFT to distill the training-split JEF-Hinter hints into its generator. Stage 1 alone reaches 35.0% on held-out WEBARENA tasks, ahead of both learning-based baselines in Table 1. Stage 2 adds 8.1 points on WEBARENA when evaluated with GPT-5.4-mini, even though it was trained on outcomes from a Qwen2.5-7B-Instruct agent. To learn when to abstain, Mem-π\pi compares the frozen agent's outcomes with and without guidance on the same task and learns to withhold guidance when it does not help.

The paper tests each design choice separately. Running Stage 2 directly on Qwen2.5-7B-Instruct without Stage 1 lowers success by 5.2 points on WEBARENA and 4.5 points on ALFWORLD. Forcing [GENERATE] on the trained policy at test time lowers WEBARENA success from 43.1% to 39.8%.

As the reviewer suggested, we evaluated SEAM and Advisor Models with 7B guidance generators, the shared training-split JEF-Hinter bank, a 256-token guidance budget, the same frozen downstream agent, and the same three-seed evaluation protocol as in the main comparison. Both methods were trained on downstream outcomes, with SEAM retaining its own utility objective. Under the same conditions, we also evaluated SkillRL [3] with a reranker and a 7B per-query rewriter on top of its retrieval.

MethodWEBARENA SR (%)ALFWORLD SR (%)
RAG31.485.5
SkillRL + rerank and rewrite34.487.2
Advisor Models33.886.1
SEAM34.886.6
Mem-π\pi (Stage 1)35.089.5
Mem-π\pi, w/o Stage 1 init (Table 2)37.986.9
Mem-π\pi, [GENERATE] forced at test time (Table 3)39.890.0
Mem-π\pi43.191.4

Mem-π\pi with learned abstention outperforms both methods even when it starts RL directly from the backbone. That variant reaches 37.9% on WEBARENA (Table 2), 3.1 points above SEAM, and the trained Mem-π\pi still leads SEAM by 5.0 points when forced to generate on every test task. SEAM performs about as well as distillation alone, while full Mem-π\pi, which combines distillation and learned abstention, has the highest success rate.

We will add the measured results of both methods to Table 1 and Table 5, with per-benchmark results in Appendix C.

AuthorsOpenReview ↗

Author Response to Reviewer teo9 (Part 2/2)

W2 and Q2. Shared-bank fairness and Memory-R1's original construction step.

The comparison is fair because Memory-R1 [4] and Mem-π\pi start from the same training-split hint bank and 7B backbone, differing only in whether they train a retrieval or a generation policy.

Under the shared-bank protocol in Appendix C.3, we adapt only Memory-R1's memory source and retain its published architecture, action space, hyperparameters, and optimization procedure. Its RL-trained manager takes JEF-Hinter hints as candidate entries and learns to add, update, or delete them or leave them unchanged (Table 6). Mem-π\pi distills the same hints into its generator in Stage 1, using the same Qwen2.5-7B-Instruct backbone. Neither method reads raw trajectories, and both are evaluated with the same frozen agent. Giving each method its own bank would change both the information it starts with and the policy it trains.

For the comparison the reviewer requested, we used Memory-R1's own construction step to build its initial entries from the same raw training-split trajectories in place of the JEF-Hinter entries. We kept its manager training, the downstream agent, and the three-seed evaluation protocol unchanged.

MethodWEBARENA SR (%)Four-benchmark avg. SR (%)
Memory-R1, shared bank33.248.8
Memory-R1, own construction from raw trajectories33.649.0
MemRL, shared bank34.049.4
Mem-π\pi (Stage 1)35.051.0
Mem-π\pi43.155.2

Using Memory-R1's own construction step leaves the ranking unchanged. Its WEBARENA success rate rises by 0.4 points, less than the shared-bank standard deviation of 0.7 points (Appendix C.11), and remains 9.5 points below Mem-π\pi. A likely reason for the small change is that the JEF-Hinter hints already capture the procedures that Memory-R1 extracts. With the starting experience held fixed, Memory-R1 learns which entries to keep and retrieve, while Mem-π\pi learns to generate guidance for the current task and to withhold it when it would not help.

[1] X. Li, H. Li, Y. Zhou, Y. Liu, and Q. Ai. Beyond experience retrieval: Learning to generate utility-optimized structured experience for frozen LLMs. arXiv:2602.02556, 2026.

[2] P. Asawa, A. Zhu, A. O'Neill, M. Zaharia, A. G. Dimakis, and J. E. Gonzalez. How to train your advisor: Steering black-box LLMs with advisor models. arXiv:2510.02453, 2025.

[3] P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, et al. SkillRL: Evolving agents via recursive skill-augmented reinforcement learning. arXiv:2602.08234, 2026.

[4] S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, et al. Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv:2508.19828, 2025.

Thank you for the detailed response and additional experiments. The new comparisons with SEAM and Advisor Models address my concerns about missing generative baselines. The additional Memory-R1 experiment also helps clarify the fairness of the evaluation. Overall, my main concerns have been adequately addressed, and I will maintain my original rating of 3 (Weak Acceptance).

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for taking the time to review our response and the additional experiments. We’re glad they addressed your main concerns and would be happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks from the Authors

Dear AC, SAC, and Reviewers,

We sincerely thank the AC and SAC for coordinating the review process and the reviewers for their constructive feedback, which helped us strengthen our baseline comparisons, the ablations of the training objective, and the explanation of the method.

Summary of the Original Reviews

Major strengths noted by all three reviewers (iyyE, 7Hu6, teo9) include a concrete, intuitive mechanism for learning when and what to generate, consistent gains across four agent benchmarks, and useful ablations and controls. Reviewers iyyE and 7Hu6 also noted that the separate memory policy works with frozen or closed agents and transfers to a different agent.

Key concerns include the missing direct comparison with generative guidance methods such as SEAM and Advisor Models (iyyE, 7Hu6, teo9), the need for a stronger retrieve-and-rewrite baseline than BM25 top-1 (iyyE, 7Hu6), isolating the components of the decision-content objective and the unclear decision-token normalization and memory-policy input (iyyE), generalization under a larger distribution shift or across sessions and sensitivity to the memory-generator backbone (7Hu6), and the fairness of the shared experience bank for Memory-R1 (teo9).

Summary of the Rebuttal and Discussion

We addressed all substantive concerns with targeted clarifications, new experiments, and supporting evidence:

  • Compared Mem-π\pi with Advisor Models and SEAM under the same bank, guidance budget, agent and evaluation: Mem-π\pi led both on WebArena (43.1% vs. 34.8% for SEAM), even without Stage 1 (iyyE, 7Hu6, teo9).
  • Added a reranker and a per-query rewriter to SkillRL, the retrieve-and-rewrite baseline with an evolving bank. With or without them, SkillRL trailed Stage 1 of Mem-π\pi alone (iyyE, 7Hu6).
  • Clarified that "w/o structured rollout" removes only the forced [ABSTAIN] branch, and added the two requested controls and a rollout-budget sweep, showing that branch coverage and token routing each contribute to the gain (iyyE).
  • Specified the decision token's weight and likelihood under forced branching, and showed that the policy receives the task and the initial observation, with the exact prompt (iyyE).
  • Showed that Mem-π\pi beat RAG in ALFWorld rooms absent from training, improved faster than retrieval-based memory over four WebArena sessions, and kept a similar Stage 2 gain with a Qwen3.5-4B policy (7Hu6).
  • Explained that Memory-R1 and Mem-π\pi start from the same hints and backbone, and ran Memory-R1 with its own memory construction, which left the ranking unchanged (teo9).

Final Resolution

Reviewers 7Hu6 and teo9 replied that our response addressed their concerns and maintained their ratings of 3 (Weak acceptance).

Reviewer iyyE did not reply during the author-reviewer discussion period and maintained a rating of 3 (Weak acceptance). We addressed all their points in detail, such as controls that isolate token routing from the content gate, a retrieve-and-rewrite baseline on the same hint bank, and direct comparisons with Advisor Models and SEAM.

We thank the AC, SAC, and reviewers again for their thoughtful engagement and hope our responses are helpful in the final decision process.

Sincerely,

The Authors of Paper 17708

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.