- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Mem-π represents reusable experience as a separate guidance-generating model for a frozen downstream agent. Supervised experience distillation first learns from training-split hints, followed by reinforcement learning with one forced abstention branch and three generated-guidance branches. Decision tokens receive a cross-branch utility comparison, while content tokens receive within-generation advantages gated by whether generation is beneficial. The memory policy is called once before the agent’s first action. Experiments cover WebArena, WorkArena, ALFWorld and LifelongAgentBench, with the main evaluation transferring from a training-time Qwen agent to GPT-5.4-mini. The reported average SR increases from 49.4 for MemRL to 55.2 for Mem-π.
Critical strengths
-
The work provides a concrete mechanism for deciding whether guidance should be delivered, rather than assuming that more memory always helps. The separation between a trainable memory policy and a frozen receiving agent is also practically useful for transfer to a different agent.
-
The experimental support is substantial: the paper reports held-out task evaluation across four benchmarks, three-seed means and standard deviations, and verified environment outcomes rather than relying on an LLM judge for the primary success metric. The gains are present on every benchmark, not just in an aggregate dominated by one setting.
-
The study includes unusually useful controls and diagnostics: a zero-shot hinter, a no-Stage-1 variant, learned abstention for retrieved hints, forced generation, leave-one-domain-out evaluation, matched bank perturbations and cross-agent routing evaluation. The latter uses separate rollouts for selecting and evaluating a per-task branch, which avoids evaluating that selector on its selection samples.
Critical weaknesses
-
The specific benefit of the decision-content objective is not cleanly isolated. The “without structured rollout” comparison reverts to vanilla GRPO, changing both branch coverage and advantage assignment. The “without delta-gating” variant uses a naive sum of decision and content advantages, rather than only removing the gate while preserving token routing. These comparisons support the full recipe but do not identify which part of the proposed decomposition matters. Moreover, Equation (7) retains per-sequence length normalization: a one-token abstention decision and the decision prefix of a long generated sequence can receive very different effective weights. The exact masking/normalization and old-policy probabilities for forced decision tokens should be specified to make the implemented surrogate reproducible.
-
The strongest evidence is against the specified shared-bank baselines, not yet against query-adaptive guidance alternatives. The RAG baseline uses a verbatim BM25 top-1 hint, and the examples in Figure 5 emphasize copying an incorrect count or overly narrow source constraint. A same-budget query-conditioned rewriter of retrieved hints would directly test whether parametric internalization is needed to correct those failures. Table 5 also identifies learned generative guides as close related approaches, but their comparison remains descriptive. The zero-shot hinter and RAG-with-abstention controls are valuable, yet neither is equivalent to retrieval followed by adaptive rewriting.
-
The actual input to the main text-only memory policy is unclear. Section 2 defines its context as task plus observation, and Appendix C.8 includes accessibility-tree tokenization/prefill in the memory-call cost. However, the text-only memory prompt in Appendix C.12 contains only the task description, without an observation placeholder. This distinction matters for the claim of context-specific guidance and for reproducibility. The policy input, observation truncation and available information should be made consistent across the method, prompt and cost descriptions.
Questions for authors
-
How does the method compare with the same structured branches and a single sequence-level advantage, and with the same decoupled objective but only the content gate removed? Please specify the exact decision-token normalization and likelihood treatment under forced branching.
-
Can a matched retrieval-plus-rewriting control, using the same hint corpus and guidance budget, close the gap? This is especially relevant to the count/constraint correction examples used to motivate generated memory.
-
Does the main text-only memory policy receive the initial environment observation or only the task? Please provide the exact input template and truncation settings that produced Table 1 and the deployment-cost measurements.
3 comments
Author Response to Reviewer iyyE (Part 2/3)
W1b and Q1b. Normalization and likelihood of the decision token.
We would clarify that the decision token is a special token and is not counted in the length normalization, which applies only to the ordinary guidance tokens of the [GENERATE] rollouts.
Each rollout starts with [ABSTAIN] or [GENERATE], receiving only the decision advantage or , where (Eqs. 5 and 6), while guidance tokens receive only the content advantage. Eq. 7 uses the compact notation of Eq. 2 without showing this separation. Written explicitly, rollout contributes inside the sum, where is the per-token term of Eq. 7 and the second term is absent for [ABSTAIN]. Each decision token thus has weight , whether it stands alone or precedes a long hint, while guidance tokens are averaged over hint length.
Eq. 4 forces one [ABSTAIN] token and three [GENERATE] tokens, but none is masked from the loss. Their likelihood ratios use the probability of the forced token, , where is the frozen policy snapshot in Eq. 2, so forcing a branch does not set its old-policy probability to one. Clipping and the token-level KL term apply to both decision and guidance tokens.
As a worked example, consider Eqs. 3 to 6 with and . Suppose the agent fails without guidance, succeeds with hints of 120 and 80 tokens, and fails with a 200-token hint.
| Rollout | TaskReward | Decision adv. | Decision-token weight | Content adv. | Weight per guidance token | |||
|---|---|---|---|---|---|---|---|---|
| [ABSTAIN], | 0 | 0 | none | 0 | 1 | 0 | none | |
| [GENERATE], | 120 | 1 | 0.953 | 1 | 1/120 | |||
| [GENERATE], | 80 | 1 | 0.969 | 1 | 1/80 | |||
| [GENERATE], | 200 | 0 | 1 | 1/200 |
Here and , so the content gate opens. The content advantages use the sample standard deviation of the three [GENERATE] rewards. The table shows advantages before the common factor, clipping, and the KL term. All four decision tokens have weight , while only guidance-token weights depend on hint length. The negative [ABSTAIN] advantage and positive [GENERATE] advantages both favor generation.
W2a and Q2. Query-time rewriting and bank updates.
The SkillRL [1] baseline we already compare against is the retrieve-and-rewrite control the reviewer describes, with a bank that keeps evolving. It retrieves from the same shared bank and rewrites stored entries from the frozen agent's training outcomes as training proceeds. On top of it, we added a reranker and a per-query rewriter after SkillRL's retrieval step.
Both controls started from the same training-split JEF-Hinter corpus as the main comparison and used the same 256-token guidance budget and three-seed held-out evaluation with GPT-5.4-mini. The added variant reranked the retrieved entries and used a 7B model to rewrite them for the current query.
| Method | WEBARENA SR (%) | Four-benchmark avg. SR (%) |
|---|---|---|
| Base Agent | 27.1 | 45.1 |
| RAG | 31.4 | 46.9 |
| Mem0 | 31.9 | 47.8 |
| Memory-R1 | 33.2 | 48.8 |
| MemRL | 34.0 | 49.4 |
| SkillRL | 33.7 | 49.8 |
| SkillRL + rerank and rewrite | 34.4 | 49.6 |
| Mem- (Stage 1) | 35.0 | 51.0 |
| Mem- | 43.1 | 55.2 |
Both controls improved on RAG but scored below Stage 1 of Mem-. With or without query-time rewriting, SkillRL can draw on only the entries it retrieves from the bank for the current task. Mem- generates guidance from experience distilled across the bank and learns to withhold it when it would not help. Both SkillRL variants supply a hint for every task.
Table 1 also compares Mem- with retrieval methods beyond plain BM25. Memory-R1 [2] learns to manage stored entries, MemRL [3] learns which entries are useful for the current task, and Mem0 [4] uses rules to manage its bank. All three score below Mem-. In Table 3, RAG with the same learned abstention objective reached 35.5% on WEBARENA, compared with 43.1% for Mem-.
Author Response to Reviewer iyyE (Part 3/3)
W2b. Comparison with learned generative guides.
Unlike Advisor Models [5] and SEAM [6], Mem- distills domain-specific workflow experience into its generator before RL on task outcomes and, more importantly, learns when to abstain.
Both guides start GRPO from the backbone without prior guidance distillation, as described in their papers, and supply guidance for every task, even when the agent can solve it alone. The submission identifies this lack of abstention as a gap in generative guidance. Mem- uses supervised fine-tuning on training-split JEF-Hinter hints in Stage 1 and reaches 35.0% on held-out WEBARENA tasks, ahead of Memory-R1 and MemRL in Table 1. Stage 2 adds 8.1 points on WEBARENA with GPT-5.4-mini at evaluation, even though its training rewards came from a frozen Qwen2.5-7B-Instruct agent. Comparing that agent's outcomes with and without guidance on the same task teaches Mem- when to withhold a hint.
The submitted ablations separate these two choices. Starting Stage 2 directly from Qwen2.5-7B-Instruct without Stage 1 lowers success by 5.2 points on WEBARENA and 4.5 on ALFWORLD. Forcing the trained policy to generate on every test task lowers WEBARENA success from 43.1% to 39.8%.
We also evaluated Advisor Models and SEAM directly. Following their respective training procedures, we trained each 7B guidance generator with GRPO on the frozen agent's outcomes. Both used the shared bank, the same 256-token guidance budget, and the same downstream agent and evaluation as in W2a.
| Method | WEBARENA SR (%) | Four-benchmark avg. SR (%) |
|---|---|---|
| Advisor Models | 33.8 | 49.1 |
| SEAM | 34.8 | 50.2 |
| Mem- (Stage 1) | 35.0 | 51.0 |
| Mem- | 43.1 | 55.2 |
Starting from the backbone without Stage 1, Mem- with learned abstention already outperforms both guides on WEBARENA. It reaches 37.9%, 3.1 points above SEAM at 34.8%. The trained policy also beats SEAM by 5.0 points when forced to generate. SEAM performs about as well as Stage 1 distillation alone, and the full policy scores highest by combining distillation and learned abstention.
W3 and Q3. Correcting the text-only memory input template and observation budget.
Thank you for catching the missing observation field in Appendix C.12. The text-only Mem- receives both the task and the initial observation, which together form the input in Section 2.
For WEBARENA and WORKARENA, the observation is the initial accessibility tree, truncated to its first 1,024 tokens. For ALFWORLD, it is the initial room description. For LAB, it is the initial shell or database state. The vision-language variant in Appendix C.9 receives the initial screenshot and screen description instead. The corrected text-only template adds the missing field.
[system]
You are a hinter. Given a web task, write one concise, actionable hint that helps the agent take its next step, or abstain when you cannot help.
[user]
Task:
{task_description}
Initial observation: {initial_observation}
First decide: output [ABSTAIN] if you cannot provide useful guidance, or [GENERATE] followed by a concise, actionable hint.
The latency measurement in Appendix C.8 used the same input, and the tokenization and prefill of the task and initial accessibility tree account for 0.7 s of the reported 1.8 s memory call. Table 1 used this input as well.
We will correct the prompt in Appendix C.12 and state the 1,024-token truncation and the single-call timing consistently across the paper.
[1] P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, et al. SkillRL: Evolving agents via recursive skill-augmented reinforcement learning. arXiv:2602.08234, 2026.
[2] S. Yan, X. Yang, Z. Huang, E. Nie, Z. Ding, Z. Li, X. Ma, J. Bi, K. Kersting, J. Z. Pan, et al. Memory-R1: Enhancing large language model agents to manage and utilize memories via reinforcement learning. arXiv:2508.19828, 2025.
[3] S. Zhang, J. Wang, R. Zhou, J. Liao, Y. Feng, Z. Li, Y. Zheng, W. Zhang, Y. Wen, Z. Li, et al. MemRL: Self-evolving agents via runtime reinforcement learning on episodic memory. arXiv:2601.03192, 2026.
[4] P. Chhikara, D. Khant, S. Aryan, T. Singh, and D. Yadav. Mem0: Building production-ready AI agents with scalable long-term memory. arXiv:2504.19413, 2025.
[5] P. Asawa, A. Zhu, A. O'Neill, M. Zaharia, A. G. Dimakis, and J. E. Gonzalez. How to train your advisor: Steering black-box LLMs with advisor models. arXiv:2510.02453, 2025.
[6] X. Li, H. Li, Y. Zhou, Y. Liu, and Q. Ai. Beyond experience retrieval: Learning to generate utility-optimized structured experience for frozen LLMs. arXiv:2602.02556, 2026.
Author Response to Reviewer iyyE (Part 1/3)
We thank the reviewer for recognizing the concrete mechanism for deciding whether guidance is delivered and the verified held-out evaluation, and address below the isolation of the decision-content objective and the matched query-adaptive baselines.
The "w/o structured rollout" ablation removes only the forced [ABSTAIN] branch and keeps separate advantages for decision and guidance tokens. Calling it "vanilla GRPO" in Section 3.2 was imprecise.
Under Eqs. 5 and 6, the decision token learns from Δ=Vabs−Vgen, while guidance tokens learn from reward differences among generated hints only when Δ<0. Stage 2 starts from a policy that favors generation. Without the forced branch, most independently sampled groups contain only [GENERATE] outputs and have no cross-branch decision signal. Training on those groups effectively reduces to GRPO over generated hints, which explains the "vanilla GRPO" description in Section 3.2. Token routing remains in place, but independent sampling cannot ensure that both decisions appear in every group.
To further demonstrate how coverage and routing each contribute, following the reviewer's suggestion, we increased the unstructured rollout budget and evaluated the two requested controls. All variants used the same training-split hint bank, frozen training-time agent, and held-out evaluations. Control (a) uses structured branches with a single sequence-level GRPO advantage for all tokens. Control (b) retains structured branches and token routing but removes only the content gate by setting it to one. Mixed groups (%) is the fraction of training groups containing both decisions, averaged over the final 50 steps.
Both branch coverage and token routing contribute to the gain, while the content gate adds little beyond them. Even with 16 unstructured rollouts, WEBARENA success remains 2.7 points below full Mem-π, whereas four structured rollouts cover both decisions in every group with one quarter as many rollouts. With coverage fixed, control (b) exceeds control (a) by 2.6 points on WEBARENA, isolating the benefit of routing the two advantages to their respective tokens.
A likely explanation is that a single sequence-level advantage penalizes the decision to generate when an individual hint fails, even when generation helps on average. Restoring the content gate adds a further 0.8 points on each benchmark.
We will correct the "w/o structured rollout" description in Section 3.2, add these controls to Table 2, and report mixed-group and gate-activation rates over training in Appendix C.