acceptodds
Under review as a conference paper at ICLR 2027

When Does Memory Become Capability? Agent Self-Improvement through Memory Evolution and Consolidation

10 views

Abstract

Agentic reinforcement learning (RL) trains LLM agents from multi-turn environment interaction, but each episode reaches the policy only through a scalar outcome reward. Procedural memory offers a complementary channel: agents store natural-language guidance from past interactions and reuse it on future tasks. However, a memory that helps its source episode may not help on other tasks, and a memory that helps when retrieved may not improve the agent when used for training. We introduce MECA (Memory Evolution and Consolidation for Agents), which brings memory management into agentic RL: a single policy learns to solve tasks and to maintain its procedural memory by adding, revising, and removing memories. MECA rewards each proposed memory edit by its transfer, the change in success it causes on tasks it was not written from, and optimizes acting and editing jointly in one RL update. The same signal decides whether a memory stays in external memory or is consolidated into the parameters through self-distillation; the updated agent then generates new experience and edits. Memory management thus becomes a learned component of agent self-improvement. Across ALFWorld, WebShop, and Search-QA with three backbones, MECA outperforms controlled agentic RL and memory-based baselines, including GiGPO trained on as many environment trajectories, learns to write increasingly transferable memories, and retains most of its gains when all memory is removed at inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic reinforcement learning, agent self-improvement, memory evolution, memory consolidation, self-distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FzgEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper build on the premise that including memories in an agentic LLM can help improve the performance. These memories can be kept externally and retrieved when needed, or trained into the LLM. This paper proposes evaluating memories on tasks they were not written from, and using this transfer signal both to train the memory editor and select memories for consolidation.

Critical strengths
  • The paper includes many ablations and diagnostic studies examining memory editing, selection, and consolidation.
  • Evaluation with memory removed and the controlled comparisons provide useful evidence about the method.
Critical weaknesses
  • I had a difficult time going through the paper, mostly due to the density of the information presented. Sections 2.1, 2.2, and 4.4 are particularly terse. It would help if the authors improved the flow and led with explanations before presenting numerical results.
  • The comparability of the published results in Table 1 is unclear. Since the authors do not run many of the baselines in Table 1 themselves, it becomes important to clarify whether the experimental and evaluation setups used to obtain those numbers are comparable with this paper's. This is particularly important since the performance gap is relatively small in many cases and differences in training or evaluation setups (e.g., data splits) can have significant impact.
  • The evidence for transfer seems to be primarily within task families and fixed environments. The broader claim about when memories should be internalized should be scoped accordingly.
Questions for authors
  1. Are the training data, evaluation tasks, and Search-QA retrieval setup the same as those underlying the copied results (e.g., SDAR UCOB)?
  2. Is it possible that a memory is selected for consolidation and later revised or deleted? How often does this happen, and how is guidance already internalized into the model handled if it is subsequently found to be incorrect?
  3. Could the authors clarify the construction of the cold-start SFT dataset and the exact training protocol used for MECA? Does the dataset include revision and deletion examples, or are these operations learned entirely through online RL?
6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer FzgE - Part 1

We thank the reviewer for the careful reading and for recognizing our ablations, diagnostic studies, memory-removed evaluation, and controlled comparisons. We address each weakness and question below and revised the paper accordingly. All changes are highlighted in color in the revised PDF.


Summary of revisions for Reviewer FzgE

  • W1: Sections 2.1, 2.2, and 4.4 rewritten to lead with the question and intuition before the numbers.
  • W2/Q1: shared framework (verl-agent) and data-split protocol stated in Section 4.1; setup of every published baseline in new Appendix P (Table 26).
  • W3: held-out Search-QA results (Table 27) and new evaluation in unseen ALFWorld environments (Table 28) in new Appendix P, cited in Section 4.4.
  • Q2: answered from existing results (Appendix G, Table 7) and new log statistics in this response.
  • Q3: Appendix C now describes how the labeler's store is built, with operation counts and cold-start SFT settings.

W1. Density of Sections 2.1, 2.2, and 4.4

We are sorry that the previous version of these sections did not convey our ideas clearly, and we rewrote them to make the ideas clearer. Below we restate the main insight of each section.

Section 2.1: a memory that helps its own episode need not help other tasks. The most direct way to reward a memory edit is by how much it helps the episode it was written from. However, many such edits record details of that episode, such as where an object happened to be, that do not recur on other tasks:

We therefore examine whether source benefit predicts transfer, the effect of an edit on other tasks of the same family.

The two are only weakly related. What matters for a reward, however, is not predicting transfer exactly but ranking edits well, and scoring an edit on other tasks of the same family selects far more transferable edits than scoring it on its own episode:

For training an editor, a reward must rank edits rather than predict transfer exactly, so we compare scores by the edits they select.

This is why MECA rewards each edit by its transfer to tasks it was not written from.

Section 2.2: a memory that helps when read need not help once learned. Reading a memory and learning from it are different: a memory can raise success while the agent reads it, yet fail to improve the policy once it is distilled into the parameters (its learning value). Transfer and learning value are only moderately correlated over all edits, but the most transferable edits are clearly enriched for learning value, so transfer is a good criterion for deciding which memories to consolidate, and it comes for free with the editing reward.

A moderate correlation over the whole pool does not prevent good selection, however, because consolidation only requires the top of the ranking to be enriched.

Section 4.4: how memory becomes capability in MECA. Training with the transfer reward changes MECA in three connected ways. The agent learns to write memories that transfer better, shifting from episode-specific notes to general procedures; the memories selected by transfer improve the policy once consolidated, even when the memory is removed; and, as a result, the agent depends less and less on reading its memory, whereas a memory baseline without this selection depends on it more.

We now examine how the transfer signal turns memory into capability over MECA's training, in three steps: what the agent learns to write, which memories it consolidates, and how its reliance on memory changes.

AuthorsOpenReview ↗

Rebuttal to Reviewer FzgE - Part 2

W2 and Q1. Comparability of the published results in Table 1

Thank you for raising this; the previous version did not state the shared setup clearly. Yes: the training data, evaluation tasks, and Search-QA retrieval setup are the same. MECA and our GRPO and GiGPO runs are built on verl-agent (github.com/langfengQ/verl-agent), the agentic RL framework released by the GiGPO authors (Feng et al., 2025), and use its data-split protocol. SDAR (Lu et al., 2026) builds on the same framework and adopts the same protocol, and the other published baselines follow SDAR's or GiGPO's protocol:

  • ALFWorld: the GiGPO training split, with results on the seen evaluation split (valid_seen).
  • WebShop: a set of 128 evaluation tasks.
  • Search-QA: the Search-R1 protocol: training on NQ and HotpotQA (in-domain), with the other five datasets held out; E5 retriever; evaluation on the seven datasets.

Table F1. Setup of the published baselines in Table 1 (as stated in each paper, in the sections listed under "Stated in"; new Appendix Table 26)

MethodProtocolALFWorldWebShop evaluationSearch-QAStated in
Ours (MECA, GRPO, GiGPO)GiGPOGiGPO split; valid_seen128 tasksSearch-R1; NQ + HotpotQA; E5Sec. 4.1; App. C, E
SDAR (Lu et al., 2026)GiGPOGiGPO split; valid_seen128 tasksSearch-R1; NQ + HotpotQA; E5Sec. 3
OPID (Yang et al., 2026a)GiGPOGiGPO split; valid_seen128 tasksSearch-R1; NQ + HotpotQA; E5Sec. 4.1; App. B.1
SEED (Wu et al., 2026)GiGPOGiGPO split; valid_seen128 tasksSearch-R1; NQ + HotpotQA; E5Sec. 4.1; App. B
UCOB (Tu et al., 2026)GiGPO, via SDARGiGPO split; valid_seen128 tasksSearch-R1; NQ + HotpotQA; E5Sec. 6.1
GRSD (Zheng et al., 2026)GiGPO, via SDARGiGPO split; valid_seen128 tasksSearch-R1; NQ + HotpotQA; E5Sec. 4.1; App. B, C
AgentOPSD (Wang et al., 2026)GiGPO, via SDARGiGPO split; valid_seen128 tasksSearch-R1; NQ + HotpotQA; E5Sec. 3.1; App. C

The benchmarks, data splits, evaluation tasks, and Search-QA retrieval setup are therefore shared; each paper keeps only its own method-specific hyperparameters. This is why the numbers in Table 1 are comparable, and why we report the published results rather than re-running each method. We now state this in Section 4.1:

All published baselines in Table 1 use the data splits and evaluation protocol of verl-agent (Feng et al., 2025), on which our pipeline is also built: the GiGPO ALFWorld split, the 128 WebShop evaluation tasks, and the Search-R1 setup for Search-QA (Appendix Table 26).

Our main conclusions still rest on Table 2, where every method, including GRPO and GiGPO, is also trained with the same base models, prompts, context limits, and training schedule as MECA. Table 1 places MECA among published methods; as Section 4.2 notes, MECA is not uniformly best there.

W3. Transfer beyond the training distribution

Thank you for this comment. Our evaluation already measures transfer to data never seen in training, and we have added a new test on unseen ALFWorld environments.

Search-QA already includes held-out datasets. Following the Search-R1 protocol, Search-QA trains only on NQ and HotpotQA, so five of the seven evaluation datasets (TriviaQA, PopQA, 2Wiki, MuSiQue, and Bamboogle) are held out: their questions, sources, and reasoning patterns are never seen in training. Splitting the Search-QA results of Table 1 accordingly:

Table F2. Search-QA exact match on training and held-out datasets (%, size-weighted; computed from Table 1, MECA with memory; new Appendix Table 27)

BackboneIn-domain: GiGPOIn-domain: MECAGainHeld-out: GiGPOHeld-out: MECAGain
Qwen2.5-7B42.245.4+3.148.552.1+3.6
Qwen2.5-3B39.143.6+4.545.048.7+3.8
Qwen3-1.7B38.442.5+4.142.845.4+2.6
Average+3.9+3.3

MECA's gain over GiGPO on the held-out datasets (+3.3 points on average) is close to its gain on the training datasets (+3.9). The memories written and consolidated from NQ and HotpotQA questions therefore carry over to question sets never seen in training.

AuthorsOpenReview ↗

Rebuttal to Reviewer FzgE - Part 3

Results on unseen ALFWorld environments [NEW]. ALFWorld provides an unseen split (valid_unseen, 134 tasks) whose environments (rooms, with their layouts and objects) never appear in training. We evaluated the final checkpoints of Table 2 on all 134 tasks (3 seeds, same evaluation protocol, memory frozen), with and without memory:

Table F3 [NEW]. ALFWorld success in seen and unseen environments (%, mean over 3 seeds; seen: Table 2 and Appendix Table 4; new Appendix Table 28)

Method7B seen7B unseen3B seen3B unseen1.7B seen1.7B unseen
GiGPO87.582.085.980.572.766.9
GiGPO, 225 iterations91.986.787.281.581.076.0
GiGPO + unfiltered memory (memory removed)88.882.383.978.178.672.9
GiGPO + unfiltered memory93.589.189.183.984.979.9
MECA (memory removed)95.889.892.787.089.883.9
MECA97.792.994.588.891.485.2

MECA leads every method in unseen environments with every backbone, both with and without memory. Moving to unseen environments lowers success by a similar amount for all methods (5.3–6.0 points on average), so MECA's advantage is largely preserved:

  • Without memory, MECA leads GiGPO by 10.4 points on average (seen: 10.7) and the stronger trajectory-matched control by 5.5 points (seen: 6.1).
  • With memory, MECA leads GiGPO + unfiltered memory by 4.7 points (seen: 5.4).
  • Reliance on memory stays low. Reading memory adds 2.1 points for MECA in unseen environments (seen: 1.8), compared with 6.5 for GiGPO + unfiltered memory (seen: 5.4). MECA's capability in new environments therefore resides mainly in the policy rather than in the memory it reads.

We added both results as Appendix Tables 27 and 28 and cite them in Section 4.4.

MECA's gains are not limited to the tasks it was trained on: they hold on held-out question sets in Search-QA and in unseen ALFWorld environments, and they are carried mainly by the policy rather than by the memory. Extending this evaluation to transfer across different types of tasks is a natural next step. We hope these results address the reviewer's concern about the evidence for transfer.

Q2. Can a consolidated memory later be revised or deleted?

Memories selected for consolidation (review memories) stay in a training-only pool, and during training one of them is retrieved into a task group's context with probability 0.125 (Appendix C, Retrieval). When this happens for a source task, the review memory appears in that task's dossier, and the editor can revise or delete it; such edits are scored by transfer and routed like any other edit. Deletions are never consolidated (Eq. 7), and when a deletion and a review-routed revision of the same memory arrive in the same iteration, the revision is kept (Appendix D, Conflicts between accepted edits).

Frequency. In our training runs, 7.2% of review memories are later revised and 0.8% are deleted, a median of 35 iterations after consolidation.

How incorrect guidance is handled. MECA addresses this mainly by making incorrect consolidation rare, and secondarily by correcting it through continued training.

  • Consolidation is reserved for reliably transferable edits. An edit is consolidated only if the lower bound of its transfer exceeds the threshold (Le≥ηL_e \ge \eta) and it passes the worst-query veto (Be>−ρB_e > -\rho); in our runs, 9% of proposed edits meet both conditions, whereas 33% are kept in external memory and 58% are discarded.
  • Consolidated guidance is rarely incorrect when re-tested. Appendix G re-scores edits on a fresh panel of query tasks drawn after routing, with new controls (Appendix Table 7). Of the 24 review-routed edits, 23 (96%) still improve success, with a mean transfer of 22.8 points, compared with 9.5 for edits kept in external memory; on an independent probe panel, their read value is 26.3 points. The online estimates used for routing are optimistic (36.7 points), but the selected edits remain strongly positive.
Reviewer_RstwOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MECA, a unified framework that teachs LLM agents to do memory edits and consolidates useful memory into agents. The authors first introduce their observations from initial analyses that the read values of memories have very weak correlation with their transfer values, and thus use the transfer values to build the rewards in memory edit RL training and the memory selection threshold for self-distillation context. Extensive experiments across environments and backbone models validate the effectiveness of MECA and justify that the core improvement comes from its design.

Critical strengths
  1. This paper studies the problem of memory edits and consolidation, which is timely and valuable.
  2. The presentation of this paper is generally good, making it easy to follow.
  3. The definition of read and transfer value is clear, and the observations about the low correlation between these two are helpful.
  4. The idea of dividing memories for reading and training according to their transfer value is straightforward given the observations and validated through extensive experiments.
Critical weaknesses
  1. The improvement of performance looks a bit marginal given the high rollout overhead of MECA. The authors mention that "MECA also collects 64 intervention rollouts per iteration to measure transfer", which is 1.5x compared to baselines (128). Although the controlled baselines with the same number of rollouts are provided for comparison, I am still concerned that the benefit of MECA is not significant.
  2. Some claims and conclusions are not very well presented or justified. See comments below.
Questions for authors
  1. L132: Where can I find the results for this?

Keeping the top 30% of edits by transfer on a separate query panel gives mean probe transfer 9.5 points, versus 5.8 by source benefit; a single disjoint query task, scored with as many rollouts as the source panel, gives 11.5, so disjointness alone accounts for this gain.

  1. Figure 2b: How is the oracle implemented?
  2. L210: I generally understand the idea of putting regular and intervention trajectories together when calculating the advantages, but if there are too many cases where an edit improved success yet all of its intervention rollouts receive the same return, the training distribution would go towards edited context. In case that this edit is not finally preserved, will this be harmful for performance after consolidation? Can the authors provide some explanation whether such cases are rare given their memory selection method or whether the influence is actually neglectable?
  3. In Sec 2.2, the authors conclude that transfer and learning value has low correlation, yet they finally use transfer value (with its variance) to determine what to consolidate and report "Transfer is the editing signal that matters" in Sec 4.3. Can the authors provide more explanation to this conflict? For example, if we select in the same way as MECA, will the correlation be much higher? What are the key differences here?
  4. L307-310: The description in this paragraph is not clear enough. The first sentence compares to the raw GiGPO (and the average is missing from Table 2) and the followed sentence compares to GiGPO + unfiltered memory. It would be better if both compare to the raw GiGPO and the trend will be easier to interpret.

Labeler SFT improve GiGPO by only 4.5, 5.3, and 3.4 points on average ... yet MECA leads it by 6.3 points

  1. In Table 12, there is a no-cold-start ablation, but in the main text the authors do not mention what is the cold-start process for MECA. I suggest the authors to include a reference to Ap. C in the main text so the readers may know about the process.
7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Rstw - Part 1

We thank the reviewer for the positive assessment, in particular of the read/transfer distinction and of our observations, and for the detailed questions, which helped us sharpen several claims. We first address the overhead concern and then answer each question in turn. All changes are highlighted in color in the revised PDF.


Summary of revisions

  • W1: margins over the matched controls with standard errors (new Appendix L); equal-budget MECA in new Appendix O (Table 23), cited in Section 4.3; Search-QA gains stated in Section 4.2.
  • Q1: selection results with confidence intervals in new Appendix O (Table 22), cited at L132; L132 sentence reworded.
  • Q2: oracle protocol described in Section 2.2.
  • Q3: intervention loss restricted to accepted edits in new Appendix O (Table 24), cited in Section 4.3.
  • Q4: Section 4.3 paragraph heading renamed; MECA's routing rule evaluated on the Observation 2 pool in new Appendix O (Table 25), cited in Section 3.3.
  • Q5: L307–310 rewritten relative to GiGPO, with averages in the text.
  • Q6: cold start named in Section 3, with a pointer to Appendix C.

W1. Rollout overhead and significance of the gains

This is a fair concern: MECA's 64 intervention rollouts per iteration must be accounted for. We summarize four pieces of evidence that, we believe, indicate the gains are not explained by the extra rollouts.

  1. Matched trajectories. Both GiGPO controls in Table 2 receive exactly as many environment trajectories as MECA: 12 rollouts per task (192 per iteration) or 225 iterations (28,800 in total). Extra rollouts alone improve GiGPO by only 1.5 and 2.0 points on average, whereas MECA improves it by 11.6 points with memory and 9.8 points without memory (Table R4 under Q5). On ALFWorld, the two controls close 33% and 37% of the GiGPO–MECA gap averaged over backbones, and only 7% and 15% on Qwen2.5-3B (Appendix F).

  2. Cost (Qwen2.5-3B on ALFWorld, where we measured it). MECA is more expensive per iteration than GiGPO (245 s vs. 183 s), since it also proposes edits, measures their transfer, and distills. Over training, however, it uses fewer environment actions than either matched control (0.55M vs. 0.62M and 0.60M; Appendix Table 6), although all three collect the same number of trajectories: MECA's episodes are shorter on average (19.1 vs. 20.8–21.5 actions), likely because successful ALFWorld episodes end before the 50-action limit. The 225-iteration control also needs about 12% more wall-clock time than MECA (225 × 183 s ≈ 41.2k s vs. 150 × 245 s ≈ 36.8k s; Figure 6b). MECA reaches GiGPO's final ALFWorld success with 50% of GiGPO's environment trajectories (Figure 6a).

  3. Significance. With the per-seed standard deviations now reported in Appendix L, MECA exceeds the stronger matched control in every backbone–benchmark pair, even with all memory removed. A Welch tt-test over the three seeds gives p<0.05p < 0.05 in seven of the nine memory-free pairs; the two exceptions are Search-QA on the Qwen2.5 models (p≈0.1p \approx 0.1), where the gains are smallest:

Table R1. Margin over the stronger trajectory-matched GiGPO control (points, ± standard error of the difference over 3 seeds; Table 2 and new Appendix Table 18)

7B ALFW7B WebS7B SQA3B ALFW3B WebS3B SQA1.7B ALFW1.7B WebS1.7B SQA
MECA (memory removed)+3.9 ± 1.3+9.6 ± 1.1+2.0 ± 1.0+5.5 ± 1.2+3.6 ± 1.0+3.3 ± 1.5+8.8 ± 1.4+31.2 ± 1.0+2.5 ± 0.7
MECA+5.8 ± 1.1+12.2 ± 1.4+3.1 ± 0.9+7.3 ± 1.2+5.9 ± 0.6+4.5 ± 1.3+10.4 ± 1.2+33.6 ± 1.1+3.4 ± 1.0

The median memory-free margin is 3.9 points. Gains are largest where GiGPO is weakest (Qwen3-1.7B WebShop) and smallest on Search-QA (2.0–3.3 points). We now state this explicitly in Section 4.2:

Gains are smallest on Search-QA, where MECA without memory leads the stronger trajectory-matched control by 2.0–3.3 points (seed variation in Appendix Table 18).

AuthorsOpenReview ↗

Rebuttal to Reviewer Rstw - Part 2

  1. Equal total budget [NEW]. For a strict test, we ran MECA with exactly GiGPO's 128 trajectories per iteration: 96 regular (16 tasks × 6 rollouts) and 32 intervention (4 edits × 8 queries × 1 rollout).

Table R2 [NEW]. MECA at GiGPO's budget (Qwen2.5-3B-Instruct, success %, mean over 3 seeds; 128 trajectories per iteration unless noted; new Appendix Table 23)

MethodALFWWebSSQAALFW (mem. removed)WebS (mem. removed)SQA (mem. removed)
GiGPO85.969.543.785.969.543.7
MECA, 128 trajectories (96 + 32)92.275.046.589.872.945.3
MECA, 192 trajectories (submitted)94.577.347.692.775.046.4

At GiGPO's budget, MECA still improves memory-free success over GiGPO by 3.9 and 3.4 points on ALFWorld and WebShop, with a smaller gain of 1.6 points on Search-QA, retaining about 60% of its memory-free gain (57%, 62%, and 59%) and about 70% of its gain with memory. It also outperforms both trajectory-matched GiGPO controls, which use 50% more trajectories (89.8 vs. 87.2, 72.9 vs. 71.4, and 45.3 vs. 43.1 for the 225-iteration control), as well as GiGPO + unfiltered memory evaluated with its memory (92.2 vs. 89.1, 75.0 vs. 73.4, and 46.5 vs. 44.2). This suggests that the extra rollouts are not what drives the gains: at a fixed budget, spending part of it on measuring transfer appears more effective than spending all of it on regular rollouts. We added these results as Appendix Table 23 and cite them in Section 4.3:

Trained with GiGPO's 128 trajectories per iteration, MECA still exceeds both controls without memory (Appendix Table 23).

Q1. L132: where are the selection results?

Thank you for catching this. The protocol is in Appendix G (Selection), but the results appeared only in the text, without uncertainty. We now report them in a new appendix (Appendix O, Table 22), cited at L132 and from Appendix G, with 95% bootstrap confidence intervals (2,000 resamples over edits, with selection redone in each resample) and a random-selection reference:

Table R3. Mean probe transfer of the top 30% of edits under each selection score (Observation 1; 410 edits, 123 selected; points, 95% CI; per-environment columns select the top 30% within each environment)

Selection scoreScored on (rollouts per condition)All editsALFWorld (n = 210)WebShop (n = 200)
Random (mean of all edits)—2.6 [1.3, 3.9]2.4 [0.6, 4.2]2.7 [0.8, 4.7]
Source benefitown source task (32)5.8 [3.6, 8.2]4.7 [1.8, 8.3]6.0 [2.9, 9.5]
Transfer, query panel (MECA's design)8 disjoint queries × 2 (16)9.5 [6.9, 11.8]9.9 [6.3, 13.0]9.1 [5.3, 12.2]
Transfer, single disjoint query1 disjoint query (32)11.5 [9.1, 13.2]10.8 [7.4, 13.1]12.2 [8.8, 15.1]

Paired bootstrap differences (all edits): query panel − source benefit = +3.7 [1.1, 6.0]; single query − source benefit = +5.7 [2.8, 8.0]; single query − query panel = +2.0 [−0.6, 4.4].

The sentence "disjointness alone accounts for this gain" was unclear. It means that scoring on a single disjoint task, with the same rollout budget as source benefit, already beats source benefit (+5.7 points; the CI excludes zero in both environments): the advantage comes from scoring on a task other than the source, not from averaging over more tasks. The single query and the 8-query panel are statistically comparable (+2.0, CI includes zero). MECA uses the panel because it needs half the rollouts per condition (16 vs. 32), and because the worst-query veto (BeB_e) needs several queries to detect edits that help on average but harm some tasks. The revised passage in Section 2.1 (formerly L132) reads:

For training an editor, [...] and retaining the top 30% yields a mean probe transfer of 9.5 points, compared with 5.8 for selection by source benefit and 2.6 for random selection. A single query task, scored with the same rollout budget as source benefit, performs comparably (11.5; Appendix Table 22), indicating that the gain stems from scoring an edit away from its source rather than from averaging over many tasks.

The end-to-end ablations are consistent with this choice: rewarding edits by source benefit instead of transfer lowers MECA's memory-free success by 6.0, 7.3, and 2.9 points on ALFWorld, WebShop, and Search-QA (Appendix Table 12).

AuthorsOpenReview ↗

Rebuttal to Reviewer Rstw - Part 3

Q2. Figure 2b: how is the oracle implemented?

Thank you; this was described only in Appendix G. The oracle selects by measured learning value, using two independent probe panels. Each of the 120 candidate additions is consolidated in its own branch, and its learning value is measured on two disjoint panels of 12 same-family probe tasks with 8 rollouts per branch. Every rule keeps the top 25% of the pool. The oracle ranks candidates by learning value on panel 1, and all rules, including the oracle, are then scored on panel 2. The oracle therefore cannot benefit from selecting on the noise of panel 1, and its 3.5 points serves as a realistic reference for the best achievable selection at this budget. We added this description to Section 2.2, next to the result:

The oracle ranks edits on one panel of probe tasks and, like every rule, is scored on a second, independent panel, so it gains nothing from selecting on noise (Appendix G).

Q3. L210: intervention loss on edits that are later discarded

Thank you for this careful question; the case you describe can indeed occur. Three properties limit its effect:

  • Most discarded edits carry non-positive advantage. An edit is discarded if U^(e)≤0\hat U(e) \le 0 or if its worst-query bound Be≤−ρB_e \le -\rho. In the first case, its intervention advantages Aq,kaug=Rq,ke−Rˉq0A^{\mathrm{aug}}_{q,k} = R^e_{q,k} - \bar R^0_q (Eq. 4) average to about U^(e)≤0\hat U(e) \le 0, so they push the policy away from behavior specific to that edit. Only edits with U^(e)>0\hat U(e) > 0 that are vetoed by the worst-query bound can carry net-positive advantage. In our training logs, only 7% of edits with U^(e)>0\hat U(e) > 0 are vetoed, so such cases are uncommon. Table R5 also shows that excluding all discarded edits changes little.
  • The update is conditioned on the edited context. Intervention tokens are scored under Cq⊕eC_q \oplus e (Eq. 5), so the update mainly teaches the policy to act given that memory. Edits routed to external memory remain retrievable, so learning to use them is not wasted.
  • The net effect after consolidation is positive. Removing the intervention loss lowers memory-free success by 1.8, 2.6, and 0.9 points on ALFWorld, WebShop, and Search-QA (Appendix Table 12). We also ran the targeted variant you suggest:

Table R5 [NEW]. Intervention loss restricted to accepted edits (Qwen2.5-3B-Instruct, memory-free success %, mean over 3 seeds; new Appendix Table 24)

VariantALFWWebS
MECA (intervention loss on all edits)92.775.0
Intervention loss on accepted edits only92.974.2
No intervention loss90.972.4

Restricting the intervention loss to accepted edits changes memory-free success by +0.2 points on ALFWorld and −0.8 on WebShop, within seed variation (std 0.9 and 1.4; Appendix Table 18). In our setting, training on the interventions of discarded edits therefore does not appear to be harmful after consolidation, whereas removing the intervention loss altogether costs 1.8 and 2.6 points. We therefore keep the simpler design, which applies the loss to all edits. We added this result as Appendix Table 24 and cite it in Section 4.3:

Restricting the intervention loss to accepted edits changes memory-free success by at most 0.8 points (Appendix Table 24).

Reviewer_Cm7NOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose to consider the memory management problem as part of the RL training. In particular, the idea of the method is to use the policy to act, edit its own memory and consolidate selected memories. To provide the training reward for RL for memory management, the authors use a set of source and query tasks, such that source tasks contain the information useful to remember and the agent is incentivized to remember that information and also to use that remembered information in query tasks. This incentive comes in the form of the reward signal in RL. The authors perform controlled studies across several memory benchmarks to demonstrate that the proposed method is competitive with prior work.

Critical strengths
  • tackles an important problem of memory consolidation and memory management improvement.
  • shows competitive performance with published memory consolidation baselines or better performance in controlled studies.
  • performs an extremely in-depth study of memory consolidation with language models.
Critical weaknesses
  • Lack of baselines, despite the authors include a significant number of baseline methods, some qualitative ones are missing. For example I think that Markovian Thinker [1]. Even though it is not directly a method for memory consolidation, still the model performs this task implicitly by learning what to put in a fixed size context via RL, which can also be a form of memory consolidation.
  • reinforcement learning significance. Normally, the reinforcement learning results should be reported with variance estimation coming from repeated training with different random seeds. All RL training results appear to be reported with only one training seed. A separate section studying the variance of the results for at least a small subset is necessary.
  • the algorithm introduces a significant amount of moving parts such as hyperparameters and overall engineering. The current ablations of the paper only cover a small amount of the introduced pipeline components and further ablations are necessary. In addition, the ablations show nearly identical performance which can potentially motivate authors to simplify the approach.

[1] Milad Aghajohari, Kamran Chitsaz, Amirhossein Kazemnejad, Sarath Chandar, Alessandro Sordoni, Aaron Courville, Siva Reddy. "The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoning." (2025).

Questions for authors

Please respond to the points outlined in the weaknesses section.

8 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Cm7N - Part 1

We thank the reviewer for the careful reading and for recognizing the depth of our study of memory consolidation. We address the three weaknesses below. All changes are highlighted in color in the revised PDF.


Summary of revisions

  • W1: Delethink (extended across tasks) and MEM1 implemented in our pipeline; results and full protocol in new Appendix K (Table A); Related Work extended (Markovian Thinker, MEM1).
  • W2: new Appendix L with std over seeds for every entry of Table 2 and the ablation table (Tables 18–19) and an analysis of seed variation; main text and Appendix E point to it.
  • W3: new sensitivity analysis for the four previously untested settings in new Appendix N (Table D); ablation standard errors in new Table 19 (Table E); MECA-lite in new Appendix M (Table F). In this response, Table C also maps each component and hyperparameter to the existing evidence in the submission.

W1. Markovian Thinker as baselines

We thank the reviewer for pointing us to the Markovian Thinker. We agree it is relevant: like MECA, it uses RL to teach a model what to keep when it cannot keep everything. We added it, and the closely related MEM1, as baselines.

  • Delethink [Aghajohari et al., 2026] reasons on one problem in fixed-size chunks; at each chunk boundary the context is reset, and the model continues from a short state it wrote. RL learns that state from outcome reward alone.
  • MEM1 [Zhou et al., 2026] applies the idea to multi-turn agents: at every turn the agent rewrites an internal state (<IS>) that merges its previous state with the new observation, and earlier turns are pruned.

Both keep information within one task and discard it when the task ends. MECA works across tasks: it rewards each memory edit by its effect on tasks it was not written from, and consolidates the most transferable memories into the weights.

New experiments (new Appendix K). The reviewer's point suggests a direct test: can a bounded memory learned by RL from outcome reward alone match MECA? We implemented both methods in our pipeline under MECA's budget: the same backbone (Qwen2.5-3B-Instruct), 150 iterations, 192 environment episodes per iteration (as many as MECA's regular and intervention rollouts combined), the same cold-start budget, and a learned state capped at 384 tokens, MECA's retrieval budget. Actions are trained by GiGPO, as in MECA.

  1. Delethink across tasks. Delethink carries its state within one problem, so we extend it across tasks, with one task playing the role of one chunk. The agent solves chains of 4 different tasks from one task family (each with its own goal and environment instance), writes a note after each task, and carries only that note into the next task. As in Delethink, the note is trained by outcome alone: the note written after task kk is rewarded by the success on tasks k+1,…,4k+1, \dots, 4, normalized over 12 rollouts of the same chain.
  2. MEM1 (original). MEM1 already targets multi-turn agents, so we use it as designed: each turn sees only the task, the latest <IS>, the last action, and the new observation, and the state replaces the recent-history window.

With memory, Delethink uses either a frozen note built on development tasks (MECA's frozen-store protocol) or online chains over the evaluation tasks (which favor the baseline); MEM1 uses its state as in training. Memory removed uses an empty note, or an empty MEM1 state at every turn. Appendix K gives the full protocol.

AuthorsOpenReview ↗

Rebuttal to Reviewer Cm7N - Part 2

Table A. Bounded-state baselines (Qwen2.5-3B-Instruct, success %, mean over 3 seeds)

MethodALFWWebSALFW (mem. removed)WebS (mem. removed)
GiGPO (no memory)——85.969.5
GiGPO + unfiltered memory89.173.483.967.4
Delethink across tasks (frozen note)88.071.186.268.0
Delethink across tasks (online chains)88.871.986.268.0
MEM1 (original)86.574.254.957.8
MECA94.577.392.775.0
  • Delethink's gain stays in the note. The note adds 1.6–2.9 points over GiGPO, but without it success returns to GiGPO's level (86.2 vs. 85.9; 68.0 vs. 69.5), so nothing is consolidated into the weights. The note also stays below GiGPO + unfiltered memory (88.0 vs. 89.1; 71.1 vs. 73.4), and online chains, which see the evaluation tasks, add only 0.8 points over the frozen note: outcome reward alone gives a weak signal about what is worth carrying forward. MECA without memory (92.7, 75.0) beats Delethink even with its note.
  • MEM1 is complementary. Its working memory helps on WebShop (+4.7) but little on ALFWorld (+0.6). Emptying the state also removes its only record of earlier turns, so the drop (54.9, 57.8) reflects reliance on working memory rather than missing consolidation. MEM1 manages context within a task and MECA across tasks, so the two could be combined. Even on WebShop, MECA without memory (75.0) exceeds MEM1 with its state (74.2).
  • MECA leads both baselines in every column, by 3.1–8.0 points with memory and 6.5–37.8 without.

In our setting, the answer to the question above is therefore negative: a bounded state learned from outcome reward alone improves an agent only while the state is present. This points to what neither baseline has, per-memory transfer credit and transfer-selected consolidation, as what lets memory become capability in MECA.

We added these results with the full protocol as Appendix K (Table 17), and Delethink and MEM1 to Related Work.

W2. Variance across training seeds

We thank the reviewer for raising this point. We would like to clarify that all RL results of our method in the submission are already averaged over three independent training seeds (seeds 0, 1, and 2, which control task order, the source/query split, and rollout sampling; Appendix E, Seeds and checkpoints). Comparisons between methods are paired across seeds and evaluation tasks and assessed with a two-level bootstrap (Appendix E, Uncertainty), and the curves in Figures 5, 6, and 7 show standard-deviation bands across seeds. We recognize, however, that this is not readily apparent from the main text, as the main tables report means only. To make the seed variability fully transparent, the revision adds a dedicated appendix that reports the standard deviation of every entry, summarized below.

Changes in the revision. A new Appendix L, "Variance across training seeds", reports the std over seeds for every entry of Table 2 (new Table 18) and of the ablation table (new Table 19), with an analysis; the main text and Appendix E point to it. Each std is taken over the three final-checkpoint evaluations, so it includes both training and evaluation randomness. To compare two methods, we use the standard error of their difference, sa2/3+sb2/3\sqrt{s_a^2/3 + s_b^2/3}.

Table B. Controlled comparison with seed variation (success %, mean ± std over seeds 0–2; means as in Table 2; new Table 18 in Appendix L)

Method7B ALFW7B WebS7B SQA3B ALFW3B WebS3B SQA1.7B ALFW1.7B WebS1.7B SQA
GiGPO87.5 ± 0.972.7 ± 1.647.2 ± 0.885.9 ± 1.069.5 ± 1.043.7 ± 0.572.7 ± 0.946.9 ± 1.441.9 ± 1.3
GiGPO, 12 rollouts91.4 ± 1.174.0 ± 1.947.5 ± 1.486.5 ± 0.770.6 ± 1.342.8 ± 1.280.5 ± 1.646.9 ± 1.641.2 ± 0.5
GiGPO, 225 iterations91.9 ± 1.874.5 ± 1.747.6 ± 1.287.2 ± 1.871.4 ± 0.943.1 ± 2.081.0 ± 1.947.7 ± 1.641.4 ± 1.1
GiGPO + labeler SFT92.2 ± 1.176.3 ± 1.547.8 ± 1.088.0 ± 1.871.9 ± 0.943.5 ± 0.882.6 ± 1.254.7 ± 1.241.7 ± 0.7
GiGPO + static memory92.7 ± 1.677.6 ± 1.348.2 ± 1.188.5 ± 1.972.9 ± 1.843.9 ± 1.684.1 ± 1.158.1 ± 1.542.1 ± 1.9
GiGPO + unfiltered memory93.5 ± 1.578.6 ± 1.648.4 ± 1.189.1 ± 1.773.4 ± 1.544.2 ± 1.484.9 ± 0.961.2 ± 1.442.4 ± 0.5
MECA (memory removed)95.8 ± 1.484.1 ± 0.749.6 ± 1.292.7 ± 0.975.0 ± 1.446.4 ± 1.889.8 ± 1.578.9 ± 0.643.9 ± 0.7
MECA97.7 ± 0.586.7 ± 1.750.7 ± 1.094.5 ± 1.077.3 ± 0.647.6 ± 1.191.4 ± 1.081.3 ± 1.244.8 ± 1.4
AuthorsOpenReview ↗

Rebuttal to Reviewer Cm7N - Part 3

Key findings:

  • Seed variation is small relative to the gains: std is 0.5–2.0 points (median 1.2) across all entries of Table 2.
  • MECA trains as stably as the baselines: its std is 0.5–1.7 points with memory and 0.6–1.8 without, vs. 0.5–2.0 for the baselines, so jointly training editing and acting does not increase seed sensitivity.
  • With memory, MECA exceeds the strongest baseline (GiGPO + unfiltered memory) in all 9 backbone–benchmark pairs, by 2.3 (Qwen2.5-7B, Search-QA) to 20.1 points (Qwen3-1.7B, WebShop); a Welch tt-test over seeds gives p<0.05p < 0.05 in 7 of them, the exceptions being Search-QA on Qwen2.5-7B and Qwen3-1.7B (p≈0.06p \approx 0.06 and 0.080.08).
  • With all memory removed, MECA exceeds the strongest trajectory-matched GiGPO control in all 9 pairs (margins 2.0–31.2 points), with p<0.05p < 0.05 in 7; the exceptions are Search-QA on the two Qwen2.5 models (p≈0.1p \approx 0.1). It also exceeds GiGPO + unfiltered memory evaluated with its memory in all 9 pairs, with p<0.05p < 0.05 in 4.
  • Ablations: on ALFWorld and WebShop, every editing-signal variant, and every consolidation variant when memory is removed, falls below MECA by more than two standard errors; refinements such as a separate editor stay within two standard errors (Table E in W3).

These standard errors ignore that methods share seeds and evaluation tasks, so they are conservative; the paired bootstrap of Appendix E gives narrower intervals.

The result also replicates across backbones: the ordering MECA > GiGPO + unfiltered memory > GiGPO holds in all 9 backbone–benchmark pairs of Table 2.

W3. Number of components; ablations look similar; simplification

We agree this deserves a clearer treatment. Our response has four parts.

(i) Coverage of the pipeline. Figure 4 showed only a subset of the ablations. Appendix Table 12 (Appendix H; definitions in Table 13) reports 16 variants and 3 budget-matched controls, and Appendix Table 14 repeats the primary ablations on Qwen3-1.7B. To make the coverage explicit, Table C lists MECA's components and the hyperparameters that govern routing, editing, and consolidation, with the evidence for each; context and length budgets shared with the memory baselines are grouped in one row. Defaults are those of Appendix Table 3; they were chosen on development tasks, and each final configuration was evaluated once (Appendix E).

Table C. MECA-specific components and hyperparameters, and where each is tested

Component / hyperparameterDefaultEvidence
Transfer reward and its query panel4 edits × 8 queries × 2 rolloutsTable 12: no transfer credit, source-task reward, evidence-overlapping queries, source benefit; Appendix G: decoupling reward from routing changes success by ≤0.7\le 0.7 points
Routing: confidence multiplier zz, review threshold η\eta1.645, 0.125Figure 8: ≤2.3\le 2.3 points on WebShop, ≤0.9\le 0.9 on Search-QA; MECA-lite drops zz
Worst-query veto ρ\rho0.25MECA-lite drops it (Table F)
External and review pools3 and 1 retrieved recordsTable 12: no external pool; consolidate, keep external
Revisions and deletionsenabledTable 12: additions only; MECA-lite
Consolidation: selection and budget16 turns per edit, ≤64\le 64 per iterationTable 12: no consolidation, step-matched, random selection, placebo memory, distill all accepted
Editing and intervention lossesweight 1 eachTable 12: no editing loss, no intervention loss
Cold start2,048 labeler demonstrationsTable 12: no cold start
Shared actor and editorone modelTable 12: separate editor
RetrieverlexicalAppendix Table 15: dense retriever
Role weights; invalid-edit / no-op reward1 / 1 / 1; −1 / 0Fixed conventions, not tuned
Context and length budgets: retrieval, memory text, source turns per trajectory3 records / 384 tokens; 128 words; 8 turnsSame budgets as GiGPO + static and unfiltered memory (Appendix E)
Edits per iteration; distillation temperature and mixture weight; review exposure4; 1.2 and 0.5; 0.125New runs (Table D)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.