acceptodds
Under review as a conference paper at ICLR 2027

Which Memories Are Worth Consolidating? Selective Memory Consolidation for Language Model Agents

Abstract

External memories enable large language model agents to reuse prior experience, improve exploration, and solve complex interactive tasks more effectively. However, relying on external memory makes agent performance dependent on retrieving appropriate memories, while noisy or conflicting guidance may lead to incorrect actions. Memory consolidation offers an alternative by selectively transferring experience that contributes to further policy improvement into model parameters, turning accumulated memories into reusable capabilities and reducing reliance on retrieval. The key challenge is to identify which memories remain useful for learning as training progresses, since a memory’s usefulness in guiding a rollout does not necessarily translate into further improvement of the memory-free policy. We propose nsolidation of emory into arameters (CoMP), a selective memory consolidation framework that identifies memories whose induced updates can persistently improve the memory-free policy. CoMP uses memories to guide trajectory generation and reinforcement learning to consolidate useful behaviors, selecting memories based on whether their induced updates are compatible with memory-free policy improvement. Experiments on ALFWorld, WebShop, and ScienceWorld show that CoMP consistently improves memory-free performance over GRPO and memory-augmented baselines. Specifically, CoMP achieves relative improvements of up to 20.3% on ALFWorld and 12.7% on WebShop over GRPO across model scales, while also demonstrating improved generalization to out-of-distribution tasks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dBrFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes COMP (Consolidation of Memory into Parameters), which addresses when an external memory should be internalized into model weights. Its key distinction is that a memory may help the current rollout yet still induce an undesirable or redundant parameter update. COMP therefore compares a memory-retained update against a memory-removed “consolidation” update and selects consolidation when the latter is better aligned with improving the eventual memory-free policy.

Critical strengths
  • The problem formulation is strong: distinguishing “useful to retrieve” from “worth consolidating” is meaningful, timely, and more precise than simply training on all successful memory-guided behavior.
Critical weaknesses
  • Ambiguity in the central gate: The practical memory-free replay gradient is computed from the same memory-guided trajectories with the memory removed, which appears very close to the defined consolidation gradient itself. If these are in fact the same objective, the gate has a substantially different interpretation from alignment with an independently estimated memory-free improvement direction. The paper needs to make that distinction mathematically explicit.
Questions for authors

s the replay-based g~free\tilde g_{\mathrm{free}} actually different from gcong_{\mathrm{con}} in implementation? If so, what exactly differs in the loss, importance ratios, reference policy, or weighting? If they are identical, I would want the paper to restate the interpretation of the gate accordingly.

Reviewer_FXMGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates how to selectively consolidate external memories into the parameters of LLM agents to reduce their reliance on memory retrieval during inference. The authors propose CoMP, a framework that uses retrieved memories to guide trajectory generation and introduces a gradient-based criterion to determine whether memories should be retained or removed during GRPO optimization. The goal is to identify memory-guided updates that benefit the memory-free policy. Experiments on ALFWorld, WebShop, and ScienceWorld demonstrate improvements over several baselines, including standard GRPO and memory-augmented approaches, with additional results suggesting better generalization to out-of-distribution tasks.

Critical strengths
  1. The paper addresses a practically meaningful problem.

  2. The proposed method achieves notable performance improvements over several baselines on ALFWorld and WebShop.

Critical weaknesses
  1. The paper defines gcong_{\mathrm{con}} as the GRPO gradient computed from memory-guided trajectories with the memory context removed. However, the reference gradient g~free\tilde{g}{\mathrm{free}}, intended to represent the memory-free policy improvement direction, is also obtained by replaying the same memory-guided trajectories without memory, while preserving the original actions, rewards, and advantages. If both gradients are computed using the same loss function, then g~free=gcon\tilde{g}{\mathrm{free}} = g_{\mathrm{con}}, reducing the proposed selection criterion to S=∣gcon∣2−⟨gcon,gkeep⟩.S = |g_{\mathrm{con}}|^2 - \langle g_{\mathrm{con}}, g_{\mathrm{keep}}\rangle. In this case, the criterion essentially uses the consolidation gradient itself as the reference for evaluating whether consolidation is preferable, rather than relying on an independently estimated memory-free policy improvement direction. More importantly, this issue also raises concerns about the applicability of the subsequent theoretical analysis in Appendix A. While the first-order approximation is derived using the true memory-free policy gradient, the practical criterion replaces it with a replay-based estimate without establishing that the approximation error is sufficiently small. Although Appendix A provides a formal error bound, it depends on the potentially uncontrolled discrepancy between the true and replay-based gradients, and therefore does not guarantee that the practical selection score correctly predicts the sign of the actual consolidation benefit. The paper neither clearly explains the distinction between these gradients nor sufficiently justifies the theoretical validity of the resulting consolidation decisions. Given that the proposed criterion forms the foundation of the entire method, these issues raise fundamental concerns about the soundness of its central theoretical contribution.

  2. First, the paper does not directly validate whether the proposed selection score can accurately predict the actual consolidation benefit, i.e., whether it correlates with the difference in memory-free performance after applying the consolidation and KEEP updates, respectively. Second, the paper treats GRPO updates on memory-guided trajectories with the memory context removed as memory consolidation. However, the fact that a trajectory is generated with memory does not necessarily imply that its successful behaviors were induced by that memory; similarly, optimizing the model on such trajectories does not necessarily imply that the knowledge or strategies contained in the memory have been internalized. The paper primarily relies on improvements in aggregate memory-free performance to demonstrate effectiveness, without establishing a causal connection between memory selection, the actual influence of memory on agent behavior, knowledge internalization, and downstream generalization. Therefore, the current experiments demonstrate that the proposed training strategy can improve model performance, but do not sufficiently substantiate the central claim that the method can reliably identify and internalize valuable memories.

Questions for authors
  1. The training trajectories are sampled from a memory-conditioned policy, whereas the consolidation gradient is computed with the memory context removed. How does the method account for the resulting distribution mismatch? In particular, is the importance ratio in GRPO sufficient to correct this mismatch?

  2. The main novelty of this paper appears to lie primarily in its proposed gradient-based selection mechanism, making its incremental contribution relatively limited compared with existing work. Moreover, the concerns regarding the theoretical validity of this selection criterion, as discussed above, further weaken the significance of its claimed contribution. Are there any other substantive methodological innovations beyond the proposed selection mechanism? Compared with SKILL0 [1], which adaptively removes external skills based on their measured impact on task performance, what justifies introducing a more computationally expensive gradient-based selection criterion? Can the authors demonstrate that this criterion more accurately identifies updates that benefit the memory-free policy and achieves better performance under matched training budgets?

  3. The OOD experiments cover only a limited number of transfer tasks, while persistent improvement is primarily evaluated through memory-free performance immediately after training. Could the authors provide more extensive cross-task generalization and continual-learning experiments to verify the transferability and long-term retention of the supposedly internalized knowledge?

[1] Lu, Zhengxi, et al. "Skill0: In-context agentic reinforcement learning for skill internalization." arXiv preprint arXiv:2604.02268 (2026).

Reviewer_qriPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose a learning scheme that first involve running the experiments and gathering the useful memory as in tips from past episodes. The main intuition is that by testing each individual tip during rollout we can estimate the useful of the tip and decide whether or not to include the tip for model update of lose it. The heart of the proposed approach is the test which if done ideally is computationally prohibitive so the paper replaces it with a simple gradient estimate proxy. Results are evaluated on ALFworld and and Webshop with a small transfer test on ScienceWorld

Critical strengths
  • The core intuition is interesting and the provides result show promise.
  • The lightweight gradient test in place of real counterfactual test is interesting
Critical weaknesses
  • The memory free reference gradient is built with the same traces guided by the tip.But based on the notation of the alg. 2 and the details of the method section, this is how consolidate gradient in made in (5). This does not sound right since then there will be effectively no reference point.

  • the consolidation branch is off policy and this is not addressed adequately and is not corrected in any sense. The tips are removed but the traces are generated with the tip. The states are also off-distribution. The agent reaches states where only may be possible because of the tip.

  • The main idea of generating with hints and training without them is not novel and been considered in context distillation (snell et al.) The main novel contribution is there mostly on the gradient gate used here which as mentioned in first item is unclear and also not theoritically validated and is mostly validated in a limited experimental manner

  • there seems to be only one training seed which makes the reliablity of result in question. If one seed is used then the fractions of some of the reported numbers in table 1 does not hold up given the number of test cases percategory. I manually inspect a few of them and they suggest numerical error in the report.

  • the OOD test is very minimal and not sufficiently big and large scale to validate the claims.

Questions for authors
  • in start of method 3 you say for each memory, but the alg 2 use a difference notation, and returns M_x for all the memories for a task. which one is correct?
  • do you retrieve top memories per each episode in the task or for the whole task, the notation in alg2. suggests the latter.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Follow-up response to Reviewer FXMG

Thank you for the follow-up questions and for pointing out where our earlier explanation was unclear.

1. What happens to the memory, and what counts as consolidation? Both losses in Eq. (11) use the same trajectories collected with memory. With LkeepL_{\mathrm{keep}}, the model sees the retrieved memory when we compute the update. With LconL_{\mathrm{con}}, we leave it out of the model input for that update. The memory stays in the bank and can still guide later training rollouts. At evaluation, we turn off all memory retrieval, including for memories assigned to KEEP. Choosing LconL_{\mathrm{con}} is a training decision. We would consider a procedure internalized if the trained model can reliably use it on new, relevant tasks without seeing the memory.

2. What does observation coverage measure, and what evidence do we have for consolidation? For each matched ALFWorld task input, we removed duplicate observation texts separately from the rollouts with and without memory. We then computed the fraction of unique observations seen with memory that had at least one match in the rollouts without memory for the same task. We counted a match when the cosine similarity between the text embeddings met or exceeded the threshold, then averaged these fractions across tasks. This metric measures semantic overlap in what the agent observes during rollout. To assess consolidation, we examine how the trained policy performs and what it does with retrieval disabled.

Figure 2&3 show that applying the same consolidation update only to memories rejected by the gate yields worse memory-free performance than applying it to the selected memories. The ALFWorld case study examines a specific behavior. With all retrieval disabled, CoMP placed the two objects one at a time, as described in a selected memory. The ablation shows that which memories are selected matters for memory-free performance, and the case study provides a concrete example of the trained policy using a memory-described procedure without access to the memory.

3. Motivation relative to Skill0. Thank you for asking us to clarify the motivation. Both methods aim to reduce reliance on external guidance. Skill0 uses the current policy’s performance with and without a skill to decide whether to keep providing that skill during rollouts. CoMP focuses on how the policy can learn from rollouts that external memory has already helped generate.

A memory can help the policy discover a useful behavior before the policy can perform it independently. CoMP uses the resulting trajectories as a training signal to explicitly encourage the policy to internalize that behavior. This lets memory serve two purposes during training: guiding exploration and providing experience the policy can learn from to act without retrieval.

Thank you for your response.

  1. In your response C1, you mentioned that all memory retrieval is turned off during evaluation. However, this still creates a train-inference mismatch. For queries assigned to "KEEP", memory should have been introduced to help solve these tasks, but it is not available during evaluation. This makes me feel that these queries have not been fully considered in the optimization loop. Therefore, CoMP still seems to have some limitations.

  2. Your explanation of memory consolidation is: "We would consider a procedure internalized if the trained model can reliably use it on new, relevant tasks without seeing the memory." However, the same goal can also be achieved by approaches like Skill0, which remove redundant or low-value skills. The comparison you mentioned in C3 still does not clearly explain the motivation for something that "only consolidation can achieve, but skill removal cannot."

  3. In your response C2, you consider the similarity between observations with and without memory, and then examine whether the actions in the two settings are similar, using this as a way to measure the degree of internalization. I agree with this setting as a proxy metric. From the results, it does demonstrate, to some extent, the effectiveness of CoMP.

Overall, given that CoMP still has some non-negligible issues in the logic of the proposed methodology, and that the motivation is not sufficiently supported, I believe the paper would benefit from further careful consideration and reorganization before it is ready for acceptance.

Thank you for recognizing the behavioral evidence. We would like to clarify the two remaining concerns.

KEEP updates and memory-free evaluation.

Tasks assigned to KEEP participate in policy optimization through LkeepL_{\mathrm{keep}}. The choice between KEEP and CONSOLIDATE is based on their estimated benefit for subsequent memory-free performance. KEEP is selected when retaining memory during that update is estimated to produce a better update for the memory-free policy. This decision is revisited as training progresses.

We disable all retrieval during evaluation, including memories assigned to KEEP, specifically to test what the policy has learned to do independently. Memory serves as training-time support, and its usefulness during training is compatible with evaluating the resulting policy without it. Figure 3 provides empirical support for selective updates, as selective consolidation achieves better memory-free performance than consolidating all memories.

Motivation relative to Skill0.

Skill0 relies on a helpfulness score to reassess whether a skill is still needed and gradually removes skills as training progresses. However, how much a skill helps the current policy during task execution does not directly measure how beneficial learning from its guided trajectories will be for subsequent memory-free performance. We therefore consider helpfulness alone insufficient for deciding which memories to consolidate.

CoMP explicitly encourages the policy to internalize behaviors from memory-guided rollouts. A memory may help the agent discover a useful procedure before it can execute that procedure independently. CoMP uses the resulting experience to promote this learning and selects updates according to their expected benefit for memory-free performance. Our comparison with Skill0 supports the effectiveness of the full method under the evaluated training-time budget, while our selection ablations demonstrate that the consolidation decision affects the resulting memory-free performance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.