- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes COMP (Consolidation of Memory into Parameters), which addresses when an external memory should be internalized into model weights. Its key distinction is that a memory may help the current rollout yet still induce an undesirable or redundant parameter update. COMP therefore compares a memory-retained update against a memory-removed “consolidation” update and selects consolidation when the latter is better aligned with improving the eventual memory-free policy.
Critical strengths
- The problem formulation is strong: distinguishing “useful to retrieve” from “worth consolidating” is meaningful, timely, and more precise than simply training on all successful memory-guided behavior.
Critical weaknesses
- Ambiguity in the central gate: The practical memory-free replay gradient is computed from the same memory-guided trajectories with the memory removed, which appears very close to the defined consolidation gradient itself. If these are in fact the same objective, the gate has a substantially different interpretation from alignment with an independently estimated memory-free improvement direction. The paper needs to make that distinction mathematically explicit.
Questions for authors
s the replay-based actually different from in implementation? If so, what exactly differs in the loss, importance ratios, reference policy, or weighting? If they are identical, I would want the paper to restate the interpretation of the gate accordingly.
Follow-up response to Reviewer FXMG
Thank you for the follow-up questions and for pointing out where our earlier explanation was unclear.
1. What happens to the memory, and what counts as consolidation? Both losses in Eq. (11) use the same trajectories collected with memory. With Lkeep, the model sees the retrieved memory when we compute the update. With Lcon, we leave it out of the model input for that update. The memory stays in the bank and can still guide later training rollouts. At evaluation, we turn off all memory retrieval, including for memories assigned to KEEP. Choosing Lcon is a training decision. We would consider a procedure internalized if the trained model can reliably use it on new, relevant tasks without seeing the memory.
2. What does observation coverage measure, and what evidence do we have for consolidation? For each matched ALFWorld task input, we removed duplicate observation texts separately from the rollouts with and without memory. We then computed the fraction of unique observations seen with memory that had at least one match in the rollouts without memory for the same task. We counted a match when the cosine similarity between the text embeddings met or exceeded the threshold, then averaged these fractions across tasks. This metric measures semantic overlap in what the agent observes during rollout. To assess consolidation, we examine how the trained policy performs and what it does with retrieval disabled.
Figure 2&3 show that applying the same consolidation update only to memories rejected by the gate yields worse memory-free performance than applying it to the selected memories. The ALFWorld case study examines a specific behavior. With all retrieval disabled, CoMP placed the two objects one at a time, as described in a selected memory. The ablation shows that which memories are selected matters for memory-free performance, and the case study provides a concrete example of the trained policy using a memory-described procedure without access to the memory.
3. Motivation relative to Skill0. Thank you for asking us to clarify the motivation. Both methods aim to reduce reliance on external guidance. Skill0 uses the current policy’s performance with and without a skill to decide whether to keep providing that skill during rollouts. CoMP focuses on how the policy can learn from rollouts that external memory has already helped generate.
A memory can help the policy discover a useful behavior before the policy can perform it independently. CoMP uses the resulting trajectories as a training signal to explicitly encourage the policy to internalize that behavior. This lets memory serve two purposes during training: guiding exploration and providing experience the policy can learn from to act without retrieval.