acceptodds
Desk rejected at ICLR 2027

Ask the Memory, Not the Teacher: Episode-Swap Gating Prevents Stale-Memory Hallucination in On-Policy Distillation of Agent Memory

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Agents that keep a memory of past episodes are increasingly consolidated into their weights. A teacher that reads a retrieved episode supervises, token by token, a student that does not, until the student no longer needs the store. We show that this on-policy consolidation writes more into the weights than it should. A retrieved episode mixes the procedure that solves a family of tasks with values that held only in that episode, such as an access code or the place where an object happened to be, and dense distillation transfers both. The consolidated student then states a past episode’s credential as a fact about a new user, with no retrieval record left to trace it. Standard token-selection signals miss this failure, because a teacher that copies from its context is confident, and the copied values can appear anywhere in a response. We propose to test each token against the memory instead of trusting the teacher. When the retrieved episode is replaced by a sibling episode of the same task family, the teacher’s predictions of procedural tokens stay the same while its predictions of episode-bound tokens change, and we use this difference as a per-token distillation gate that costs two extra teacher forward passes with one sibling. In a diagnostic environment where the provenance of every token is known, the gate separates the two kinds of tokens almost perfectly while confidence and position heuristics barely beat chance. During consolidation it eliminates stale-value emission at equal task success, nearly doubles honest abstention, and carries over to a second model family and to embodied agents in ALFWorld.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, agent memory, episodic memory, on-policy distillation, memory consolidation, counterfactual gating, embodied agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_am5QOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates stale-memory engraving in memory-augmented language-model agents, where on-policy distillation can transfer episode-specific information into model parameters. The proposed episode-swap gating mechanism separates transferable procedural knowledge from episode-bound content during distillation. Experiments on ProcWorld and ALFWorld demonstrate substantial reductions in stale-value emissions while preserving task performance, though evidence of generalization to realistic agent environments remains limited.

Critical strengths
  1. The paper identifies stale-memory engraving, where on-policy distillation transfers episode-specific information into memory-free agent weights. To address this, it proposes an intuitive token-level episode-swap gating mechanism that separates transferable procedural knowledge from episode-bound information using counterfactual teacher predictions. The approach tackles an important and underexplored problem in LLM-agent memory consolidation and offers a distinctive, interpretable, and broadly compatible distillation framework.

  2. The paper provides strong evidence on ProcWorld that episode-swap gating effectively reduces stale-memory contamination while preserving task performance. Across multiple models, it achieves high token-detection accuracy, eliminates observed exact-match stale emissions, and maintains comparable task success. The results, supported by baselines, ablations, and cross-model evaluations, demonstrate effective mitigation of stale-memory engraving without sacrificing procedural learning, though broader generalization remains to be established.

Critical weaknesses
  1. The proposed mechanism assumes procedural predictions remain stable across sibling episodes while episode-specific predictions vary. Although this assumption appears valid in ProcWorld, it may not hold in realistic settings where valid procedures differ across episodes. The observed performance degradation in ALFWorld before introducing a supervision floor further suggests that swap divergence may not reliably distinguish undesirable memorization from legitimate context-dependent behavior.

  2. While the ALFWorld results are promising, the reported gains are not statistically conclusive, as the 95% bootstrap confidence interval for the performance difference includes zero. Additionally, the mismatched-retrieval robustness study uses only a single seed per condition, and the natural-language ProcWorld variant retains a synthetic task structure. Therefore, the current evidence does not yet demonstrate reliable prevention of stale-memory engraving in realistic, heterogeneous agent environments.

  3. The method depends on sibling episodes sharing procedural knowledge while differing in episode-specific information. When sibling memories contain similar or correlated values, swap divergence may fail to identify undesirable information, as shown by the homogeneous-swap ablation. Additionally, although provenance labels are not needed during distillation, the gate requires calibration using labeled development data, and the weaker performance of the untuned gate indicates that such calibration is important.

  4. The evaluated baselines are mainly heuristic token-weighting methods, while stronger alternatives such as provenance-aware filtering, counterfactual consistency supervision, and oracle token masking are not considered. Also, the exact-match stale-emission metric captures only direct value reproduction and may overlook paraphrased or semantically equivalent leakage, limiting the strength of the memorization-prevention claims.

Questions for authors
  1. How does episode-swap gating perform when sibling episodes from the same task family require different valid procedures? Can the authors systematically evaluate procedural variation and quantify instances where useful supervision is incorrectly suppressed? Whether adaptive supervision floors or procedure-aware sibling selection improve robustness compared to the fixed floor used in ALFWorld?

  2. To what extent does the method depend on labeled development data for gate calibration and on diversity among sibling memories? Can the authors explore fully unsupervised calibration and evaluate robustness when episode-specific values are highly correlated or repeatedly retrieved across memories?

  3. Could the authors strengthen the ALFWorld evaluation with additional independent seeds and direct measurements of stale-action transfer? It would also be valuable to isolate the effect of honest-abstention warmup and compare against stronger provenance-aware filtering baselines using leakage metrics that go beyond exact-match evaluation.

1 comment

The rebuttal deals with all of the concerns via a number of procedural variation experiments, diversity filtering, stronger baselines, more ALFWorld seeds and broader leakage audits. Most of the ProcWorld additions are still based on a single seed. The ALFWorld effects are still pretty mixed and largely statistically non-significant. Label dependent gate calibration is still required. There's still very little in the way of confirmed ALFWorld leakage so we don't have any real evidence of practical mitigation in a real world setting. I therefore stand by my rating.

Reviewer_6qXDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Once in-context learning was recognized as a kind of learning, the question of how to move back and forth from in-context learning and in-weights learning became both interesting and important. Once a model is capable enough to learn from successful examples, a natural question became whether that same capability (or even better!) could be achieved via weight adaptation, e.g. via a LoRA. Now, in general, the community believes that for the same number of parameters, hard prompts are worse than soft prompts which are in turn worse than LoRAs. At considerable length, hard prompts and soft prompts are also expensive at inference time via attention and other computations. Consequently, having the learning done by a LoRA is attractive. Teacher/student distillation is a natural way to do this, where the teacher is just the model with a longer prompt and the student is the LoRA-augmented model.

Few-shot prompting is inherently underspecified. Which details are actually truly relevant to the task and which are idiosyncratic to the specific problem being solved --- that's hard to pull apart from a single solved example. That has been well understood. The consequence for this kind of finetuning is what the paper gives a memorable name: stale-memory engraving. The teacher itself is likely to copy irrelevant aspects of the training exemplars into the problem it is trying to solve --- so it is no surprise (in hindsight) that this might also happen to the student. Those "stale memories" have been engraved into the student.

This is a problem and that's the problem that the paper is addressing.

The core insight of the paper is that when one has multiple training examples, each of which might be enough to solve the problem, then this can help us understand and mitigate this problem.

In a sense, they discover a kind of process reward model that comes from the prompted teacher --- or more accurately, from multiple copies of differently prompted teachers. Whereas a naive perspective might view only the teacher's confidence in the tokens as containing relevant signal, the paper realizes that the disagreement between differently prompted teachers can point to which tokens are actually irrelevant information. By leveraging this, the paper is able improve LoRA training for the student by reducing the importance of those tokens in the on-policy generations.

The authors use a specially designed toy problem to isolate this issue and show improvements. They then adapt the technique to a more standard benchmark (ALFWorld) and show the gains hold up.

Critical strengths

The problem is important and I would categorize this into the general category of context-enhanced learning. The investigation feels systematic.

The ablations are natural and seem to make the case, within computational limitations.

Critical weaknesses

The writing is quite bad in many places, including the abstract. I almost stopped reading at the first sentence of the abstract --- that sentence has really bad grammar and does not make sense. When the first sentence of the abstract has problems, you know you're in for a rough read. Thankfully, things get a bit better reading wise. However, nowhere does the paper lay out as clean of a story for what is going on as I do in my overview. Furthermore, the citations of prior work in this area are quite poor. Citing a bunch of 2026 papers for consolidating memory into weights is quite strange for a stream of work that dates back far longer --- I didn't use any AI here on purpose but I am pretty sure that a good AI would tell you to look at the work of Snell for example, as well as the context-enhanced learning literature since then.

The philosophical connection to process reward models is also not made clear. And the role of counterfactuals here seems clearly to have a conceptual intellectual family tree that connects back to DPO style work.

Questions for authors
  1. If instead of doing the gating, I just make sure that different prompts for the same task are included together within single batches, how much of these gains do I get?

  2. Why is this written so badly? You've got a good idea here. Is this how you would teach it to a human? I am sure that superhuman LLMs are capable of understanding this, but what happened here? Was this trying to compress to fit in 10 pages? Were you afraid that if you told the story clearly people would dismiss you has obvious? Or did an AI mangle this writeup? Or is this actually how you think about the problem?

  3. The literature review is bad to the point that it is making me lower my rating from a 3 to a 2. What is the actual relevant background for this work? I am willing to raise my score if I have confidence that the final paper will actually teach people how to connect the question here to the literature.

Reviewer_hXssOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates stale-memory engraving in on-policy memory consolidation for language-model agents. The authors identify a failure mode where a teacher conditioned on retrieved episodic memory transfers not only generalizable procedural knowledge but also episode-specific information, such as credentials or object locations, into a memory-free student's parameters through dense distillation.

To address this issue, the paper proposes Episode-Swap Gating, a token-level distillation weighting method based on counterfactual memory interventions. Specifically, the method compares the teacher's predictions under the original retrieved episode, an empty memory context, and sibling episodes from the same task family. The resulting deletion and swap divergences are used to distinguish transferable procedural knowledge from potentially stale episode-specific information and to construct token-level distillation weights.

The authors introduce ProcWorld, a synthetic benchmark with token-level provenance annotations, and evaluate the method across multiple model configurations. The proposed approach achieves strong episode-bound token detection performance and reduces stale-value emission from 16.9% to 0.0% while maintaining comparable task success. Additional experiments on ALFWorld investigate its applicability to embodied agents, including robustness under mismatched memory retrieval.

Critical strengths

1. Important and well-motivated problem formulation.

The paper identifies a meaningful failure mode in memory consolidation: distilling a memory-conditioned teacher can unintentionally transfer episode-specific information into the student's parameters. This problem is particularly relevant to agents that aim to internalize procedural knowledge while reducing their dependence on external memory. The distinction between transferable procedural knowledge and stale episode-specific information provides a useful perspective on the reliability of memory consolidation.

2. Intuitive methodology and strong results in a controlled setting.

Episode-Swap Gating is conceptually simple and can be integrated into existing distillation objectives through token-level weighting without modifying the model architecture. The proposed counterfactual comparison provides a clear motivation for distinguishing procedural knowledge from episode-specific information.

The experiments on ProcWorld demonstrate strong token-level detection performance, with swap-based AUROC values above 0.96 across the evaluated settings. The reduction of stale emission from 16.9% to 0.0%, while maintaining comparable task success, is encouraging. The inclusion of multiple model configurations, ablations, and an embodied-agent evaluation further strengthens the empirical investigation.

Critical weaknesses

1. Strong reliance on the episode-invariance assumption, which is closely aligned with the synthetic benchmark design.

The central assumption underlying Episode-Swap Gating is that procedural knowledge remains stable across sibling episodes, while episode-specific information changes. However, this assumption is largely guaranteed by the construction of ProcWorld, where episodes within the same task family share a fixed procedure but contain different episode-specific values.

Consequently, although the near-perfect detection performance on ProcWorld is encouraging, it remains unclear how well the proposed criterion generalizes to settings where different episodes of the same task family legitimately involve different procedures, strategies, or actions.

This concern is also reflected in the ALFWorld experiments. The original gate can suppress useful procedural supervision because legitimate actions vary across scenes, leading to substantial performance degradation in the reported preliminary evaluations. The authors introduce an additional minimum supervision weight to address this issue, suggesting that the original criterion does not directly generalize beyond the assumptions of the synthetic environment.

This limits the strength of the paper's broader claim that episode-swap sensitivity can reliably distinguish transferable knowledge from undesirable episode-specific information.

2. Limited evidence of effectiveness in more realistic agent environments.

While the proposed method achieves strong results on ProcWorld, the evidence from ALFWorld is considerably less conclusive.

With the modified floored gate, the mean success rate improves from 38.5% to 42.7%, but the paired per-seed differences are +5.2, -4.2, and +11.5 percentage points. The reported 95% bootstrap interval for the mean paired improvement is [-1.7, 10.1], which includes zero. Therefore, the results do not establish a statistically reliable improvement in task success.

Furthermore, the mismatched-retrieval experiment, which provides the strongest evidence for robustness against stale memories in ALFWorld, is evaluated using only a single seed. The authors also acknowledge that their embodied stale-visit metric cannot reliably distinguish memorization of a specific historical episode from reasonable semantic priors.

As a result, it remains difficult to determine whether the method effectively prevents stale-memory engraving in realistic agent settings, rather than primarily benefiting from the controlled structure of ProcWorld. Stronger evidence beyond the synthetic benchmark is important for substantiating the practical significance of the proposed approach.

Questions for authors

Q1. Generalization beyond the episode-invariance assumption.

The proposed method assumes that procedural knowledge remains invariant across sibling episodes, whereas episode-specific information changes. This assumption is well aligned with the construction of ProcWorld, but may not hold in realistic agent environments where different episodes of the same task family can legitimately require different procedures or actions.

Could the authors provide further analysis or experiments examining how Episode-Swap Gating behaves when procedural knowledge varies across sibling episodes? In particular, how can the method distinguish legitimate procedural variation from undesirable episode-specific information?

It would also be helpful to understand how sensitive the method is to the availability, diversity, and quality of sibling episodes, especially when suitable siblings are difficult to identify or share similar stale information.

Q2. Stronger evidence of effectiveness in realistic agent environments.

The results on ProcWorld are promising, particularly the elimination of measured stale-value emission without sacrificing task success. However, the evidence on ALFWorld appears less conclusive. The improvement in success rate is not statistically significant, and the mismatched-retrieval robustness experiment is based on a single seed.

Could the authors provide additional paired-seed results under mismatched retrieval or further evidence that the proposed method consistently reduces stale-memory transfer in embodied-agent settings?

In particular, given that the original gate requires a supervision floor to avoid suppressing useful actions in ALFWorld, it would be helpful to clarify how much of the observed robustness improvement comes from the episode-swap criterion itself and how much comes from the additional flooring mechanism.

Addressing these concerns would materially affect my assessment of the generality and practical significance of the proposed method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.