- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates stale-memory engraving in memory-augmented language-model agents, where on-policy distillation can transfer episode-specific information into model parameters. The proposed episode-swap gating mechanism separates transferable procedural knowledge from episode-bound content during distillation. Experiments on ProcWorld and ALFWorld demonstrate substantial reductions in stale-value emissions while preserving task performance, though evidence of generalization to realistic agent environments remains limited.
Critical strengths
-
The paper identifies stale-memory engraving, where on-policy distillation transfers episode-specific information into memory-free agent weights. To address this, it proposes an intuitive token-level episode-swap gating mechanism that separates transferable procedural knowledge from episode-bound information using counterfactual teacher predictions. The approach tackles an important and underexplored problem in LLM-agent memory consolidation and offers a distinctive, interpretable, and broadly compatible distillation framework.
-
The paper provides strong evidence on ProcWorld that episode-swap gating effectively reduces stale-memory contamination while preserving task performance. Across multiple models, it achieves high token-detection accuracy, eliminates observed exact-match stale emissions, and maintains comparable task success. The results, supported by baselines, ablations, and cross-model evaluations, demonstrate effective mitigation of stale-memory engraving without sacrificing procedural learning, though broader generalization remains to be established.
Critical weaknesses
-
The proposed mechanism assumes procedural predictions remain stable across sibling episodes while episode-specific predictions vary. Although this assumption appears valid in ProcWorld, it may not hold in realistic settings where valid procedures differ across episodes. The observed performance degradation in ALFWorld before introducing a supervision floor further suggests that swap divergence may not reliably distinguish undesirable memorization from legitimate context-dependent behavior.
-
While the ALFWorld results are promising, the reported gains are not statistically conclusive, as the 95% bootstrap confidence interval for the performance difference includes zero. Additionally, the mismatched-retrieval robustness study uses only a single seed per condition, and the natural-language ProcWorld variant retains a synthetic task structure. Therefore, the current evidence does not yet demonstrate reliable prevention of stale-memory engraving in realistic, heterogeneous agent environments.
-
The method depends on sibling episodes sharing procedural knowledge while differing in episode-specific information. When sibling memories contain similar or correlated values, swap divergence may fail to identify undesirable information, as shown by the homogeneous-swap ablation. Additionally, although provenance labels are not needed during distillation, the gate requires calibration using labeled development data, and the weaker performance of the untuned gate indicates that such calibration is important.
-
The evaluated baselines are mainly heuristic token-weighting methods, while stronger alternatives such as provenance-aware filtering, counterfactual consistency supervision, and oracle token masking are not considered. Also, the exact-match stale-emission metric captures only direct value reproduction and may overlook paraphrased or semantically equivalent leakage, limiting the strength of the memorization-prevention claims.
Questions for authors
-
How does episode-swap gating perform when sibling episodes from the same task family require different valid procedures? Can the authors systematically evaluate procedural variation and quantify instances where useful supervision is incorrectly suppressed? Whether adaptive supervision floors or procedure-aware sibling selection improve robustness compared to the fixed floor used in ALFWorld?
-
To what extent does the method depend on labeled development data for gate calibration and on diversity among sibling memories? Can the authors explore fully unsupervised calibration and evaluate robustness when episode-specific values are highly correlated or repeatedly retrieved across memories?
-
Could the authors strengthen the ALFWorld evaluation with additional independent seeds and direct measurements of stale-action transfer? It would also be valuable to isolate the effect of honest-abstention warmup and compare against stronger provenance-aware filtering baselines using leakage metrics that go beyond exact-match evaluation.
The rebuttal deals with all of the concerns via a number of procedural variation experiments, diversity filtering, stronger baselines, more ALFWorld seeds and broader leakage audits. Most of the ProcWorld additions are still based on a single seed. The ALFWorld effects are still pretty mixed and largely statistically non-significant. Label dependent gate calibration is still required. There's still very little in the way of confirmed ALFWorld leakage so we don't have any real evidence of practical mitigation in a real world setting. I therefore stand by my rating.