- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the memory bottleneck in online in-context reinforcement learning, where Transformer policies require growing context windows for long-horizon adaptation. It proposes Recurrent Summary Memory (RSM), a dual-memory architecture combining a fixed-capacity long-term summary of learned latent tokens with a short-term working buffer. At each segment boundary, a causal Transformer overwrites the summary using learnable write queries that attend to the old summary and recent records. RSM achieves O(1) deployment state and per-decision computation in task length. Experiments on Dark Key-to-Door, CountRecall, MazeRunner, and T-Maze show that frozen RSM policies sustain adaptation over eight times the training horizon and re-adapt after task changes, while growing-memory baselines deteriorate. Causal interventions and ablations on summary content, write rules, and memory length support the design.
Critical strengths
1.The paper introduces a genuinely useful memory design for online in-context reinforcement learning. RSM combines a fixed-capacity summary with a short working buffer, and its overwrite rule keeps the deployment state and per-decision computation O(1) in task length. The theoretical bounds in Proposition 1 and Proposition 2 are simple but meaningful, and the long-running evaluations show that a frozen RSM policy sustains adaptation over eight times the training horizon and re-adapts after hidden task changes. This is consequential because it directly addresses a practical scalability bottleneck in ICRL and supports the claim with both analysis and experiments.
2.The causal analysis of summary use is unusually thorough and strengthens the paper’s scientific contribution. The authors clear, transplant, and block the summary, probe hidden task variables from it, and analyze attention to summary slots. These interventions show that the policy actually depends on summary content rather than merely carrying memory. The overwrite versus additive write comparison further explains why a bounded summary can remain stable where a residual summary drifts. This matters because it turns the memory mechanism from a plausible architectural choice into an empirically grounded and interpretable design.
Critical weaknesses
1.RSM’s memory design is conceptually close to hierarchical context reasoning or hierarchical memory methods, where long-term information is consolidated at a higher level and provided to lower-level short-term inference. It is suggest that the authors add a discussion of how RSM relates to and differs from this line of work, which would help clarify the novelty boundary of the dual-memory design.
2.The experimental setup and discussion of Figure 4(f) are not sufficiently clear. First, the explanation of “donor summary” is unclear: is the author trying to say that the original summary is replaced by the donor summary from the preceding sub-task of the Key-to-Door task (i.e., finding the key)? Second, the paper lacks an explanation for why a cleared summary performs better in this task.
Questions for authors
See weaknesses
3 comments
Dear Reviewer j9G4,
The authors have submitted their rebuttal and we are closing in on the end of the discussion period. Please take some time to carefully read over the rebuttal and respond whether your concerns have been addressed.
Best, AC
Reply to author
Thank you to the author for the response; my concerns have been addressed.
Hierarchical memory, and Figure 4(f)
We thank the reviewer for the positive and thoughtful review, and for recognising the usefulness of the memory design, its bounds and long-running evaluations, and the causal analysis. Both suggestions strengthen the paper, and we adopt them in full. To make the paper easier to follow, the revision will also open Section 3 with a step-by-step walk-through of one segment: each decision reads the summary and the segment's recent steps; at the segment's end, the write slots read both and produce a new summary, which replaces the old one for the next segment. The new experiments run for the other reviews, summarised in our comment to all reviewers, further support the design.
Weakness 1. How does RSM relate to hierarchical memory and hierarchical context methods?
Answer. We thank the reviewer for this suggestion, which helps state the novelty boundary. RSM can be read as the simplest two-level memory: exact attention over a short working buffer below, and a fixed set of summary tokens above that carries everything older. The paper already frames it this way (Sections 1 and 5, with Rae et al., 2020 and Ge et al., 2024); the revision will add the hierarchical methods the reviewer points to.
How RSM differs.
The novelty boundary: in RSM the summary tokens are the only route by which a segment can influence later decisions, they are rewritten at every boundary, and both levels are trained end to end by the RL objective.
Revision. Section 5 will discuss hierarchical memory and hierarchical in-context RL with these differences. If the reviewer has further works in mind, we will include them.
Weakness 2. What is the donor summary in Figure 4(f), and why does a cleared summary do better?
Answer. We thank the reviewer for raising this; the revision will make the panel self-explanatory. The donor is a different held-out Key-to-Door task, not an earlier sub-task of the same task. Figure 4(f) measures misdirection, how often the next attempt goes first to the donor's key, so lower is better there. A cleared summary does better than the donor's but not than the task's own: an empty summary sends the policy back to searching, while the donor's sends it to the wrong place.
Evidence.
When cleared, the policy climbs back towards its own level; with the donor's summary it stays misled.
Revision. The Figure 4 caption and Section 4.3 will define donor and recipient, state that lower is better in panel (f), add this explanation and point to Appendix F.3 for doors opened.
We thank the reviewer again for the careful reading and helpful suggestions, and hope these answers resolve both points; we would be glad to clarify anything further.