acceptodds
Under review as a conference paper at ICLR 2027

Remember Less, Adapt Longer: Bounded-State In-Context Reinforcement Learning with Recurrent Summary Memory

Abstract

In-context reinforcement learning (ICRL) enables agents to adapt to new tasks without parameter updates, relying on memory to retain what they learn from experience. For agents operating continually, existing approaches using memory trade off bounded cost against access to past experience. Approaches using growing memories, such as full-history attention and accumulating summaries, incur storage and per-decision costs that increase with task length. Fixed-size memories bound storage but restrict access: sliding windows discard older steps, while recurrent networks compress even recent interactions into a hidden state. To avoid this trade-off, we introduce Recurrent Summary Memory (RSM), a Transformer policy combining exact attention over a short-term working buffer with a fixed-capacity long-term summary of earlier experience. When the buffer fills, RSM rewrites the summary from its previous contents and buffered interactions, then clears the buffer, keeping deployment memory and per-decision cost in task length. On long-running partially observable benchmarks, frozen RSM policies sustain adaptation performance for eight times the training horizon using about 1% of full-history memory, and outperform all evaluated baselines at the final horizon, with a widening lead over growing memories. When the underlying task configuration changes mid-run, RSM adapts again and outperforms growing-memory and recurrent baselines. The policy relies on the summary: erasing it or blocking policy access removes its benefit, while swapping in another task's summary steers behaviour toward that task. Together, these results show that our fixed-capacity summary can retain task knowledge far beyond the training horizon.

Primary area:
reinforcement learning
Keywords:
in-context reinforcement learning, meta-reinforcement learning, memory, recurrent memory transformer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_j9G4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses the memory bottleneck in online in-context reinforcement learning, where Transformer policies require growing context windows for long-horizon adaptation. It proposes Recurrent Summary Memory (RSM), a dual-memory architecture combining a fixed-capacity long-term summary of learned latent tokens with a short-term working buffer. At each segment boundary, a causal Transformer overwrites the summary using learnable write queries that attend to the old summary and recent records. RSM achieves O(1) deployment state and per-decision computation in task length. Experiments on Dark Key-to-Door, CountRecall, MazeRunner, and T-Maze show that frozen RSM policies sustain adaptation over eight times the training horizon and re-adapt after task changes, while growing-memory baselines deteriorate. Causal interventions and ablations on summary content, write rules, and memory length support the design.

Critical strengths

1.The paper introduces a genuinely useful memory design for online in-context reinforcement learning. RSM combines a fixed-capacity summary with a short working buffer, and its overwrite rule keeps the deployment state and per-decision computation O(1) in task length. The theoretical bounds in Proposition 1 and Proposition 2 are simple but meaningful, and the long-running evaluations show that a frozen RSM policy sustains adaptation over eight times the training horizon and re-adapts after hidden task changes. This is consequential because it directly addresses a practical scalability bottleneck in ICRL and supports the claim with both analysis and experiments.

2.The causal analysis of summary use is unusually thorough and strengthens the paper’s scientific contribution. The authors clear, transplant, and block the summary, probe hidden task variables from it, and analyze attention to summary slots. These interventions show that the policy actually depends on summary content rather than merely carrying memory. The overwrite versus additive write comparison further explains why a bounded summary can remain stable where a residual summary drifts. This matters because it turns the memory mechanism from a plausible architectural choice into an empirically grounded and interpretable design.

Critical weaknesses

1.RSM’s memory design is conceptually close to hierarchical context reasoning or hierarchical memory methods, where long-term information is consolidated at a higher level and provided to lower-level short-term inference. It is suggest that the authors add a discussion of how RSM relates to and differs from this line of work, which would help clarify the novelty boundary of the dual-memory design.

2.The experimental setup and discussion of Figure 4(f) are not sufficiently clear. First, the explanation of “donor summary” is unclear: is the author trying to say that the original summary is replaced by the donor summary from the preceding sub-task of the Key-to-Door task (i.e., finding the key)? Second, the paper lacks an explanation for why a cleared summary performs better in this task.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Hierarchical memory, and Figure 4(f)

We thank the reviewer for the positive and thoughtful review, and for recognising the usefulness of the memory design, its bounds and long-running evaluations, and the causal analysis. Both suggestions strengthen the paper, and we adopt them in full. To make the paper easier to follow, the revision will also open Section 3 with a step-by-step walk-through of one segment: each decision reads the summary and the segment's recent steps; at the segment's end, the write slots read both and produce a new summary, which replaces the old one for the next segment. The new experiments run for the other reviews, summarised in our comment to all reviewers, further support the design.

Weakness 1. How does RSM relate to hierarchical memory and hierarchical context methods?

Answer. We thank the reviewer for this suggestion, which helps state the novelty boundary. RSM can be read as the simplest two-level memory: exact attention over a short working buffer below, and a fixed set of summary tokens above that carries everything older. The paper already frames it this way (Sections 1 and 5, with Rae et al., 2020 and Ge et al., 2024); the revision will add the hierarchical methods the reviewer points to.

How RSM differs.

  • HCAM (Lampinen et al., NeurIPS 2021) stores every past chunk with a summary and retrieves detailed chunks through their summaries, so its memory grows with the episode. RSM stores nothing: its top level is overwritten, and its state is constant in task length.
  • The Compressive Transformer (Rae et al., ICLR 2020) keeps a compressed memory of older activations beside its recent memory, trained with an auxiliary reconstruction loss. RSM's summary is learned from task reward alone, through every write.
  • Hierarchical in-context RL, such as IDT (Huang et al., ICML 2024), places the hierarchy over decisions, with high-level decisions guiding low-level actions, rather than over memory.

The novelty boundary: in RSM the summary tokens are the only route by which a segment can influence later decisions, they are rewritten at every boundary, and both levels are trained end to end by the RL objective.

Revision. Section 5 will discuss hierarchical memory and hierarchical in-context RL with these differences. If the reviewer has further works in mind, we will include them.

Weakness 2. What is the donor summary in Figure 4(f), and why does a cleared summary do better?

Answer. We thank the reviewer for raising this; the revision will make the panel self-explanatory. The donor is a different held-out Key-to-Door task, not an earlier sub-task of the same task. Figure 4(f) measures misdirection, how often the next attempt goes first to the donor's key, so lower is better there. A cleared summary does better than the donor's but not than the task's own: an empty summary sends the policy back to searching, while the donor's sends it to the wrong place.

Evidence.

  • The swap. At one segment boundary (decision 257) we replace a task's four summary tokens with the summary that the same frozen policy holds at that boundary in another task, with a different key and door; buffer and weights are unchanged. The cleared control resets the summary once, at the same boundary, to the learned initial summary.
  • Where the next attempt goes. It visits the donor's key cell before its own key in 83% of eligible tasks after the swap, against 22% after clearing and 10% with the task's own summary (tasks whose two keys coincide are excluded).
  • Doors opened afterwards. Doors per 50 steps, with the summary replaced at step 257 of the 500-step task (Appendix F.3):
Summary from step 257 onSteps 251–300 (swap at 257)Steps 351–400Steps 451–500 (end of task)
The task's own (unchanged)3.54.04.2
Cleared once0.81.93.0
The donor's0.60.50.7

When cleared, the policy climbs back towards its own level; with the donor's summary it stays misled.

  • Why clearing recovers. The initial summary is what the policy reads at the start of every training task, so it behaves as at a task's start: it explores, finds the key and door again and rewrites its summary. The donor's summary describes another layout, and since no training task ever contradicts its own summary, the policy has not learned to discard it. This is the same effect as a stale summary after a layout change, which our reply to Reviewer uSVn reports as a limitation.

Revision. The Figure 4 caption and Section 4.3 will define donor and recipient, state that lower is better in panel (f), add this explanation and point to Appendix F.3 for doors opened.

We thank the reviewer again for the careful reading and helpful suggestions, and hope these answers resolve both points; we would be glad to clarify anything further.

Dear Reviewer j9G4,

The authors have submitted their rebuttal and we are closing in on the end of the discussion period. Please take some time to carefully read over the rebuttal and respond whether your concerns have been addressed.

Best, AC

Reviewer_j9G4OpenReview ↗

Reply to author

Thank you to the author for the response; my concerns have been addressed.

Reviewer_uSVnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper discusses continuous in-context reinforcement learning with recurrent summary memory for agents, where an agent adapts to unseen tasks from its interaction history without parameter updates. Existing memory approaches either grow linearly with the horizon at deployment or compress the context into a lossy fixed state. The proposed Recurrent Summary Memory (RSM) is a Transformer policy that pairs exact causal attention over a short working buffer with a small fixed-capacity learned summary, overwritten at each segment boundary by a learned projection. According to the authors, this yields constant deployment state and constant per-decision compute, and the model is trained end-to-end with an off-policy actor-critic learner via backpropagation through time across boundaries, without auxiliary losses. On several partially observable benchmarks for empirical study, RSM matches full-history Transformers and a recent memory baseline at the training horizon and outperforms them when the deployment horizon is extended well beyond training or when the task changes mid-run, settings where growing memories degrade under context distribution shift. The design is clean and the long-horizon robustness claim is pretty interesting, but the contribution is incremental relative to prior literature and the evidence is a bit narrow.

Critical strengths

The problem is well motivated and the solution is clean. The bounded-state objective is clear and reasonable, with a proposed architecture that is based on understandable changes to a standard causal transformer; besides, the efficient deployment claim is proved with explicit byte accounting. The causal evidence is thorough. The authors conduct thorough diagnostic interventions, including summary resetting, cross-task summary transplants, attention score masking, and linear probes, proving that the policy causally relies on stored summary representations to retain task-critical hidden variables Frozen RSM policies maintain high performance over 8x training horizon, whereas in contrast the growing-memory baselines experience severe performance degradation due to length distribution shift

Critical weaknesses

The scale is a bit small and generality is a bit unclear. The backbone is 3 layers of moderate sizes of parameters; tasks are at most 500 steps; benchmarks are pretty much grid worlds, a card-counting stream, and a corridor, with low-dimensional vector observations. The abstract and introduction speak of robots and long-running services, but there is no visual, continuous-control, or long-horizon embodied result. Baseline strength and attribution of the gain are not very convincing. Memo is re-implemented on a different backbone/learner/task, so its collapse may not always reflect the method, as the port/migration can also matter. More importantly, the closest conceptual competitors for the bounded state are discussed in related work only. The only bounded baselines actually compared are a GRU and a sliding window. When environment layouts change mid-deployment, RSM's carried summary can act as an outdated prior. In Key-to-Door layout changes, RSM fails to solve some subsequent layouts because lingering task memory hinders re-exploration relative to a memoryless baseline. The core design claims that a fixed-capacity summary is enough, but this is only weakly stressed by the benchmarks. The summary-length ablation finds little consistent difference with respect to M values on Key-to-Door in Table 3. Since no benchmark saturates the summary, the paper cannot tell the reader where capacity starts to matter, which could be a central question to this bounded-memory paper. There is some concern on slower learning convergence and high sample complexity in the proposed method. On Dark Key-to-Door, RSM exhibits slower training convergence than full-history and Memo baselines, requiring 4.4~5.2M steps to reach performance thresholds, compared to less than 3.6M steps for baselines.

Questions for authors

n/a

8 comments
AuthorsOpenReview ↗

what we did for each concern, and the contribution (Part 1)

We thank the reviewer for a careful and constructive review, and for recognising the clean design, the explicit state accounting, the thorough causal evidence and the long-horizon result. We put the concerns as eight questions and answered each with new experiments under the paper's protocol (three seeds, 256 held-out tasks, paired 95% intervals). In short, RSM's long-horizon gain comes from its bounded summary, not from a weak port of Memo (Gupta et al., NeurIPS 2025) or a missing competitor; capacity is now measured on every task; and the later start on Key-to-Door has a measured cause.

At a glance (details in parts 1–5; Tables R1 and R4 are in our reply to Reviewer Z9eu).

QuestionWhat we didWhat we foundWhat we will change
1. Is the contribution incremental? (part 1)Placed RSM among memory-token methods in RLA fixed-size memory keeps an agent adapting far beyond its training horizon, where growing memories fall backSection 5 places RSM beside RATE
2. Could Memo's collapse come from our port? (part 2)Four checks; both Memo forms; Streaming MemoNo. Memo is strong at the training horizon and recovers when restarted, but every form falls behind RSM beyond itChecks in this thread; Streaming Memo in Tables 1 and 9
3. How does RSM compare with the closest bounded memory? (part 2)Added RATE (Cherepanov et al., ICLR 2026), with and without its cacheNo consistent difference at the training horizon. Beyond it, RSM leads on MazeRunner by 7.8–13.1 goals, on every seedResults in the appendix; Section 5 discusses RATE
4. Is the design sound? (part 3)Tested each choice against an alternativeFor example, without gradients through the write, CountRecall falls from 98.9% to 91.4%Tests in Section 4.4 and the appendix
5. Is the evaluation too small? (part 3)Ran RSM for 32,000 steps, 64 times its training horizonIn run length, no: RSM keeps its performance at constant memory. In model size and task variety, yesStated as a limitation; robots as motivation only
6. Can the carried summary hurt after a layout change? (part 4)Restarts at announced changes; RATE's cacheSometimes: on Key-to-Door RSM finds a new door less often than without memory (34% vs 72%), yet completes more doors per layout (9.4 vs 1.9; at most 1.6 for the full history, GRU and Memo)Claim narrowed; limitation in Section 6
7. Where does a fixed capacity start to matter? (part 4)One token; how much of the summary is used; a trained bottleneckPolicies use at most about 3% of the summary. On CountRecall, passing one number to the next segment cuts accuracy from 98.7% to 77.3%; four numbers give 94.5%Table R5 and the sweep in the appendix
8. Does RSM learn more slowly? (part 5)Scored each checkpoint with the summary carried and clearedIt starts later, as it must first learn what to keep, then climbs about as fast as the full history and GRUAppendix F.8; Section 6 says "starts learning later"

Carried: memory runs on as trained, with no boundary signal. Restarted: memory cleared and positions reset at an announced boundary (Table R4).

Question 1. Is the contribution incremental?

Answer. The contribution is to show that a fixed-size memory can keep an agent that learns online adapting far beyond its training horizon, where memories that grow fall back, and why. The mechanism itself is established, and Section 3 credits it: in RL it has been used offline (RATE) and as a fixed-size baseline in the Memo paper, whose own method keeps every summary and grows (Section 5). Prior work leaves open what such a memory does far beyond the training horizon, which four findings now answer.

  • It keeps adapting where memories that grow fall back. At 4,000 steps RSM reaches 26.6 goals per 500 steps on one MazeRunner map, against 0.8 for the full history and 1.8 for Memo (Table 1). It still opens 46.7 doors per 500 steps at 32,000 steps on Key-to-Door, and restarting the baselines does not close the gap on a fixed layout (Table R4).
  • Little memory is needed. Four tokens, trained with no auxiliary loss, are competitive with the full history at the training horizon, and each policy uses at most about 3% of their 1,024 dimensions (part 4).
  • Decisions use what the summary holds. With another layout's summary swapped in, the next attempt goes first to that layout's key in 83% of tasks, against 10% with the task's own summary (Section 4.3).
  • Design rules for bounded memory. Replacing the summary keeps it stable where adding to it drifts (part 5); RATE's retention valve costs 5.7 [1.0, 10.4] doors at the training horizon when added to RSM (part 2); a summary carried through an unannounced change can mislead, and a short cache of recent records helps (part 4).

Revision (updated 15 Nov). Section 5 places RSM beside RATE and hierarchical memories.

AuthorsOpenReview ↗

Could Memo's collapse come from our port, and how does RSM compare with the closest bounded memory? (Part 2)

Question 2. Could Memo's collapse come from our port?

Answer. Four results point to the method rather than the port. We ported Memo from the authors' released code (Appendix A.1) in two forms: as published, with random segment lengths in training, and with fixed segments. Both share the common backbone, learner, tasks and recipe. The evidence:

  • Strong where it was trained. At the training horizon Memo is among our strongest baselines: it matches RSM on Key-to-Door (29.6 against 29.1 doors) and is ahead on MazeRunner (86.3% against 80.4% of goals).
  • A decline exactly where it exceeds its trained length. On Key-to-Door its rate rises to 2.5 doors per 32 steps by step 480, then falls to 1.3 by steps 513–544, just past the 500-step length of its training tasks.
  • Recovery when restarted. On one MazeRunner map the same weights collapse to 1.8 goals per 500 steps when carried, but reach 26.9 when restarted at every attempt, the best of any restarted memory (Table R4).
  • Both forms trail RSM beyond the horizon. With fixed segments Memo holds up better on Key-to-Door (31.4 doors at 4,000 steps, against 10.6 as published), yet RSM leads it there by 14.6 [10.0, 18.8] doors, and by 23.8 [20.1, 27.5] goals on one MazeRunner map (Table 8).

The Memo paper reports the same kind of limit in its own setting: beyond about 2.5 times its training context, Memo and the full-context Transformer both degrade "due to limitations in context length extrapolation" (its Section 4.3). Our runs go to 8 times, well past that point. Streaming Memo, the Memo paper's own rule for long streams, does not remove the limit either (Table R1: 7.0–12.8 doors against 46.0).

Revision (updated 15 Nov). These checks stay in this reply rather than in Appendix A.1; Streaming Memo is in Tables 1 and 9.

Question 3. How does RSM compare with the closest bounded memory?

Answer. We added RATE, with and without its cache; without it, RATE keeps the same state as RSM, the like-for-like comparison. Neither is consistently better at the training horizon; beyond it, RSM is the stronger on MazeRunner, and RATE's cache, at 1.8 times the state, helps it after a Key-to-Door layout change. Of the bounded memories in Section 5, RATE is built most like RSM, carrying memory tokens from segment to segment, while state-space models (Lu et al., NeurIPS 2023), gated linear attention (Pramanik et al., TMLR 2024) and Stable Hadamard Memory (Le et al., ICLR 2025) keep a fixed recurrent state, as the GRU does. RATE adds a retention valve, an extra attention step in which the old memory reads the new write, and a cache of the previous segment's records. We ported it from the released code into our backbone and online learner, with our segment-local positions and causal mask, and trained it on all four benchmarks under an analysis fixed in advance.

Table R3. RATE at the training horizon (as in Table 1) and over 4,000 steps (as in Table R1).

At the training horizonKD doorsCountRecall (%)T-Maze (%)MazeRunner goals (%)
RSM29.198.910080.4
RATE without cache (same state as RSM)28.898.081.979.5
RATE with cache (1.8× RSM's state)31.099.510072.0
Over 4,000 stepsKD, one layoutKD, per later layoutMazeRunner, one mapMazeRunner, new map every 1,000
RSM46.09.426.622.5
RATE without cache47.110.017.314.7
RATE with cache46.026.213.510.7

Evidence.

  • Training horizon. RSM minus RATE without its cache is +0.3 [−2.2, +3.0] doors, with no practical difference on CountRecall or MazeRunner; RATE without its cache fails the T-Maze on one seed.
  • Fixed layout. RATE's memory also holds over 4,000 steps (46.0–47.1 doors), so stability there is not specific to RSM's design.
  • MazeRunner. RSM leads RATE without and with its cache by 9.3 [6.4, 12.5] and 13.1 [10.1, 16.6] goals on one map, and by 7.8 [5.3, 10.5] and 11.9 [9.7, 14.0] with a new map every 1,000 steps, on every seed. Restarted at every attempt, RSM and RATE without its cache are close (24.5 against 24.2): carrying adds 2.1 [0.5, 3.7] goals to RSM and costs RATE 6.9 [5.1, 8.8].
  • Layout change. RATE's cache lifts it to 26.2 doors per later layout; without it, at the same state, RATE is close to RSM (10.0 against 9.4).
  • RATE's parts added to RSM. The valve costs 5.7 [1.0, 10.4] doors on Key-to-Door and makes carrying harmful on MazeRunner (−5.8 [−7.9, −3.9] goals). The carried tokens as write queries make no practical difference on three benchmarks and are inconclusive on Key-to-Door (−2.6 [−5.9, +0.9]), though they start learning 0.8M steps earlier (part 5).

Revision (updated 15 Nov). Table R3 is Table 5 (Appendix A.3); Section 5 discusses RATE.

AuthorsOpenReview ↗

Is the design sound, and is the evaluation too small? (Part 3)

Question 4. Is the design sound?

Answer. Each part serves one goal, a constant-size memory that the RL objective alone can teach what to keep, and each part is tested by a controlled comparison.

Evidence, part by part.

  • One causal Transformer over read slots, records and write slots. Each 32-step segment is one attention problem: four read slots holding the summary from the last boundary, the segment's records, then four write slots, under one causal mask in every layer. Every decision sees the summary and the exact records up to its own step, and nothing later, as online RL requires. The write slots come last, so they see the old summary and the whole segment while no decision sees them: writing the next summary cannot leak into the current actions. RSM reuses the Transformer blocks of AMAGO (Grigsby et al., ICLR 2024) under this mask, so it differs from the full-history baseline only in the attention pattern, in positions that restart each segment and in about 2% more parameters (Table 4). The comparison with the full history therefore isolates the memory.
  • Write slots separate from read slots. The write slots hold learned queries, not a copy of the summary as in RATE, and read the old summary and the whole segment, so one learned function decides at every boundary what to keep and what to replace. What learned queries write is worth keeping: over a long run on one MazeRunner map, carrying the summary from attempt to attempt adds about 2 goals per 500 steps over restarting it (+2.1 [0.5, 3.7]), while a summary written by copies of itself, as in RATE, gains nothing (−0.4 [−2.6, +1.7]). A direct comparison points the same way, though not conclusively (+2.7 [−0.7, +6.3]).
  • A single, direct overwrite. The two alternatives we tested both do worse. An additive write keeps old content by default: it learns fastest, but its summary piles up and drifts over long runs (part 5). RATE's retention valve adds a second attention step in which the old memory reads the new write; added to RSM, it costs 5.7 [1.0, 10.4] doors at the training horizon and delays learning (parts 2 and 5).
  • Gradients through every write. Training replays whole tasks and backpropagates through every carried summary. The write has no loss of its own, so it learns only from how useful its content proves to later decisions. Cutting this gradient at the boundaries (RSM detach, Section 4.4) makes the summary both worse and slower to learn: final CountRecall accuracy falls from 98.9% to 91.4%, and during training the detached policy needs 3.6–4.8 million steps (by seed) before it outperforms RSM without memory, against 1.6 million for RSM.
  • Positions that restart every segment. However long the agent runs, attention sees only positions from its training range. The sliding window shows why both this and the summary matter. Its content is bounded too, but its positions keep growing: it peaks at 31.9 doors per 500 steps in steps 501–1,000 and falls to 4.9 by 4,000 steps. Restarted every 500 steps, it returns to about 25, still 21.1 [17.7, 25.3] below carried RSM, because it keeps nothing older than 40 records.

Revision (updated 15 Nov). The tests are in Section 4.4 and Appendices A.3 and F.2.

Question 5. Is the evaluation too small?

Answer. In model size and task variety, yes, and the revised paper will list this as a limitation. In the dimension the paper's claim is about, how long an agent keeps using its memory beyond its training length, we scaled far.

  • Horizon. Every memory runs 4,000 steps on Key-to-Door and MazeRunner, 8 times its training horizon, and RSM now also 32,000 steps on Key-to-Door, 64 times. It still completes 46.7 doors per 500 steps in the last window, against 43.8 in steps 501–1,000 (+2.9 [1.1, 5.1]), at 250 KB of state (a full history: about 197 MB).
  • A fair comparison at this size. Every Transformer memory uses the same three-layer, 256-wide backbone, and the GRU matches its width and depth, so differences between rows come from the memory, not from model capacity.
  • Benchmarks chosen for causal tests. Dark Key-to-Door (Laskin et al., ICLR 2023) and MazeRunner are AMAGO's memory tasks, Dark Key-to-Door is also used by the Memo paper, the T-Maze (Ni et al., NeurIPS 2023) by the Memo paper and RATE, and CountRecall comes from POPGym (Morad et al., ICLR 2023). Their hidden variables (key and door, cue, counts, maze and action mapping) are known, which is what lets us test causally what the summary stores; with pixels that is much harder.

Revision. Section 6 will list scale as a limitation (a small backbone, training tasks of at most 500 steps, low-dimensional observations), with larger, visual, continuous-control and embodied settings as future work. Robots and long-running services will be presented as motivation only.

Reviewer_Z9euOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to improve in-context reinforcement learning using recurrent summary memory. The authors propose a Recurrent Summary Memory (RSM), which uses a transformer policy that takes as input a buffer of the current segment of up to C=32 steps and M=4 summary tokens of the past transitions which are overwritten at every segment boundary to efficiently model experience. The paper compares RSM to full-history attention, sliding window attention, accumulating summary token attention (Memo) and GRU architecture on Dark Key-to-Door, CountRecall, MazeRunner, and T-Maze with a training budget of 8M timesteps. The authors show on unseen tasks that RSM is not consistently worse than baseline methods within the training horizon, and it outperforms baseline methods on a longer evaluation horizon on Dark Key-to-Door and MazeRunner. This is evaluated by letting the frozen policy act for 4000 steps in a single continuing run, with memory carried throughout, either on the same task (repeated attempts in one Key-to-Door layout or repeated laps on one MazeRunner maze) or with the hidden task replaced at fixed intervals (a new Key-to-Door layout every 500 steps, or a new MazeRunner maze every 1000 steps). Further, the authors empirically justify the components of their method through analyses on how the summary tokens affect the decisions made by the policy and through ablations.

Critical strengths
  • Clearly written.
  • RSM shows performance gains compared to GRU on the longer horizon evaluation with repeated attempts at the same task or switching to a new task mid-run. GRU has no positional encoding, so the comparison is not confounded by out-of-distribution positions.
  • Extensive causal tests of summary use within RSM show that decisions depend on summary tokens' content.
  • The authors make an anonymized version of the complete code available.
Critical weaknesses
  • The training horizon results show that RSM does not outperform the best transformer-based baselines, learning slowest on Key-to-Door. Its claimed advantage rests entirely on the long horizon evaluation.
  • There are differences to the training hyperparameters compared to AMAGO v3.4.0's defaults: less updates per environment step (128 per 8000 steps for RSM vs 1000 per 12000 for AMAGO v3.4.0), smaller batch size (16 vs 24), smaller replay buffer capacity (10000 vs 20000), smaller total budget (8M vs 12M), faster exploration annealing (50000 vs 500000), and different warmup (128 gradient steps vs 500 steps).
  • Streaming Memo as a more memory-efficient version of Memo is missing as a baseline.
  • The long horizon evaluation is limited. It either repeats one task or stitches together independently sampled tasks.
  • The absolute time index of the full-history and sliding window transformer baselines keeps increasing and is not reset at boundary points further pushing them to an out-of-distribution regime in the long horizon setting compared to RSM which resets its positions every segment by design.
  • With the per-attempt memory wipes in MazeRunner in Table 9, Memo (whose positions reset with memory) matches RSM-wiped, and the carried summary RSM version adds in performance about 4 goals over the wiped variants. Only the fixed-state memory methods (RSM and GRU) benefit from retaining memory, and in both cases the full-history transformer baseline declines likely because it operates outside its trained positional input range.
  • A more suitable test would be extrapolation to larger mazes. The repository shows the authors ran this evaluation (25x25, results/figure_data/tables_three_benchmarks_table_main.md), and there RSM (0.296) does not outperform Memo (0.321) or the full-history transformer (0.301). It should be reported in the paper.
Questions for authors
  • How do RSM, the full-history transformer, Memo and the GRU perform when trained with AMAGO v3.4.0's default settings, at the training horizon and the long horizon?
  • How does Streaming Memo compare to RSM?
  • What is the performance of all methods in the long horizon setting for Key-to-Door and MazeRunner with memory wiping and with resetting their positional encoding (where applicable) at boundary points?
  • Why is maze size extrapolation (25x25, results/figure_data/tables_three_benchmarks_table_main.md) not included in the paper?
7 comments
AuthorsOpenReview ↗

what the paper claims, and Weakness 1 (Part 1)

We thank the reviewer for a detailed review. We have run all four requested experiments: Streaming Memo (Q2), RSM, Memo, the full history and the GRU wiped with their positions restarted at each boundary (Q3), every maze size (Q4), and training under AMAGO's defaults (Q1, in progress). In brief, the new results support the paper's central claim: RSM's lead beyond the training horizon remains when the baselines' positions are restarted (on a fixed Key-to-Door layout and on changing MazeRunner maps), RSM leads Streaming Memo in every setting, and its performance holds over 64 times the training horizon.

What the paper claims. Two things:

  1. At the training horizon, a small memory of fixed size (four tokens, 250 KB) performs about as well as memories that keep the whole history.
  2. Beyond the training horizon, on the same tasks, this small memory keeps working, while memories that grow do not.

Three of the seven weaknesses judge the paper on claims it does not make:

  • W1 expects RSM to beat the other memories at the training horizon. We claim parity there, not superiority.
  • W4 expects long runs with structure shared across tasks. We test a task that continues and a task that changes.
  • W7 expects generalisation to larger mazes, which lie outside the training distribution for every method.

W5 attributes the baselines' decline to their growing positions. That is the limitation the paper studies, not a flaw of our evaluation: a memory that grows soon runs on more context, at later positions, than it ever saw in training. Even with positions restarted, the gap remains on a fixed Key-to-Door layout and on MazeRunner with changing maps (Table R4).

The other four (W2, W3, W5, W6) bear directly on the claims, and we ran or started new experiments for them. All seven are answered below.

Weakness 1. No advantage at the training horizon. RSM holds 250 KB of state. At 500 steps the full history holds 12 times that and Memo 2.2 times, and both keep growing. Against the other memories whose state does not grow, RSM is clearly ahead already at the training horizon:

  • Sliding window of 40 records (246 KB, the same state as RSM): 29.1 against 24.1 doors on Key-to-Door, +4.9 [1.4, 9.0]: about 20% more, and the whole interval lies above the 1-door margin we fixed in advance for a meaningful difference.
  • GRU (a fixed-size recurrent state): on MazeRunner it reaches 64.3% of the goals against 80.4% for RSM, a difference of 16.1% [12.3, 19.9]. On CountRecall it answers 93.0% of the queries exactly against 98.9% for RSM, a difference of 5.9% [1.7, 9.5].

Against the memories that grow, RSM is roughly level, and no method is best everywhere:

  • MazeRunner, goals reached: RSM 80.4%, full history 73.7%, Memo 86.3%.
  • CountRecall, queries answered exactly: RSM 98.9%, full history 99.8%, Memo 95.6%.

At the training horizon, four tokens thus do about what the whole history does. Beyond it, RSM pulls far ahead: at 4,000 steps on Key-to-Door it opens 46.0 doors per 500 steps, against 28.2 for the full history and 10.6 for Memo (Table R4, part 3 of this reply).

Slower start on Key-to-Door. RSM starts learning later, but not more slowly. Because it replaces its summary at every boundary, it must first learn what to write into it. We checked this at each of the 20 training checkpoints by clearing the summary (details under Reviewer uSVn):

  • Until 2.4M steps, clearing changes the score by at most 1 door: the summary is not yet used.
  • From 2.8M steps, on every seed, the summary drives RSM's gains. RSM then goes from 5 to 25 doors in about 2M steps, as fast as the full history, and at 8M it is level with the other memories (29.1 against 27.4–29.6 doors).

RATE (Cherepanov et al., ICLR 2026), the memory-token Transformer for RL that we now compare against (Table R3, in our reply to Reviewer uSVn), also overwrites its memory at every segment. It starts no earlier than RSM: it reaches 20 doors at 4.4–5.6M steps, against 4.4–5.2M for RSM. The Memo paper reports the same slow start for its own overwriting baseline (its Appendix A.7). A later start is thus common to memories that overwrite, not specific to RSM. It costs training steps, mostly on Key-to-Door, but not final performance, and the revision reports these measurements.

AuthorsOpenReview ↗

Weakness 2, Q1 training settings, Weakness 3, Q2 Streaming Memo (Part 2)

W2/Q1. Settings differ from AMAGO v3.4.0's defaults. The six differences are as listed. They do not favour RSM:

  • Fixed before RSM, used for everything. The recipe and the 8M budget were set while qualifying the four benchmarks with the full-history Transformer, before RSM was implemented, and were then applied unchanged to every memory, benchmark and ablation. Nothing was tuned for RSM. C = 32 and M = 4 were fixed in advance, and M was ablated only afterwards (Table 3).
  • A reduced schedule, chosen for cost. 128 updates per 8,000 steps, instead of 1,000 per 12,000, is what made three seeds of every cell and ablation affordable.
  • The baselines learn every task well under it. At the training horizon Memo reaches 29.6 doors on Key-to-Door and 86.3% of MazeRunner goals, the full history answers 99.8% of CountRecall queries, and every baseline of the paper solves the T-Maze (100%). On every benchmark at least one baseline matches or exceeds RSM, so the recipe does not under-train them.
  • It cannot explain the long-horizon result. The growing memories decline once their context passes the trained length. Restarted within that length, they do not decline at all (Table R4, part 3/4).

Q1. We have started training on Key-to-Door under all six of AMAGO's defaults. A training run under the defaults takes several GPU-days, so some may not reach 12M steps within the discussion period. We will post both horizons for each in this thread, with the steps it reached. The paper's comparison does not depend on these runs. Every memory was trained and evaluated under one recipe, fixed before RSM existed and never tuned for it, so the comparison between memories is fair as it stands.

W3/Q2. Streaming Memo. We added Streaming Memo exactly as Section 4.6 of the Memo paper defines it. It is a rule for which memory Memo keeps while it acts, not a different model, so, as in the Memo paper, we apply it to the same trained Memo policies as in our paper: only the memory kept during evaluation differs. Memo adds a new set of summary tokens every 32-step segment and keeps them all. Streaming Memo keeps only the K most recent sets, plus the current segment, and drops older ones; the kept sets keep the positions they were written at. The Memo paper suggests keeping as many as Memo holds when it performs best. We read this from accumulating Memo's own runs:

  • Key-to-Door: its rate is highest in segment 15 (steps 449–480), near the end of the 500-step training task, so K = 15.
  • MazeRunner: its rate is highest in segment 2 (steps 33–64). An attempt (one pass through the maze to its three goals) is one training episode, and half of them end within 54 steps, so K = 2. Larger budgets do worse there: K = 15 and K = 24 give 0.7 and 0.5 goals, against 1.8.

We also run K = 24, the Memo paper's value on ExtObjNav. Every streaming run matches accumulating Memo exactly until it first drops a set.

Table R1. Doors or goals per 500 steps: the last 500 of 4,000 steps, or the mean over layouts 2–8 where the layout changes.

MemoryKD, one layoutKD, new layout every 500 stepsMazeRunner, one mapMazeRunner, new map every 1,000 steps
RSM46.09.426.622.5
Memo, accumulating10.60.51.81.8
Streaming Memo, K = 15 (KD) / 2 (MazeRunner)12.84.31.81.6
Streaming Memo, K = 247.02.00.50.6

RSM leads Streaming Memo in all four settings, for both K, with every paired 95% interval above zero: by 33.2 [30.5, 35.9] doors on one Key-to-Door layout and 24.8 [21.7, 28.1] goals on one MazeRunner map. After Key-to-Door layout changes, streaming does improve on accumulating Memo (+3.9 [3.3, 4.4] doors with K = 15), but RSM still leads it by 5.1 [3.8, 6.4].

Streaming Memo's rate falls in the window in which it evicts its first summary set. From then on, the sets it keeps sit at position indices that grow past those seen in training, and what the evicted sets held about the task (on Key-to-Door, where the key and door are) is gone. RSM bounds its state in a different way: a single summary is rewritten at every segment, so what was learned earlier is carried forward, at positions that restart every 32 steps.

In the Memo paper's own setting, Memo still generalises beyond its training length when streaming begins: trained on 4,096-step trials, it keeps improving to about 10k steps, and streaming starts at 6k (its Sections 4.3 and 4.6). Here, Memo performs best near the end of its trained length (above), so streaming starts where Memo no longer generalises. Within its trained length the port is sound: at the training horizon it matches RSM on Key-to-Door and leads it on MazeRunner.

Revision. Streaming Memo will become one baseline in our paper.

AuthorsOpenReview ↗

Weakness 5 and Q3 positions and restarts (Part 3)

W5/Q3. The baselines' growing positions. Restarting the positions does not close the gap on a fixed Key-to-Door layout or on MazeRunner with changing maps (Table R4). On one Key-to-Door layout, carried RSM still leads every restarted memory by more than 16 doors per 500 steps. With a new MazeRunner map every 1,000 steps, it leads restarted Memo and the restarted full history by 14 and 18 goals, without any boundary signal.

The control also measures how much of the full history's own decline comes from its positions. On one MazeRunner map, wiping its memory at every attempt while its position index keeps counting leaves it at 0.7 goals per 500 steps. Restarting the index as well raises it to 12.9, and to 17.9 once the wipe is placed correctly (see W6, part 4 of this reply). This is the limitation the paper studies, not a flaw of the evaluation: a memory that grows with time also runs on positions it never saw in training as soon as it runs longer than in training. We ran each baseline as it was trained and as its own paper runs it. Wiping the memory and resetting positions requires telling the agent when every boundary occurs, and gives a different, restarted agent. Table R4 reports that agent for RSM, Memo, the full history and the GRU.

Table R4. Doors (Key-to-Door) or goals (MazeRunner) per 500 steps: the last 500 of 4,000 steps (with new MazeRunner maps, Table 1 averages the fourth map instead), or, for Key-to-Door with new layouts, the mean over layouts 2–8. Restarted: memory wiped and positions reset at each boundary, from the new attempt's first observation. Carried: run as trained, with no boundary signal. Bold: best in each column.

MemoryKD, one layout (restart every 500 steps)KD, new layout every 500 steps (restart at each change)MazeRunner, one map (restart every attempt)MazeRunner, new map every 1,000 steps (restart at each change)
RSM, restarted29.430.824.523.0
Memo, restarted29.631.226.98.7
Full history, restarted27.229.617.94.2
GRU, restarted27.427.915.916.2
RSM, carried46.09.426.622.5
Memo / full history / GRU, carried10.6 / 28.2 / 40.60.5 / 0.8 / 1.61.8 / 0.8 / 19.11.8 / 0.4 / 16.2

A restart brings each memory back to its fresh-start score, and no further. On one Key-to-Door layout the restarted memories make 27–30 doors, close to their first 500 steps, while carried RSM makes 46.0, ahead of restarted Memo, full history and GRU by 16.3 [14.5, 18.4], 18.7 [16.5, 21.0] and 18.5 [13.7, 24.9]. The long runs measure what a memory accumulates beyond its training length, and a restart throws that away.

A restart also rescues a growing memory only when restarts come within its training length. On MazeRunner a training episode is a single attempt, so a growing memory has never seen one attempt follow another in its context. Only when it is restarted after every attempt, with the end of each attempt announced, does Memo stay within that regime and match carried RSM (26.9 against 26.6; −0.3 [−2.7, +2.3]). Restarted only when the map changes, every 1,000 steps, Memo and the full history carry many attempts between restarts and decline again, and carried RSM leads them by 13.8 [10.9, 16.7] and 18.3 [15.6, 21.0] goals. RSM itself needs no restart: 22.5 carried, 23.0 restarted (−0.5 [−1.7, +0.8]).

Only one setting favours an announced restart over carrying: a new Key-to-Door layout every 500 steps, where every restarted memory exceeds carried RSM (9.4 doors). Given the same announcement, RSM shows no consistent difference from the best restarted memory (30.8 against Memo's 31.2; −0.4 [−1.3, +0.5]).

This most likely follows from what the summary stores. In Dark Key-to-Door the agent never sees the key or the door, so beyond the last 32 steps the summary is its only record of where they are. After an unannounced change that record is confidently wrong, and nothing in the observations contradicts it until the agent reaches the old key cell; a summary transplanted from another layout has the same effect (Figure 4(f)). On MazeRunner the agent observes the goals and its distances to the nearest walls, so a new maze shows itself at once, while the action mapping, which it must infer, is kept across a map change; carrying the summary through the change makes no consistent difference (22.5 against 23.0).

The GRU has no positions, so the comparison with it has no positional confound: carried RSM leads it by 5.4 [1.9, 8.5] doors at 4,000 steps, 6.1 [3.5, 8.7] at 32,000, and 7.6 [5.7, 9.5] goals on MazeRunner.

Revision. Table R4 and the cases above will be added to the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

To all reviewers and the Area Chair: summary and revisions

Updated 19 November: final summary. In our revised version, every change is in blue.

We sincerely thank the AC for their time and support throughout the discussion. We also thank the three reviewers for their careful and constructive feedback. We were glad that they found the problem well motivated, the design clean and useful, and the paper clearly written, and that all three highlighted the causal evidence that the policy relies on its summary. We are grateful for their valuable suggestions on the analysis, which led to new experiments and findings that strengthen the paper.

The paper's claim and contributions

  • A bounded memory. RSM is a summary memory for online in-context RL. Its size stays fixed, at about 1% of a full history after 4,000 steps, as does its per-decision cost.
  • Parity at the training horizon. RSM reaches the same performance level as memories that keep everything. Matching them with far less memory is already a substantial gain.
  • Sustained adaptation beyond the training horizon. RSM keeps adapting over eight times that horizon, and over 64 times in a new run (Appendix F.2), while growing memories break down.
  • Decisions rely on the summary. Clearing, transplanting and blocking the summary show that the decisions use what it holds.
  • Findings on bounded memory. It needs little capacity, a write that overwrites rather than adds, positions within the trained range, and a signal when the task changes.

Where the new results are

RebuttalExperimentRevised paper
Table R1Streaming Memo, as in the Memo paperTables 1 and 9, Figure 3
Table R2Larger mazesAppendix G, Table 13
Table R3RATE's design choices on RSMAppendix A.3, Table 5
Table R4Restarted memoriesAppendix F.2, Table 11
Table R5Summary capacityFigure 7, Appendix F.7
Window, positions in rangeTables 2, 9 and 11, Appendix A.2
AMAGO's default settingsAppendix A.5

Main concerns and how we resolved them

Does the long-horizon gain come from RSM's memory? (Z9eu, uSVn) Yes. Streaming Memo falls behind RSM in every long-running setting. With its positions kept in range, the sliding window is level with RSM on one Key-to-Door layout. Carrying no stale summary, it re-adapts better after unannounced layout changes, but RSM leads it on Key-to-Door at the training horizon and on MazeRunner on every seed: there the summary keeps what the window cannot.

Is the stale summary an unaddressed failure? (uSVn, Z9eu) No. After a change unseen in training, RSM lacks a signal of the change, not the ability to re-adapt: told of it, RSM matches the best restarted memory. RATE's cache supplies this signal, and Section 6 presents it as a design choice.

Are the training settings fair? (Z9eu, uSVn) Yes. AMAGO's defaults were set for the full history. Their exploration stays high for the first 6M steps, so of the three memories trained under them, only the full history learns within 12M steps. Under our settings every memory does.

Why does RSM start later on Key-to-Door? (Z9eu, uSVn) It must first learn what to keep, then climbs about as fast as the full history and the GRU and ends at their level.

Concerns that do not bear on the paper's claim

  • RATE. RATE's memory design is the closest to RSM's, but it was built for offline RL, so it is not a comparable baseline. We can compare only design choices, which prove complementary: at the same state RSM leads on long MazeRunner runs, and RATE's cache helps after an unannounced layout change, at 1.8 times the state.
  • Restarted memories. A restart relies on an announcement that in-context RL does not provide, and it only returns a memory to its training-horizon score. Carried RSM leads every restarted baseline on one Key-to-Door layout and with changing MazeRunner maps. Only after Key-to-Door layout changes do announced restarts do better.
  • Larger mazes. They are an out-of-distribution shift: every memory degrades, and the policy without memory degrades least.
  • Scale. It matches what several of the closest works use for controlled tests of memory, with tasks whose hidden variables are known. Section 6 lists it as a limit.

Changes in the revision

  1. The claims (abstract, Sections 4.2 and 6): RSM is competitive at the training horizon and re-adapts better than the growing-memory and recurrent baselines.
  2. Streaming Memo and the window with positions in range join Tables 1, 2 and 9.
  3. Section 6 states what a bounded memory needs and the study's limits.
  4. Section 3 walks through a segment, Section 4.1 explains the MazeRunner protocol, and Section 4.3 defines donor and recipient.
  5. Table 9 (now 10) is unchanged. Appendix F.2 explains how Table 11's corrected restart differs.

In our earlier replies, Tables 8 and 9 and Appendix F.7 are now Tables 9 and 10 and Appendix F.8.

We thank the reviewers again for helping us improve the paper.

Dear reviewers,

The authors have submitted their rebuttal. Please read it carefully, along with the other reviews, and respond whether it adequately addresses your concerns and whether you plan to update your score.

If concerns remain, please state the open points clearly and specifically so the authors can follow up during the discussion period.

Thank you for your engagement!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.