acceptodds
Under review as a conference paper at ICLR 2027

ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learning

Abstract

In partially observable reinforcement learning (RL), a later observation can make stored information obsolete or change what it implies for the next decision. Memory architectures and benchmarks for RL mostly test retention, the ability to keep information unchanged until it is needed. We formalize two further requirements. Rewriting sets the decision-relevant content to a value independent of the old one, and experience fusion transforms the old content by a rule that a later observation specifies. For tasks built from such updates, we count the memory states that a solution needs, and several baselines reach their lowest success rates on compositions that need more states. We introduce ALER (Adaptive Learnable Experience Rewriting), an agent that pairs an LSTM with a slot memory. An independently addressed Gumbel-Softmax write that concentrates its weight on one slot overwrites that slot, and a learned gate fuses the retrieved content with the recurrent state before the policy and value heads. We also introduce Rune-Mazes, three environments in which rune observations invert, cancel, reset, or repeat updates of a hidden cue under vector and pixel observations. Against seven baselines, ALER reaches a success rate of at least in all sixteen Endless T-Maze configurations and at least on all five Rune T-Maze compositions, and it has the highest mean success rate on four-branch Rune Multi-Corridor with an Invert rune. On pixel-based Rune MiniGrid Memory, it has a higher mean success rate than PPO-LSTM in eight of ten configurations. Project page: https://anon-aler.github.io/aler/

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Memory, POMDP, Long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lry8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ALER, an LSTM agent with learnable slot memory, and Rune-Mazes to study memory retention, rewriting, and rule-based updates. It formalizes memory requirements and evaluates ALER through baseline comparisons, length extrapolation, and mechanism analyses.

Critical strengths
  1. ALER achieves promising reported results on Endless T-Maze, including better corridor-length extrapolation than PPO-LSTM in the tested setting.
  2. The explicit state-count analysis clarifies the memory requirements of the rune compositions and provides a useful framework for examining baseline performance.
Critical weaknesses
  1. The paper shows that memory stores historical information and that the full ALER model outperforms PPO-LSTM in several settings. However, it does not establish whether these gains come from retaining more useful information than the LSTM. For example, on RepeatPrev + SkipNext + Invert, initial-cue decoding is 98% from the LSTM versus 85% from memory, and required-turn decoding is 70% versus 57%. This does not invalidate ALER’s improvements, but the mechanism behind them remains unclear. More targeted controls would help explain what the memory contributes beyond the LSTM.

  2. Some reported results are difficult to reconcile, and these do not seem to be isolated cases: similar curve/table gaps appear across several configurations in Figures 6–7, and multiple entries in Table 9 raise questions about the reported SEM.

For example, on 3×Invert + ResetRules, Figure 7 shows ALER mostly around 0.5–0.65, well below S5’s later performance around 0.9–0.95, while Table 3 reports 0.99±0.00 for both. I understand that the table selects each seed’s best checkpoint, but per-seed curves, checkpoint times, and any smoothing details would help explain this large gap. Likewise, Table 9 reports 0.84±0.22 for ALER on S9 and 0.84±0.23 for PPO-LSTM on S11Random, both described as SEM over three seeds. At a mean of 0.84, the largest possible SEM is 0.16, and rounding cannot explain these differences. Could these be standard deviations instead?

Reconciling these patterns and providing the underlying per-seed results would substantially strengthen my confidence in the paper’s findings.

  1. The paper does not discuss or compare against modern gated linear-attention models such as Gated DeltaNet and Kimi Delta Attention, which also support controlled forgetting and selective memory updates. These capabilities are directly relevant to the paper’s emphasis on retention and rewriting. Although several memory-based baselines are included, a comparison with a representative model from this family under a comparable PPO training setup would help establish the practical benefit of ALER’s explicitly addressed slot memory.
Questions for authors
  1. Could you explain the gaps between Figures 6–7 and the corresponding tables? For example, on 3×Invert + ResetRules, Figure 7 shows ALER well below S5, while Table 3 reports 0.99±0.00 for both. Could you provide the per-seed curves, selected checkpoint times, and any smoothing details? Also, are Table 9’s error bars SEM or STD? Results such as 0.84±0.22 over three seeds seem incompatible with the stated SEM.

  2. S5 is highly competitive on the core tasks. Why was it excluded from the length-extrapolation and pixel experiments? Could you add S5 under comparable settings to establish whether ALER’s advantages hold against this strong baseline?

  3. Beyond the existing PPO-LSTM performance comparisons, could you directly test whether LSTM plus memory retains information better than an independently trained LSTM-only model? For example, could you compare held-out cue and target decoding across increasing delays under matched settings? Comparing the complete agents, rather than only the memory and LSTM states within ALER, would help clarify whether the gains reflect better retention.

5 comments

We sincerely thank the reviewer for the detailed and constructive review and for the time and effort devoted to our paper. Thank you for highlighting ALER's corridor-length extrapolation and the use of explicit state counts to interpret the rune tasks. We have added controls that distinguish cue retention from action selection and extended the baseline comparisons. Please see the blue revisions in the attached PDF, especially Section 7 (pp. 8-10) and Appendices E.2 and H (pp. 22-26 and 30-31).

Weakness 1. Retention and the use of external memory.

The corridor-length experiment tests robustness to increasing the interval between observing a cue and using it to choose a turn, and ALER substantially outperforms both S5 and PPO-LSTM in this comparison. In Figure 4 (Section 7, p. 8), ALER maintains a success rate of 1.00 through corridor length 30, whereas PPO-LSTM reaches 0.00 at that length and S5 is already near zero at length 20. Since the cue is shown only at the start of each corridor, completing these episodes requires preserving cue-dependent information until the junction and using it to select the correct turn. ALER's success therefore supports retention and use of that information beyond the training horizon.

The probes then examine what information remains accessible inside the agents. In the separate matched experiment without coordinate input, the current cue can be decoded from both ALER's and PPO-LSTM's recurrent states at reached junctions with balanced accuracy 1.00 ± 0.00 at length 50 (Appendix E.2, Figures 8 and 11, pp. 23 and 25). Cue information can therefore remain recoverable even when episode success decreases. The Rune T-Maze probes also decode the initial cue and the required turn separately, distinguishing retention of the original cue from representation of the target after the rune rules have been applied (Figure 13, p. 26).

Finally, the memory-replacement experiment directly tests whether ALER uses its external bank to choose actions. We kept the recipient's LSTM state unchanged and replaced only the bank immediately before the turn in Rune T-Maze Invert. When the donor had the opposite cue and the same rune presence, success fell from 1.00 ± 0.00 to 0.52 ± 0.22 for episodes with Invert, and from 1.00 ± 0.00 to 0.10 ± 0.07 for episodes without it. Replacing the bank with one having the same cue and rune presence preserved performance. Changing the bank alone thus changes the agent's decisions, showing that the external memory contributes to behavior even when the cue remains decodable from the recurrent state (Appendix H.2, Table 20, p. 30).

Weakness 2. Learning curves and the Table 9 uncertainty measure.

Table 9 reports STD over three training seeds. Thank you for pointing out the label, which we corrected to STD in the caption (p. 20) without changing the numerical results. Table 5 also uses STD, and Section 6 (p. 7) now explicitly distinguishes these two MiniGrid tables from our vector-task results, which use SEM across training seeds.

The curves and tables aggregate checkpoints differently. Figures 6 and 7 average validation scores across the plotted runs at each common training step, while Tables 4 and 3 first select a checkpoint separately within each seed and then average the selected scores. The selected steps can differ between seeds, so the table mean can exceed every point on the mean curve. Appendix D.3 (p. 20) now explains this distinction.

Weakness 3. Comparison with Gated DeltaNet and Kimi Delta Attention.

We implemented online PPO agents using the original FLA blocks for Gated DeltaNet and Kimi Delta Attention. The adaptation supplies an observation encoder, actor and critic heads, and recurrent-state resets at episode boundaries, while preserving the memory-update rules.

Gated DeltaNet and Kimi Delta Attention maintain a matrix-valued recurrent state updated by a delta rule. Gated DeltaNet controls forgetting per head, while Kimi Delta Attention controls it across key dimensions. ALER maintains explicit memory slots with separate read and write addresses, and its fusion gate combines the retrieved vector with the LSTM state. Appendix H.1 (p. 30) now explains these architectural differences and the PPO adaptation in blue.

We transferred ALER's PPO settings to both baselines and ran three seeds on three Rune T-Maze tasks and two four-branch Rune Multi-Corridor tasks, using 6M steps per run. Checkpoint selection followed Section 6 (p. 7): evaluation every 100,000 steps on 100 fixed dev episodes, maximum success rate, and mean return to break ties. The dev episodes and selection rule matched ALER's for the corresponding task, and dev episodes were excluded from training rollouts and gradient updates. After selection, each checkpoint was fixed and evaluated on 100 episodes per seed. Table 19 (Appendix H.1, p. 30) reports these results and states the budgets alongside the 6M-step main results.

TaskOriginal ALERGated DeltaNetKimi Delta Attention
Rune T-Maze Invert1.00 ± 0.001.00 ± 0.001.00 ± 0.00
Rune T-Maze 3 Invert + ResetRules0.99 ± 0.000.99 ± 0.000.99 ± 0.00
Rune T-Maze RepeatPrev + SkipNext + Invert0.99 ± 0.010.99 ± 0.010.99 ± 0.00
Multi-Corridor K=4, no rune1.00 ± 0.000.32 ± 0.080.43 ± 0.09
Multi-Corridor K=4, Invert0.83 ± 0.170.27 ± 0.120.19 ± 0.10

Both added agents achieve near-ceiling success on the tested Rune T-Maze compositions. Their success is lower on the four-branch Rune Multi-Corridor tasks, where ALER reaches 1.00 without runes and 0.83 with Invert. Appendix H.1, Table 19 (p. 30), gives the full comparison, including PPO-LSTM and S5.

Question 1. Can the curve/table gap, selected checkpoints, smoothing, and STD/SEM be clarified?

Table 9 reports STD over three training seeds. Thank you for pointing out the label, which we corrected to STD in the caption (p. 20) without changing the numerical results. Table 5 also uses STD, and Section 6 (p. 7) now explicitly distinguishes these two MiniGrid tables from our vector-task results, which use SEM across training seeds.

The curves and tables aggregate checkpoints differently. Figures 6 and 7 average validation scores across the plotted runs at each common training step, while Tables 4 and 3 first select a checkpoint separately within each seed and then average the selected scores. Section 6 (p. 7) specifies selection by maximum validation success rate, with mean return breaking ties. Since the selected steps can differ between seeds, the table mean can exceed every point on the mean curve. Appendix D.3 (p. 20) now explains this distinction, and the revised PDF presents the aggregate curves and the selection rule.

Question 2. Why was S5 absent from the length-extrapolation and pixel experiments?

We have added S5 to both experiments in Section 7. Figure 4 (p. 8) compares robustness to increasing the interval between observing and using a cue: ALER maintains full success through length 30, where both S5 and PPO-LSTM are near zero. Table 5 (p. 10) adds S5 beside ALER and PPO-LSTM on all ten Rune MiniGrid Memory configurations.

In the length-extrapolation experiment across five seeds, S5's mean success falls from 1.00 ± 0.00 at length 10, the length of training corridor, to 0.01 ± 0.01 at length 20 and stays below 0.01 at every tested length from 30 to 100. ALER maintains a mean success rate of 1.00 through length 30 and still reaches 0.82 ± 0.18 at length 50, where both baselines are near zero. These results show substantially greater robustness to longer cue-to-decision intervals for ALER than for both S5 and PPO-LSTM.

For the visual experiments, S5 used 30M steps and three seeds per configuration, with the RGB encoder and training settings given in Appendix F (p. 27). Figure 4 reports SEM, and Table 5 reports STD. ALER remains competitive with S5 on several visual configurations, reaching 0.77 versus 0.79 on fixed-length NoOp + Invert and 0.69 versus 0.70 on random-length RepeatPrev + SkipNext + Invert. On random-length NoOp + Invert, ALER reaches 0.92 versus S5's 0.72, and it exceeds PPO-LSTM in 8 of the 10 configurations overall. On Endless T-Maze, ALER is substantially more robust than S5 to longer cue-to-decision intervals (Figure 4).

Question 3. Can cue and target decoding be compared between ALER and independently trained LSTM agents across increasing delays?

We added the requested comparison between ALER and independently trained PPO-LSTM agents in Appendix E.2 (pp. 22-26). The three PPO-LSTM agents use a recurrent width of 256 and ALER's PPO settings, with no coordinate input. Both architectures are probed on fixed-length, 20-corridor episodes using the same 128-episode generation protocol. For each checkpoint and length, a standardized ridge classifier is fitted on 96 episodes and evaluated on the remaining 32, with whole episodes kept separate between the two splits.

We measure both agents at the first visit to each reached junction, after processing the observation and before choosing the action. At lengths 20, 30, 40 and 50, the current cue is decoded from both models' recurrent states with balanced accuracy 1.00 ± 0.00 when the probe is fitted at that length. Figure 11 (p. 25) pairs the PPO-LSTM decoding scores with success on the same episodes.

Series / lengthCue from h_tCue from c_tEpisode success
New, length 201.00 ± 0.001.00 ± 0.001.00 ± 0.00
New, length 501.00 ± 0.001.00 ± 0.000.66 ± 0.17
New, length 1001.00 ± 0.001.00 ± 0.000.49 ± 0.29

The cue can remain available in the recurrent state even when the agent fails to complete the episode. At length 100, PPO-LSTM's cell-state decoding remains 1.00 ± 0.00 while success is 0.49 ± 0.29 (Appendix E.2, Figure 11, p. 25). The performance drop therefore coexists with retained cue information, highlighting the role of using that information correctly over a whole episode.

We also checked whether a decoder learned at one delay transfers to another. In Appendix E.2, a PPO-LSTM probe fitted at length 10 achieves 0.33 ± 0.17 from h_t and 0.50 ± 0.00 from c_t when applied at the first junction at length 50. Fitting the probe at length 50 recovers 1.00 ± 0.00 from both states. The cue is therefore recoverable at the longer delay, even though the decoder fitted at the shorter delay transfers poorly.

Task / targetALER h_tALER memoryPPO-LSTM h_tPPO-LSTM c_t
Invert / initial cue1.00 ± 0.001.00 ± 0.001.00 ± 0.001.00 ± 0.00
Invert / required turn1.00 ± 0.000.71 ± 0.100.97 ± 0.000.97 ± 0.00
RepeatPrev + SkipNext + Invert / initial cue0.98 ± 0.010.85 ± 0.061.00 ± 0.001.00 ± 0.00
RepeatPrev + SkipNext + Invert / required turn0.70 ± 0.040.57 ± 0.100.80 ± 0.030.77 ± 0.02

The Rune T-Maze probes extend the comparison to rule-dependent decisions by decoding the initial cue and required turn separately at junctions. We use all episodes and report SEM across three training seeds, as shown in the blue Figure 13 (Appendix E.2, p. 26). These probes identify accessible information, while the memory-replacement experiment tests its behavioral role by changing only the external bank and measuring the resulting decisions (Appendix H.2, Table 20, p. 30).

Reviewer_zjAWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes ALER, a PPO agent that combines an LSTM with a slot memory, which is read and written with Gumbel-Softmax addressing, and a learned gate that mixes the read vector with the LSTM state. The authors also formalize how later observations can retain, rewrite, or transform a stored cue, and they introduce the Rune-Mazes environments. They report that ALER reaches at least 0.820.82 in all Endless T-Maze configurations and at least 0.990.99 on all Rune T-Maze compositions, against seven baselines.

Critical strengths

The rune rules and the vector environments are specified exactly (Appendix C.1 and C.2), and Appendix A.3 explains how the state counts in Table 1 are computed. Rune T-Maze also clearly separates FFM and SHM from S5, GTrXL, and ALER on the compositions with Q⋆≥4Q^\star \geq 4 (Table 3), so it seems to be a useful small diagnostic for memory agents, independent of ALER.

It is also good that the paper reports results that do not favor ALER, for example the 0.000.00 for ALER with Invert at K=8K=8 and K=16K=16 in Table 4 and the probe results for the read vector in Appendix E.2, which makes the claims checkable.

Critical weaknesses

My main problem is that the results do not seem to support that ALER is a better memory agent, and several numbers do not fit together. S5 reaches at least 0.800.80 in all Endless T-Maze configurations and has the same values as ALER in Table 3. On Multi-Corridor with Invert, ALER is close to PPO-LSTM at K=4K=4 (0.83±0.170.83 \pm 0.17 vs. 0.77±0.000.77 \pm 0.00) and gets 0.000.00 at K=8K=8 and K=16K=16, where PPO-LSTM gets 0.350.35 and 0.100.10 (Table 4). The best checkpoint is also selected on the same 100 episodes that are reported (lines 322-323 and 335-337). Some ±\pm values in Table 2 cannot be SEMs over five seeds, e.g., 0.69±0.310.69 \pm 0.31, and the ALER curves in Figures 6 and 7 stay far below the tables for some settings, e.g., around 0.60.6 on 3 Invert + ResetRules vs. 0.99±0.000.99 \pm 0.00 in Table 3. Maybe I misread the small figures, but for me it is not clear which runs produced the tables.

The second problem is that the analysis in Section 8 and Appendix E seems to contradict the division of work that motivates ALER. The read vector, which is the only path from the memory to the policy, decodes the current cue in Endless T-Maze with only 0.580.58 to 0.600.60, while the LSTM state decodes it with 1.001.00 (Figure 8), and the slot identity carries almost no information about position or cue (lines 1123-1125). Also, Section 4 gives retention to the slots (lines 197-198), but the conclusion gives it to the LSTM (line 473). Setting the gate to zero at evaluation lowers success (Table 10), but this is not the same as training without the memory, so for me it is still not clear that the memory is the reason for ALER's results.

Questions for authors
  1. Which runs produce Figures 6 and 7 and Tables 2 to 4? In particular, how many seeds and which selection rule were used for the Endless T-Maze baselines in Table 2, where some values, e.g., 0.73±0.270.73 \pm 0.27, cannot be SEMs over five seeds?
  2. Can you report all methods on fresh episodes that were not used for checkpoint selection, and add S5 to Figure 4 and Table 5?
  3. How does ALER perform on the vector tasks when it is trained without the memory but with the same LSTM size and PPO settings, including its entropy coefficient of 0.100.10 on Endless T-Maze and Multi-Corridor (Table 11)?
5 comments

We sincerely thank the reviewer for the detailed and constructive review and for the time and effort devoted to our paper. Thank you for highlighting the precise specification of the rune rules and state counts, and for recognizing Rune T-Maze as a diagnostic that differentiates memory agents independently of ALER. We have clarified the scope of the comparisons, the use of SEM and STD and the role of the external bank. Please see the blue revisions in the attached PDF, particularly Sections 4, 6 and 7 (pp. 4 and 7-10) and Appendices E.2 and H (pp. 22-26 and 30-31).

Weakness 1. Baseline comparisons, uncertainty reporting, and learning curves.

Comparison with S5. ALER achieves higher success than S5 on four-branch Rune Multi-Corridor, where the cue has more than two possible targets. ALER reaches 1.00 ± 0.00 without runes and 0.83 ± 0.17 with Invert, compared with 0.55 ± 0.14 and 0.55 ± 0.16 for S5 (Section 7, Table 4, p. 9). This comparison tests retention and transformation of a larger cue space than the binary-cue tasks on which both models approach full success.

The added S5 experiments test robustness to longer intervals between observing and using a cue, as well as performance with visual observations. In Figure 4 (p. 8), ALER maintains success of 1.00 through corridor length 30, while S5's success is already near zero at length 20 and PPO-LSTM reaches 0.00 at length 30, showing a substantial extrapolation advantage for ALER. Table 5 (p. 10) now includes S5 on all ten Rune MiniGrid Memory configurations. ALER remains competitive with S5 on several visual configurations, reaching 0.77 versus 0.79 on fixed-length NoOp + Invert and 0.69 versus 0.70 on random-length RepeatPrev + SkipNext + Invert. On random-length NoOp + Invert, ALER reaches 0.92 versus S5's 0.72, and it exceeds PPO-LSTM in 8 of the 10 configurations overall. On Endless T-Maze, ALER is substantially more robust than S5 to longer cue-to-decision intervals (Figure 4).

SEM and STD. Our vector-task results use SEM across training seeds, while Tables 5 and 9 use STD over three seeds for the two MiniGrid benchmarks. We corrected the STD label in Table 9 (p. 20). In Table 2, the five-seed SEM applies to our ALER, S5 and DNC runs, while imported baseline results retain the uncertainty measure reported in their source. Section 6 (p. 7) and the captions of Tables 2 and 8 (pp. 8 and 20) now distinguish these measures.

Learning curves and checkpoint selection. The curves and tables also aggregate checkpoints differently. Figure 6 shows Rune Multi-Corridor, corresponding to Table 4, and Figure 7 shows Rune T-Maze, corresponding to Table 3. The curves average validation scores across the plotted runs at each common training step, while the tables first select a checkpoint separately within each seed and then average the selected scores. Since the selected steps can differ between seeds, the table mean can exceed the highest point on the mean curve. We made this distinction explicit in Appendix D.3 (p. 20).

Weakness 2. The roles of recurrent state and external memory.

ALER learns its recurrent state and external memory jointly, allowing both components to retain task-relevant information. The learned fusion gate determines how their representations contribute to the policy. Sections 4 and 9 (pp. 4 and 10) now describe this shared role consistently. The bank probes and memory-replacement experiment examine what is stored externally and whether the policy uses it.

The bank itself contains decodable cue information. At corridor length 50, the full bank after writing supports current-cue decoding with balanced accuracy of 0.94-0.99 across the sampled positions (Appendix E.2, Figure 9, p. 24). Slot identity describes the address used to access memory, while the probe of the full bank examines the stored vectors. The read vector is a projected, weighted combination of those vectors. These measurements concern different stages of memory use, which explains why their decoding scores need not coincide.

The memory-replacement experiment directly tests the bank's contribution to decisions. With the recipient's LSTM state unchanged, an opposite-cue bank lowers success to 0.52 ± 0.22 when both episodes contain Invert and to 0.10 ± 0.07 when neither does (Appendix H.2, Table 20, p. 30). Changing the bank alone changes behavior, showing that the policy uses its contents. We also trained the requested LSTM-only control with ALER's recurrent width and PPO settings and report its results in our answer to Question 3 and in Appendix H.3, Table 21 (p. 31).

Question 1. Which runs produced Figures 6 and 7 and Tables 2 to 4, and how were checkpoints and seeds handled?

Figures 6 and 7 (p. 21) show Rune Multi-Corridor and Rune T-Maze, corresponding to Tables 4 and 3. Tables 3 and 4 report our runs for all eight methods over five training seeds. The curves average validation scores across the plotted runs at each common training step, while the tables first select a checkpoint separately within each seed and then average the selected scores. The selected steps can differ between seeds. Appendix D.3 (p. 20) now explains this aggregation.

Table 2 combines our ALER, S5 and DNC runs with PPO-LSTM, PPO-MLP, GTrXL, SHM and FFM results from the cited prior work. The five-seed SEM applies to our runs, while imported baseline results retain the uncertainty measure reported in their source. The captions of Tables 2 and 8 (pp. 8 and 20) now make this distinction explicit.

For the runs we trained, Section 6 (p. 7) specifies evaluation every 100,000 steps on 100 fixed validation episodes. Within each training seed, the checkpoint with the highest success rate is selected, with mean return breaking ties. We average the selected scores across seeds only after that selection.

Question 2. Can all methods be evaluated on fresh episodes, and can S5 be added to Figure 4 and Table 5?

The main tables report the validation success of the checkpoint selected for each training seed. Under Section 6 (p. 7), checkpoints are evaluated every 100,000 steps on 100 fixed validation episodes and selected by the highest success rate, with mean return breaking ties. All methods trained in the paper use the same validation episodes and selection rule for the corresponding task, and these episodes are excluded from training rollouts and from policy and value-function gradient updates. The comparison therefore uses common evaluation tasks and a common selection criterion, while keeping model fitting separate from model selection. We then average the selected validation scores across training seeds within each method's stated training budget.

We have also added S5 to the length-extrapolation experiment in Figure 4 (Section 7, p. 8). This also tests robustness to increasing the interval between observing the cue and using it to choose a turn. Across five seeds, S5's mean success falls from 1.00 ± 0.00 at length 10, the length of training corridor, to 0.01 ± 0.01 at length 20 and stays below 0.01 at every tested length from 30 to 100. ALER maintains a mean success rate of 1.00 through length 30 and still reaches 0.82 ± 0.18 at length 50, where both baselines are near zero. These results show substantially greater robustness to longer cue-to-decision intervals for ALER than for both S5 and PPO-LSTM.

For visual observations, we trained S5 on all ten Rune MiniGrid Memory configurations using three seeds and 30M steps per configuration and seed, with the same checkpoint-selection rule for S5, ALER and PPO-LSTM. Appendix F (p. 27) gives the visual encoder and PPO settings, and Table 5 places the three models side by side. ALER remains competitive with S5 on several visual configurations, reaching 0.77 versus 0.79 on fixed-length NoOp + Invert and 0.69 versus 0.70 on random-length RepeatPrev + SkipNext + Invert. On random-length NoOp + Invert, ALER reaches 0.92 versus S5's 0.72, and it exceeds PPO-LSTM in 8 of the 10 configurations overall. On Endless T-Maze, ALER is substantially more robust than S5 to longer cue-to-decision intervals (Figure 4).

Table 5 reports mean ± STD over three seeds, as does Table 9 for retention-only MiniGrid Memory. Section 6 (p. 7) distinguishes this use of STD from SEM across training seeds for our vector-task results.

Question 3. How does ALER perform when trained without memory but with the same LSTM size and PPO settings?

We trained the requested PPO-LSTM with no external bank, a shared 256-unit recurrent core and ALER's family-specific PPO settings. Each task uses three seeds and a 6M-step budget, with entropy coefficient 0.10 on Endless T-Maze and Rune Multi-Corridor.

Following Section 6 (p. 7), checkpoints were evaluated every 100,000 steps on 100 fixed dev episodes and selected by maximum success rate, with mean return breaking ties. The dev episodes and selection rule matched ALER's for the corresponding task, and dev episodes were excluded from training rollouts and gradient updates. This keeps the conditions for model selection consistent. After selection, each rune-task checkpoint was fixed and evaluated on 100 episodes per seed. Appendix H.3, Table 21 (p. 31), gives the full results and protocol.

TaskOriginal ALEROriginal PPO-LSTMPPO-LSTM, ALER settings
Rune T-Maze Invert1.00 ± 0.000.84 ± 0.041.00 ± 0.00
Rune T-Maze RepeatPrev + SkipNext + Invert0.99 ± 0.010.83 ± 0.020.96 ± 0.01
Multi-Corridor K=4, no rune1.00 ± 0.001.00 ± 0.001.00 ± 0.00
Multi-Corridor K=4, Invert0.83 ± 0.170.77 ± 0.000.87 ± 0.13
Multi-Corridor K=8, no rune0.00 ± 0.000.00 ± 0.000.12 ± 0.00
Multi-Corridor K=8, Invert0.00 ± 0.000.35 ± 0.100.04 ± 0.03

The matched PPO-LSTM reaches 1.00 ± 0.00 on Rune T-Maze Invert and four-branch Rune Multi-Corridor without runes. Table 21 places these results alongside the original five-seed validation results. For Endless T-Maze, Table 22 (p. 31) compares final PPO-LSTM checkpoints with the selected ALER analysis checkpoints on 20-corridor episodes without coordinate input. This supplies the requested training control, while the memory-replacement experiment addresses the complementary question of whether the trained ALER policy actually uses its bank.

The extrapolation results are reproduced below (Table 22, Appendix H.3). Each entry is success rate, mean ± SEM over three training seeds, on 20-corridor episodes without coordinate input.

Corridor lengthALERPPO-LSTM, ALER settings
200.99 ± 0.010.79 ± 0.15
300.65 ± 0.170.65 ± 0.19
400.64 ± 0.180.67 ± 0.21
500.62 ± 0.190.40 ± 0.30

ALER retains higher mean success at the longest tested delay: 0.62 ± 0.19 at corridor length 50, compared with 0.40 ± 0.30 for PPO-LSTM with ALER's settings. From length 30 to 50, ALER's mean success changes only from 0.65 to 0.62, whereas PPO-LSTM's falls from 0.65 to 0.40. This comparison shows more stable long-delay extrapolation for ALER under matched recurrent width and PPO settings.

Reviewer_egFXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that memory in partially observable RL should be tested on rewriting and transforming stored information, not only on retention, and uses minimal automata to count the memory states each rune composition needs. ALER adds a slot memory with Gumbel-Softmax read and write to PPO-LSTM and mixes the read vector with the LSTM state through a sigmoid gate. It is evaluated on Endless T-Maze and three new Rune-Maze environments against seven baselines, with probes and gate experiments.

Critical strengths
  1. Testing memory beyond retention is a useful direction, and every rune task comes with an exact memory-state count.
  2. The protocol is described in detail, and the authors try to check what the model actually stores.
Critical weaknesses
  1. ALER is not clearly better than S5. On Endless T-Maze the two trade places: S5 gets 0.80 ± 0.12 on the two constant-length n=1 settings where ALER gets 1.00, ALER gets 0.82 ± 0.17 on l=10, n=3 where S5 gets 1.00, and both get 1.00 on the other 13. The “at least 0.82” in the abstract is only 0.02 above S5’s lowest Endless T-Maze value, and lines 366 to 368 themselves list ALER and S5 as the only architectures at or above 0.80 in every Endless T-Maze configuration. On Rune T-Maze their numbers are identical. ALER is ahead on Multi-Corridor with K=4 (1.00 vs 0.55 ± 0.14 without runes, 0.83 ± 0.17 vs 0.55 ± 0.16 with Invert), and at K ≥ 8 neither exceeds 0.12, against a random-guess rate of 0.125 at K=8 and 0.0625 at K=16. The abstract highlights K=4 with Invert, where ALER’s lead over PPO-LSTM is 0.83 ± 0.17 vs 0.77 ± 0.00, but not K=8 with Invert, where ALER gets 0.00 and PPO-LSTM 0.35 ± 0.10. S5 is also missing from Figure 4 and from the MiniGrid experiments, although MiniGrid uses the Rune T-Maze compositions, where S5 is the strongest baseline.
  2. The main tables report the success rate of the selected checkpoint on the same 100 validation episodes used to select it (Section 6), about 60 evaluations per run for ALER on the vector tasks (6M steps) and 300 on MiniGrid (30M steps). This can bias the numbers upward by an unknown amount, which matters where margins are small. On fixed-length MiniGrid corridors, ALER’s leads on the compositions with Q* ≥ 4 are 0.04 to 0.11 (0.57 to 0.60 vs 0.49 to 0.53), with three seeds and STD instead of SEM (lines 336 to 337 say all tables use SEM), and neither method exceeds 0.60 on these three settings. Seed spread also looks large: S5 gets 0.80 ± 0.12 (SEM over five seeds) on both constant-length classic T-Mazes, which requires at least one seed near or below 0.5, while S5 gets 1.00 ± 0.00 with 3 to 10 corridors. Results on held-out episodes (as in Table 10) and significance tests are needed.
Questions for authors
  1. What happens if the memory is replaced by the memory of another episode at test time?
  2. The agents succeed in almost all episodes (Table 10). Why does required-turn decoding from z_t go from 0.64 (all episodes) to 0.89 (successful episodes) in Figure 11?
  3. Line 320 lists only ALER, S5 and DNC as trained on Endless T-Maze. Were the other Endless T-Maze baselines rerun, and how were DNC and GTrXL tuned?
2 comments

We sincerely thank the reviewer for the detailed and constructive review and for the time and effort devoted to our paper. Thank you for highlighting the value of testing memory beyond retention and the effort to make our evaluation and memory analysis transparent. We have added the S5 comparisons and clarified the evaluation protocol. Please see the blue revisions in the attached PDF, particularly Sections 6-7 (pp. 7-10) and Appendix H (pp. 30-31).

Weakness 1. Comparison with S5 on extrapolation and visual tasks.

ALER achieves higher success than S5 on four-branch Rune Multi-Corridor, where the cue has more than two possible targets. ALER reaches 1.00 ± 0.00 without runes and 0.83 ± 0.17 with Invert, compared with 0.55 ± 0.14 and 0.55 ± 0.16 for S5 (Section 7, Table 4, p. 9). This comparison tests retention and transformation of a larger cue space than the binary-cue tasks on which both models approach full success.

We added S5 to the length-extrapolation experiment in Figure 4 (Section 7, p. 8). This also tests robustness to increasing the interval between observing the cue and using it to choose a turn. Across five seeds, S5's mean success falls from 1.00 ± 0.00 at length 10, the length of training corridor, to 0.01 ± 0.01 at length 20 and stays below 0.01 at every tested length from 30 to 100. ALER maintains a mean success rate of 1.00 through length 30 and still reaches 0.82 ± 0.18 at length 50, where both baselines are near zero. These results show substantially greater robustness to longer cue-to-decision intervals for ALER than for both S5 and PPO-LSTM.

Corridor lengthS5 success rateALER success ratePPO-LSTM success rate
101.00 ± 0.001.00 ± 0.001.00 ± 0.00
200.01 ± 0.011.00 ± 0.000.47 ± 0.29
300.00 ± 0.001.00 ± 0.000.00 ± 0.00
400.00 ± 0.000.83 ± 0.170.00 ± 0.00
500.00 ± 0.000.82 ± 0.180.00 ± 0.00
600.00 ± 0.000.59 ± 0.210.00 ± 0.00
700.00 ± 0.000.49 ± 0.290.00 ± 0.00
800.00 ± 0.000.16 ± 0.160.00 ± 0.00
900.00 ± 0.000.16 ± 0.160.00 ± 0.00
1000.00 ± 0.000.00 ± 0.000.00 ± 0.00

We also added S5 alongside ALER and PPO-LSTM in the existing Rune MiniGrid Memory table (Table 5, p. 10), using three seeds and 30M steps per configuration. The table covers all five rune compositions at both length settings and reports STD for each model. ALER remains competitive with S5 and it exceeds PPO-LSTM in 8 of the 10 configurations overall. On Endless T-Maze, ALER is substantially more robust than S5 to longer cue-to-decision intervals (Figure 4).

CompositionLengthPPO-LSTMALERS5
InvertFixed0.80 ± 0.030.69 ± 0.090.99 ± 0.01
InvertRandom0.93 ± 0.070.94 ± 0.061.00 ± 0.00
NoOp + InvertFixed0.94 ± 0.060.77 ± 0.030.79 ± 0.14
NoOp + InvertRandom0.67 ± 0.070.92 ± 0.080.72 ± 0.06
SkipNext + InvertFixed0.51 ± 0.030.59 ± 0.010.72 ± 0.06
SkipNext + InvertRandom0.62 ± 0.070.63 ± 0.050.80 ± 0.05
3 Invert + ResetRulesFixed0.53 ± 0.040.57 ± 0.010.77 ± 0.12
3 Invert + ResetRulesRandom0.53 ± 0.010.57 ± 0.010.78 ± 0.02
RepeatPrev + SkipNext + InvertFixed0.49 ± 0.030.60 ± 0.000.64 ± 0.02
RepeatPrev + SkipNext + InvertRandom0.56 ± 0.000.69 ± 0.060.70 ± 0.03

Weakness 2. Checkpoint selection and uncertainty reporting.

The main tables report the validation success of the checkpoint selected for each training seed. Under Section 6 (p. 7), checkpoints are evaluated every 100,000 steps on 100 fixed validation episodes and selected by the highest success rate, with mean return breaking ties. All methods trained in the paper use the same validation episodes and selection rule for the corresponding task, and these episodes are excluded from training rollouts and from policy and value-function gradient updates. The comparison therefore uses common evaluation tasks and a common selection criterion, while keeping model fitting separate from model selection.

Our vector-task results report SEM across training seeds, while Rune MiniGrid-Memory and MiniGrid-Memory report STD over three seeds in Tables 5 and 9 (pp. 10 and 20). We corrected the STD label in Table 9 and made the use of SEM and STD explicit in the blue clarification in Section 6 (p. 7). For the imported Endless T-Maze baselines, Tables 2 and 8 (pp. 8 and 20) preserve the uncertainty reported in the source.

Question 1. What happens if the memory is replaced by the memory of another episode at test time?

Replacing the external bank substantially changes ALER's decisions even when its LSTM state is held fixed. We tested this in Rune T-Maze Invert by replacing only the bank at the junction, immediately before the action, with a bank from another episode of the same checkpoint. With an opposite donor cue and matched rune presence, success dropped from 1.00 ± 0.00 to 0.52 ± 0.22 when the recipient saw Invert, and from 1.00 ± 0.00 to 0.10 ± 0.07 when it did not. The drop directly shows that the trained policy uses the contents of the external bank.

The comparison uses 1,000 episode pairs per condition and training seed, with 500 recipients that saw Invert and 500 that did not. Within each group, both the initial cues and the required turns are balanced equally between left and right (250 each per training seed). The blue Table 20 in Appendix H.2 (p. 30) reports mean ± SEM over three seeds and uses 0.50 as the chance baseline.

Donor conditionRecipient with InvertRecipient without rune
Same cue, same rune presence1.00 ± 0.001.00 ± 0.00
Opposite cue, same rune presence0.52 ± 0.220.10 ± 0.07
Same cue, different rune presence1.00 ± 0.000.93 ± 0.07
Opposite cue, different rune presence0.61 ± 0.260.24 ± 0.12

When donor and recipient differ in both cue and rune presence, they require the same final turn, yet replacing the bank still changes success. This is consistent with the policy combining the donor bank's content with the recipient's existing recurrent state, so matching the target turn alone does not preserve the original behavior (Appendix H.2, p. 30).

Question 2. Why does required-turn decoding from z_t rise from 0.64 on all episodes to 0.89 on successful episodes when Table 10 reports almost perfect success?

The two decoding scores come from separately fitted probes on the full probe sample and its successful-episode subset. The first gives 0.64 ± 0.01 and the second 0.89 ± 0.04, with training and test splits separated by whole episodes. Table 10 (p. 22) reports policy success in the gate experiments, while the original Figure 11, now Figure 12 (p. 25), uses a separately collected set of 128 episodes per checkpoint for representation analysis. The high success rate in Table 10 therefore does not specify the proportion of successful episodes in that probe sample. We made this distinction explicit in the blue caption of Figure 12 and describe the probe protocol in Appendix E.2 (pp. 22-26).

Question 3. Were the other Endless T-Maze baselines rerun, and how were DNC and GTrXL tuned?

For Endless T-Maze, we ran ALER, S5 and DNC, while the PPO-LSTM, PPO-MLP, GTrXL, SHM and FFM results were taken from the cited prior work. The revised captions of Tables 2 and 8 (pp. 8 and 20) identify the results from our runs and specify that the five-seed SEM applies to those runs. For the imported results, we preserve the source's uncertainty measure.

For the rune tasks, hyperparameter selection used 2M-step runs with three seeds per candidate. We selected one configuration per model and environment family by mean validation success across those seeds, then trained the selected configuration separately with the full 6M-step budget for the main comparison. GTrXL tuning covered architecture first and PPO settings second, and the table below lists those ranges together with the PPO parameters varied for DNC. Section 6 (p. 7) now states the selection criterion and separates the tuning budget from the final training budget.

Model / parameterTested values
DNC learning rate0.0001, 0.0003, 0.0005
DNC GAE lambda0.95, 0.98
DNC entropy coefficient0.01, 0.10
DNC PPO clip0.10, 0.20
GTrXL (memory length, blocks)(64, 2), (100, 2), (100, 4), (128, 4)
GTrXL (initial learning rate, gamma)(0.000275, 0.995), (0.0001, 0.995), (0.0005, 0.995), (0.000275, 0.99)
GTrXL fixed PPO settingsGAE 0.95, clip 0.10, entropy 0.001 to 0.000001, final learning rate 0.00001

The selected GTrXL settings and the DNC architecture are listed in Appendix F, Tables 15 and 17 (pp. 28-29). GTrXL uses memory length 100 in both rune environment families, with four blocks on Rune T-Maze and two on Rune Multi-Corridor.

Comments on the paper

1 comment

General response

We thank the reviewers for their time and constructive feedback, and for recognizing the value of the rune tasks and their explicit memory-state requirements. During the rebuttal, we added the following experiments and clarifications. All changes are highlighted in blue in the attached revised PDF.

S5 new experiments and additional baselines. We added S5 to corridor-length extrapolation and all ten Rune MiniGrid Memory configurations (Section 7, Figure 4 and Table 5), and evaluated PPO agents with Gated DeltaNet and Kimi Delta Attention on Rune T-Maze and Rune Multi-Corridor (Appendix H.1, Table 19).

Memory contribution. We replaced ALER's external bank while keeping its LSTM state fixed in Rune T-Maze Invert, and trained PPO-LSTM with ALER's recurrent width and PPO settings (Appendices H.2-H.3, Tables 20-22).

Decoding across delays. We added cue and target decoding for independently trained PPO-LSTM-256 agents, using the same environment settings and sampling points as the ALER comparison (Appendix E.2, Figures 11 and 13).

Protocol and model clarifications. We clarified checkpoint selection, hyperparameter selection, the source of baseline results, SEM versus STD, and curve/table aggregation (Section 6, Appendix D.3 and Tables 2, 8 and 9). Sections 4 and 9 now consistently describe how the recurrent state and external bank jointly contribute to the policy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.