- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes ALER, an LSTM agent with learnable slot memory, and Rune-Mazes to study memory retention, rewriting, and rule-based updates. It formalizes memory requirements and evaluates ALER through baseline comparisons, length extrapolation, and mechanism analyses.
Critical strengths
- ALER achieves promising reported results on Endless T-Maze, including better corridor-length extrapolation than PPO-LSTM in the tested setting.
- The explicit state-count analysis clarifies the memory requirements of the rune compositions and provides a useful framework for examining baseline performance.
Critical weaknesses
-
The paper shows that memory stores historical information and that the full ALER model outperforms PPO-LSTM in several settings. However, it does not establish whether these gains come from retaining more useful information than the LSTM. For example, on RepeatPrev + SkipNext + Invert, initial-cue decoding is 98% from the LSTM versus 85% from memory, and required-turn decoding is 70% versus 57%. This does not invalidate ALER’s improvements, but the mechanism behind them remains unclear. More targeted controls would help explain what the memory contributes beyond the LSTM.
-
Some reported results are difficult to reconcile, and these do not seem to be isolated cases: similar curve/table gaps appear across several configurations in Figures 6–7, and multiple entries in Table 9 raise questions about the reported SEM.
For example, on 3×Invert + ResetRules, Figure 7 shows ALER mostly around 0.5–0.65, well below S5’s later performance around 0.9–0.95, while Table 3 reports 0.99±0.00 for both. I understand that the table selects each seed’s best checkpoint, but per-seed curves, checkpoint times, and any smoothing details would help explain this large gap. Likewise, Table 9 reports 0.84±0.22 for ALER on S9 and 0.84±0.23 for PPO-LSTM on S11Random, both described as SEM over three seeds. At a mean of 0.84, the largest possible SEM is 0.16, and rounding cannot explain these differences. Could these be standard deviations instead?
Reconciling these patterns and providing the underlying per-seed results would substantially strengthen my confidence in the paper’s findings.
- The paper does not discuss or compare against modern gated linear-attention models such as Gated DeltaNet and Kimi Delta Attention, which also support controlled forgetting and selective memory updates. These capabilities are directly relevant to the paper’s emphasis on retention and rewriting. Although several memory-based baselines are included, a comparison with a representative model from this family under a comparable PPO training setup would help establish the practical benefit of ALER’s explicitly addressed slot memory.
Questions for authors
-
Could you explain the gaps between Figures 6–7 and the corresponding tables? For example, on 3×Invert + ResetRules, Figure 7 shows ALER well below S5, while Table 3 reports 0.99±0.00 for both. Could you provide the per-seed curves, selected checkpoint times, and any smoothing details? Also, are Table 9’s error bars SEM or STD? Results such as 0.84±0.22 over three seeds seem incompatible with the stated SEM.
-
S5 is highly competitive on the core tasks. Why was it excluded from the length-extrapolation and pixel experiments? Could you add S5 under comparable settings to establish whether ALER’s advantages hold against this strong baseline?
-
Beyond the existing PPO-LSTM performance comparisons, could you directly test whether LSTM plus memory retains information better than an independently trained LSTM-only model? For example, could you compare held-out cue and target decoding across increasing delays under matched settings? Comparing the complete agents, rather than only the memory and LSTM states within ALER, would help clarify whether the gains reflect better retention.
5 comments
Weakness 3. Comparison with Gated DeltaNet and Kimi Delta Attention.
We implemented online PPO agents using the original FLA blocks for Gated DeltaNet and Kimi Delta Attention. The adaptation supplies an observation encoder, actor and critic heads, and recurrent-state resets at episode boundaries, while preserving the memory-update rules.
Gated DeltaNet and Kimi Delta Attention maintain a matrix-valued recurrent state updated by a delta rule. Gated DeltaNet controls forgetting per head, while Kimi Delta Attention controls it across key dimensions. ALER maintains explicit memory slots with separate read and write addresses, and its fusion gate combines the retrieved vector with the LSTM state. Appendix H.1 (p. 30) now explains these architectural differences and the PPO adaptation in blue.
We transferred ALER's PPO settings to both baselines and ran three seeds on three Rune T-Maze tasks and two four-branch Rune Multi-Corridor tasks, using 6M steps per run. Checkpoint selection followed Section 6 (p. 7): evaluation every 100,000 steps on 100 fixed dev episodes, maximum success rate, and mean return to break ties. The dev episodes and selection rule matched ALER's for the corresponding task, and dev episodes were excluded from training rollouts and gradient updates. After selection, each checkpoint was fixed and evaluated on 100 episodes per seed. Table 19 (Appendix H.1, p. 30) reports these results and states the budgets alongside the 6M-step main results.
| Task | Original ALER | Gated DeltaNet | Kimi Delta Attention |
|---|---|---|---|
| Rune T-Maze Invert | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |
| Rune T-Maze 3 Invert + ResetRules | 0.99 ± 0.00 | 0.99 ± 0.00 | 0.99 ± 0.00 |
| Rune T-Maze RepeatPrev + SkipNext + Invert | 0.99 ± 0.01 | 0.99 ± 0.01 | 0.99 ± 0.00 |
| Multi-Corridor K=4, no rune | 1.00 ± 0.00 | 0.32 ± 0.08 | 0.43 ± 0.09 |
| Multi-Corridor K=4, Invert | 0.83 ± 0.17 | 0.27 ± 0.12 | 0.19 ± 0.10 |
Both added agents achieve near-ceiling success on the tested Rune T-Maze compositions. Their success is lower on the four-branch Rune Multi-Corridor tasks, where ALER reaches 1.00 without runes and 0.83 with Invert. Appendix H.1, Table 19 (p. 30), gives the full comparison, including PPO-LSTM and S5.
Question 1. Can the curve/table gap, selected checkpoints, smoothing, and STD/SEM be clarified?
Table 9 reports STD over three training seeds. Thank you for pointing out the label, which we corrected to STD in the caption (p. 20) without changing the numerical results. Table 5 also uses STD, and Section 6 (p. 7) now explicitly distinguishes these two MiniGrid tables from our vector-task results, which use SEM across training seeds.
The curves and tables aggregate checkpoints differently. Figures 6 and 7 average validation scores across the plotted runs at each common training step, while Tables 4 and 3 first select a checkpoint separately within each seed and then average the selected scores. Section 6 (p. 7) specifies selection by maximum validation success rate, with mean return breaking ties. Since the selected steps can differ between seeds, the table mean can exceed every point on the mean curve. Appendix D.3 (p. 20) now explains this distinction, and the revised PDF presents the aggregate curves and the selection rule.
Question 2. Why was S5 absent from the length-extrapolation and pixel experiments?
We have added S5 to both experiments in Section 7. Figure 4 (p. 8) compares robustness to increasing the interval between observing and using a cue: ALER maintains full success through length 30, where both S5 and PPO-LSTM are near zero. Table 5 (p. 10) adds S5 beside ALER and PPO-LSTM on all ten Rune MiniGrid Memory configurations.
In the length-extrapolation experiment across five seeds, S5's mean success falls from 1.00 ± 0.00 at length 10, the length of training corridor, to 0.01 ± 0.01 at length 20 and stays below 0.01 at every tested length from 30 to 100. ALER maintains a mean success rate of 1.00 through length 30 and still reaches 0.82 ± 0.18 at length 50, where both baselines are near zero. These results show substantially greater robustness to longer cue-to-decision intervals for ALER than for both S5 and PPO-LSTM.
For the visual experiments, S5 used 30M steps and three seeds per configuration, with the RGB encoder and training settings given in Appendix F (p. 27). Figure 4 reports SEM, and Table 5 reports STD. ALER remains competitive with S5 on several visual configurations, reaching 0.77 versus 0.79 on fixed-length NoOp + Invert and 0.69 versus 0.70 on random-length RepeatPrev + SkipNext + Invert. On random-length NoOp + Invert, ALER reaches 0.92 versus S5's 0.72, and it exceeds PPO-LSTM in 8 of the 10 configurations overall. On Endless T-Maze, ALER is substantially more robust than S5 to longer cue-to-decision intervals (Figure 4).
Question 3. Can cue and target decoding be compared between ALER and independently trained LSTM agents across increasing delays?
We added the requested comparison between ALER and independently trained PPO-LSTM agents in Appendix E.2 (pp. 22-26). The three PPO-LSTM agents use a recurrent width of 256 and ALER's PPO settings, with no coordinate input. Both architectures are probed on fixed-length, 20-corridor episodes using the same 128-episode generation protocol. For each checkpoint and length, a standardized ridge classifier is fitted on 96 episodes and evaluated on the remaining 32, with whole episodes kept separate between the two splits.
We measure both agents at the first visit to each reached junction, after processing the observation and before choosing the action. At lengths 20, 30, 40 and 50, the current cue is decoded from both models' recurrent states with balanced accuracy 1.00 ± 0.00 when the probe is fitted at that length. Figure 11 (p. 25) pairs the PPO-LSTM decoding scores with success on the same episodes.
| Series / length | Cue from h_t | Cue from c_t | Episode success |
|---|---|---|---|
| New, length 20 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |
| New, length 50 | 1.00 ± 0.00 | 1.00 ± 0.00 | 0.66 ± 0.17 |
| New, length 100 | 1.00 ± 0.00 | 1.00 ± 0.00 | 0.49 ± 0.29 |
The cue can remain available in the recurrent state even when the agent fails to complete the episode. At length 100, PPO-LSTM's cell-state decoding remains 1.00 ± 0.00 while success is 0.49 ± 0.29 (Appendix E.2, Figure 11, p. 25). The performance drop therefore coexists with retained cue information, highlighting the role of using that information correctly over a whole episode.
We also checked whether a decoder learned at one delay transfers to another. In Appendix E.2, a PPO-LSTM probe fitted at length 10 achieves 0.33 ± 0.17 from h_t and 0.50 ± 0.00 from c_t when applied at the first junction at length 50. Fitting the probe at length 50 recovers 1.00 ± 0.00 from both states. The cue is therefore recoverable at the longer delay, even though the decoder fitted at the shorter delay transfers poorly.
| Task / target | ALER h_t | ALER memory | PPO-LSTM h_t | PPO-LSTM c_t |
|---|---|---|---|---|
| Invert / initial cue | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |
| Invert / required turn | 1.00 ± 0.00 | 0.71 ± 0.10 | 0.97 ± 0.00 | 0.97 ± 0.00 |
| RepeatPrev + SkipNext + Invert / initial cue | 0.98 ± 0.01 | 0.85 ± 0.06 | 1.00 ± 0.00 | 1.00 ± 0.00 |
| RepeatPrev + SkipNext + Invert / required turn | 0.70 ± 0.04 | 0.57 ± 0.10 | 0.80 ± 0.03 | 0.77 ± 0.02 |
The Rune T-Maze probes extend the comparison to rule-dependent decisions by decoding the initial cue and required turn separately at junctions. We use all episodes and report SEM across three training seeds, as shown in the blue Figure 13 (Appendix E.2, p. 26). These probes identify accessible information, while the memory-replacement experiment tests its behavioral role by changing only the external bank and measuring the resulting decisions (Appendix H.2, Table 20, p. 30).
We sincerely thank the reviewer for the detailed and constructive review and for the time and effort devoted to our paper. Thank you for highlighting ALER's corridor-length extrapolation and the use of explicit state counts to interpret the rune tasks. We have added controls that distinguish cue retention from action selection and extended the baseline comparisons. Please see the blue revisions in the attached PDF, especially Section 7 (pp. 8-10) and Appendices E.2 and H (pp. 22-26 and 30-31).
Weakness 1. Retention and the use of external memory.
The corridor-length experiment tests robustness to increasing the interval between observing a cue and using it to choose a turn, and ALER substantially outperforms both S5 and PPO-LSTM in this comparison. In Figure 4 (Section 7, p. 8), ALER maintains a success rate of 1.00 through corridor length 30, whereas PPO-LSTM reaches 0.00 at that length and S5 is already near zero at length 20. Since the cue is shown only at the start of each corridor, completing these episodes requires preserving cue-dependent information until the junction and using it to select the correct turn. ALER's success therefore supports retention and use of that information beyond the training horizon.
The probes then examine what information remains accessible inside the agents. In the separate matched experiment without coordinate input, the current cue can be decoded from both ALER's and PPO-LSTM's recurrent states at reached junctions with balanced accuracy 1.00 ± 0.00 at length 50 (Appendix E.2, Figures 8 and 11, pp. 23 and 25). Cue information can therefore remain recoverable even when episode success decreases. The Rune T-Maze probes also decode the initial cue and the required turn separately, distinguishing retention of the original cue from representation of the target after the rune rules have been applied (Figure 13, p. 26).
Finally, the memory-replacement experiment directly tests whether ALER uses its external bank to choose actions. We kept the recipient's LSTM state unchanged and replaced only the bank immediately before the turn in Rune T-Maze Invert. When the donor had the opposite cue and the same rune presence, success fell from 1.00 ± 0.00 to 0.52 ± 0.22 for episodes with Invert, and from 1.00 ± 0.00 to 0.10 ± 0.07 for episodes without it. Replacing the bank with one having the same cue and rune presence preserved performance. Changing the bank alone thus changes the agent's decisions, showing that the external memory contributes to behavior even when the cue remains decodable from the recurrent state (Appendix H.2, Table 20, p. 30).
Weakness 2. Learning curves and the Table 9 uncertainty measure.
Table 9 reports STD over three training seeds. Thank you for pointing out the label, which we corrected to STD in the caption (p. 20) without changing the numerical results. Table 5 also uses STD, and Section 6 (p. 7) now explicitly distinguishes these two MiniGrid tables from our vector-task results, which use SEM across training seeds.
The curves and tables aggregate checkpoints differently. Figures 6 and 7 average validation scores across the plotted runs at each common training step, while Tables 4 and 3 first select a checkpoint separately within each seed and then average the selected scores. The selected steps can differ between seeds, so the table mean can exceed every point on the mean curve. Appendix D.3 (p. 20) now explains this distinction.