- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper asks whether recurrent policies trained on short delays can still work over much longer ones. The authors show that monostable memory eventually forgets, while multistable memory can retain information indefinitely. This distinction matters because many popular parallelizable RNNs have monostable memory. The authors conducted experiments on T-maze and MiniGrid tasks whose findinds support the central claim of the paper.
Critical strengths
- The question of whether a policy trained on short delays still works at much longer ones is clear and practical. The criterion the paper gives for checking this is also simple, since it only depends on whether the memory dynamics are monostable or multistable.
- Testing memory all the way up to 10⁶ steps with 30 seeds is convincing. I especially liked Figure 3, which shows that the effective horizon of monostable models varies so much across seeds that hyperparameter search can’t control it.
- The observation that popular parallelizable RNNs are monostable by construction is timely and useful for people choosing architectures for memory-heavy RL tasks.
Critical weaknesses
- The delayed-cue POMDP setting seems narrow, and the authors introduce it without positioning it against prior work or discussing its relevance to modern deep RL benchmarks. The benchmarks use, including T-maze, MiniGrid-Memory and MiniGrid-GoToDoor, are all simple versions of the same cue-delay-decision structure.
- That multiple attractors can store discrete memories is well known in dynamical systems and neuroscience (e.g., Hopfield, 1982; Sussillo and Barak, 2013), so the main claim of this paper appears limited in novelty.
- Only a fraction of the trained GRU and BRC models end up multistable, and the paper does not propose a way to make training reliably reach multistability.
Questions for authors
- Could the authors discuss how the delayed-cue POMDP setting relates to existing memory benchmarks in deep RL? How common is this cue–delay–decision structure in those tasks?
- Do the authors expect their conclusions to hold on more complex memory tasks, such as ones with multiple cues, continuous cues, or memory that needs to be updated over time? Any experiments beyond the T-maze style tasks would help show the broader relevance.
- I'm curious how multistable models like BMRU do compared with monostable ones on standard POMDP benchmarks where memory needs to be graded or updated often. Is there a tradeoff there?
- Since only some of the trained GRU and BRC models end up multistable, do the authors have ideas for a training procedure that would get standard cells like GRU or LSTM there more reliably?
- Could the authors comment on how the framework would extend to tasks where the cue is continuous rather than discrete?
- How robust are the multistable models in practice to larger perturbations, like strong observation noise or distractors during the delay? It would be nice to see how performance changes as noise increases, compared with the monostable models.