- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the instability issue of online RL when mamba is used as the history encoder. It proposes Episodic Forgetting (EF) and Memory Rebalancer (MR) to suppress the information of previous episodes and to mitigate the forgetting problem when the episode is long. The experiments on Brax show the proposed method to have superior or comparable performance with baselines.
Critical strengths
The paper is easy to follow.
Critical weaknesses
- The methodological novelty is limited. Handling the hidden state reset at the boundary is a basic requirement for online RL. EF itself is not a sufficiently compelling contribution.
- MR is essentially a simple regularization trick. Its value is hard to evaluate without numerous experiments.
- The empirical improvements are insufficient. Carefully tuned baselines such as LSTM, transformer, RetNet, should be included. More benchmarks should also be evaluated.
Questions for authors
What is the difference between META and RetNet [1]?
[1] Sun, Yutao, Li Dong, Shao-Han Huang, Shu-Ming Ma, Yuqing Xia, Ji-Long Xue, Jianyong Wang and Furu Wei. “Retentive Network: A Successor to Transformer for Large Language Models.” ArXiv abs/2307.08621 (2023): n. pag.