- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a new plug-an-play exploration strategy for RL algorithms named MemoRL. The proposed approach stores and learns valuable trajectories (high relative returns + relatively short). It uses BC as intrinsic reward to reinforce the RL policy over these trajectories. Theorem 1 claims the intrinsic reward keeps the optimal action unchanged. Theorem 2 claims the gated policy is at least as good as the base policy. The approach is validate on a diverse set of algorithms and domains.
Critical strengths
-
The proposed method is extensively and thoroughly evaluated on various RL algorithms and domains. Results are strong on hard exploration domains.
-
Learning from failures (not only from successful trajectories) is novel to the best of my knowledge and the ablation study shows that doing so is beneficial.
Critical weaknesses
-
Missing baselines and related work. 1. Go-Explore, 2. Agent57 (cited in the appendix as an intrinsic-motivation example), 3. Ensemble-UCB gating, 4. #Exploration and E3B, 5. Episodic control. These should all be compared against or at least discussed.
-
Tables 3 and 4 report no source, budget, seeds or variance. Some entries, such as the missing CR-SAC HalfCheetah result, were taken from the original papers? It is unclear whether their settings match MemoRL’s.
-
The abstract claims "it achieves up to a 15× improvement on Montezuma’s Revenge." but that is somewhat misleading because it is over basic PPO. Including RND already brings this down to x3.7.
-
Seeds and confidence intervals are missing for MuJoCo and Figs. 10 and 11.
-
PPO is reported as “440.3 ± 0” over 5 seeds.
-
Table 1 averages 5 games, while the text says 12.
-
No experiment shows that tracks importance or novelty. The difference from Yan et al. (2023; 2024) is not stated.
-
Theorem 1 assumes the bonus is below the smallest action gap, which is near zero in Atari. Theorem 2 assumes is greedy with respect to , which behavior cloning does not ensure.
Questions for authors
-
How is “failure” defined in tasks without terminal death states (e.g., Swimmer, Reacher)? How do the memory networks handle continuous actions?
-
How does your adaptive coordination differ from Agent57’s meta-controller? How does the Prospective Module differ from Yan et al. (2023; 2024)?