- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper applies an ensemble reinforcement learning method built on Cascade to the setting of unsupervised RL as a plug and play mechanism to improve existing unsupervised RL algorithms. They mainly focus on using it on top of METRA. They also introduce and prove some results related to the method and motivations behind it. Their experiments test the method in environments from OGBench and DeepMind Control Suite. They test mainly for coverage, but also for downstream use of learned skills in zero-shot goal reaching settings and for hierarchical control. They find benefits of using their proposed approach. They also present auxiliary experiments investigating the impact of some choices in the method and on its learned behaviors.
Critical strengths
- I think evaluating the method for coverage, zero-shot capability after pretraining, and hierarchical use of learned skills is a good choice.
- The paper reports improvements from the proposed approach across several environments.
- I appreciate the inclusion of auxiliary results for ablations and sensitivity studies.
Critical weaknesses
I see several weaknesses with the paper that lead to my recommendation to reject.
Limited novelty.
The paper’s main focus is on applying an existing ensemble method (which previous work already tested in other RL problems) to a specific RL setting (unsupervised RL) with small adaptations. I think this is a limited novelty. I also don’t see the paper presenting enough arguments for this setting being special in how much it can benefit from such an approach compared to other RL settings. With respect to the mathematical propositions the paper introduces, I don’t find them to be sufficiently interesting in and of themselves. I don’t find them particularly valuable nor do I think the paper convinces of such value.
Presentation and clarity.
I think the paper’s presentation could be significantly improved.
Experiments.
- I think the results could be organized by the research question they address. The way the results are presented makes the links to the 4 research questions the paper introduces harder to determine.
- Since the contribution is EXPAND, I would like to see more comparisons between methods and their + expand version. Several methods appear without their + expand extensions.
- In terms of organization, figure 4 and Table 1 seem to address RQ1. I think it would be better if their results were discussed jointly. Why separate them?
- While it is good that there are parameter-matched controls for METRA, these are limited. For many results, it is not clear what benefits are from the ensemble approach the paper focuses on vs simply increasing the capacity of the policy.
- I think some of the ablations could be moved to the appendix to better discuss the main results. The experiment sections seem very compressed.
- I did not find reporting on the hyperparameter tuning/selection procedure used for the method and baselines.
- With respect to the comparison in Figure 8. Since all the components keep on being trained as more stages are added, the question it addresses is different from asking how many stages should be used or how varying the number of stages would affect coverage. How would you compare what the experiment addresses versus testing running the whole pre-training with different numbers of total stages?
- Minor: The meaning of History, Replay, and Policy coverage should be explained in the main text in my opinion.
Questions for authors
None