- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposed intrinsic reward policy optimization (IRPO), a meta reinforcement learning algorithm to encourage exploration. In IRPO, they first train ALLO intrinsic rewards, each rewarding agents to proceed different directions. They maintain multiple sub-policies optimized for different ALLO intrinsic reward to ensure the exploration, in the inner loop. In the outer-loop, the initial policy is updated to maximize the environmental reward. Theoretically, they have provided the evidence that within this framework, sub-policies, updated for intrinsic rewards will converge to the stationary points. In experiment, they validated the performance of IRPO in several goal-reaching environments.
Critical strengths
- While meta-RL is typically used for adaptation across different tasks, this paper applies it to exploration in sparse-reward environments, which I found refreshing. The separation of intrinsic-reward-based inner-loop and extrinsic-reward-based outer-loop provides a clear and intuitive framework.
- The theoretical guarantee justifies the usage of non environmental rewards for the sub-policy updates
- The diversity of the baselines, from vanilla on-policy methods to other intrinsic bonuses, strengthened the experimental results.
Critical weaknesses
The main weakness is the assumption that initial states can be sampled uniformly from the state space when pretraining ALLO intrinsic rewards. While this may be possible in the relatively small simulated environments used in the paper, it would be difficult in real-world environments or even in challenging simulated environments with sparse reward such as Montezuma's Revenge. Since IRPO relies on pretrained and fixed ALLO intrinsic rewards, violating this assumption could substantially degrade its performance. This may limit the applicability of the proposed method. Related to this, the paper also lacks the investigation on the how the coverage of the data for ALLO training affects the resulting RL training.
Questions for authors
- Can authors provide some results on the relationship between the coverage of the data for ALLO training and the RL performance?
- Is it possible to train ALLO intrinsic reward online manner to overcome the limitation posed in the weakness?
10 comments
Thank you for your rebuttal and additional experiments. The new experiments, which pretrain ALLO using random walks from fixed initial states while largely maintaining performance, alleviate my original concern that the algorithm requires the ability to reset the environment to arbitrary states.
However, my broader concern remains. In large, hard-exploration environments (often sparse reward), random walks from the initial state covers only a small portion of the state space. The performance degradation observed with fewer ALLO pretraining samples suggests that insufficient training data can affect performance, and limited state-space coverage may pose an even greater challenge, particularly in regions not visited during pretraining. (It would be helpful to know how much of the reachable state space is covered by random walks in environments such as Maze and PointMaze.)
This is why I raised online ALLO training as a question that could affect my assessment. I understand that evaluating this possibility within the rebuttal period was difficult. Nevertheless, I believe that the dependence on the coverage of the ALLO pretraining data remains a limitation of the current method. Therefore, it is difficult to raise my score at this stage.
New results on online training
Thank you for your thoughtful response. We are glad we were able to address your concerns regarding reset requirements.
We now better understand your concern regarding how much coverage of the state space is required for ALLO to produce useful intrinsics. We agree that our current method may struggle to handle environments where "sufficient" coverage of the state space (via random walk for ALLO) is not possible within a reasonable amount of samples.
We were actually able to generate results where ALLO is jointly trained (online) with IRPO policies (in FourRooms PointMaze, and FetchReach). We added these results to Section 4.3 of the revision.
- We see that joint IRPO-ALLO achieves similar converged performance to IRPO-ALLO (200k pretrained ALLO, which generally provides good coverage). While joint IRPO-ALLO can require more samples than IRPO-ALLO, it still outperforms baselines.
- We summarize peak eval return below. Learning curves can be found in Figure 9 of the revision. | Environment | IRPO-ALLO | Joint IRPO-ALLO | |---|---:|---:| | FourRooms | 0.8773 | 0.8529 | | PointMaze-v1 | 0.7801 | 0.7799 | | FetchReach | 0.9679 | 0.9378 |
- We have provided the implementation detail of the joint training of IRPO in Appendix C.2.
- And thank you for the idea of online joint training.
Author Reponse
We thank the reviewer for finding our work refreshing and seeing value in our theoretical guarantee and diversity of baselines. We also appreciate the concerns and suggested improvements.
1. [Edited November 16] The main weakness is the assumption that initial states can be sampled uniformly from the state space when pretraining ALLO intrinsic rewards.
2. [Edited November 16] Can the authors provide results on the relationship between the coverage of the data for ALLO training and the RL performance?
3. Is it possible to train ALLO intrinsic reward online manner to overcome the limitation posed in the weakness?