- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper uses Go-Explore to collect successful trajectories on individual training levels, then fits a policy to these trajectories with behavior cloning and fine-tunes it with PPO. The main change to Go-Explore is to sample return states from the current successful trajectory after a solution has been found. Experiments use four Procgen games and three MiniHack tasks, with policies evaluated on unseen level seeds. Under matched interaction budgets, GAGE-RL reaches a reported aggregate solve rate of 58%, compared with 28% for PPO and 31% for RND. The authors also examine larger search budgets, different numbers of training instances, and training a shared policy across games.
Critical strengths
The results on Heist and Corridor are worth reporting. PPO and RND struggle with these tasks, and initializing from demonstrations makes a substantial difference. The main comparison also counts search and return steps, so the demonstrations are not treated as free expert data. Testing on held-out seeds is appropriate for the question being asked. I found Appendix F useful because it tests trajectory optimization separately, even though the result is largely negative after RL fine-tuning. These experiments show that the pipeline can help on the selected games. I would regard this as a useful application of existing methods, but not enough of a contribution for acceptance.
Critical weaknesses
The new solution-path sampling rule is a small change, and Appendix F reports no significant improvement from it after RL fine-tuning. With that result, I am left with an existing search procedure followed by standard BC and PPO, evaluated on another set of games. Collecting the demonstrations is allocated 800 million steps in Procgen and 150 million in MiniHack, across 20,000 available training instances. This dependence on extensive per-instance search is difficult to justify as a broadly useful approach. Appendix J shows collapse below 1,000 instances in its setting, so instance diversity matters and obtaining it requires more searches. Yet search is tested only in discrete-action games, with hand-designed cells and reliable return to saved states. There is no evidence here that random-action expansion remains practical in high-dimensional continuous control. Resetting is needed for data collection rather than policy execution, but that still restricts where the required training data can be obtained.
Questions for authors
-
How many distinct instances actually supply demonstrations under the stated search budgets, and how much of the budget goes to unsuccessful searches? Can the authors report the time and resources for the complete pipeline on comparable hardware, including BC and PPO? Search throughput alone leaves this unclear.
-
Why does Appendix J report collapse below 1,000 instances while Appendix G obtains gains with 500? Is critic warmup responsible for this difference? It would also help to separate the effect of increasing instance diversity from that of increasing the total search budget.
-
Could the authors test a continuous-control task in a robot simulator such as MuJoCo, varying action dimension and the accuracy of state restoration? This would require an additional experiment, but would directly test whether the demonstration-generation procedure scales beyond the conditions used here.
3 comments
Response [2/2]
Q2a: Thank you for pointing out this discrepancy. Both experiments use the same critic warmup period, but this detail was accidentally omitted from the main text. We will update the main text accordingly. Appendix J and G report different results for a small number of demonstrations, likely due to differing KL constraints. The 500 seed experiments in Appendix G are tuned for that setting and use a stronger KL constraint weight of 1.0, while the 1000 seed experiments in Appendix J use the tuned value of 0.5 from the main experiments in Section 4. We will update the text of both Appendices to note these details and update Appendix N to include the hyperparameters for Appendix G.
Q2b: We study the impact of increasing the search budget in Section 4.1.2, and the impact of increasing the number of instances in Appendix J. Appendix J allows up to 2M search steps per instance, so it does not isolate the effect of increasing instances from increasing search steps. Studying the effect of increasing instances with a fixed search budget would be an interesting future study.
Thank you for taking the time to thoroughly consider our work and helping to identify points of confusion in our experiments. If you feel that we have addressed your questions and concerns, please consider raising your score to support sharing our results with the ICLR community.
[1] Gallouédec, Quentin, and Emmanuel Dellandréa. "Cell-free latent go-explore." International Conference on Machine Learning. PMLR, 2023.
[2] Höftmann, Marc, Jan Robine, and Stefan Harmeling. "Time-myopic go-explore: learning a state representation for the go-explore paradigm." arXiv preprint arXiv:2301.05635 (2023).
[3] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. "On the importance of exploration for generalization in reinforcement learning." Advances in Neural Information Processing Systems 36 (2023): 12951-12986.
[4] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.
[5] Zisselman, Ev, et al. "Explore to generalize in zero-shot rl." Advances in Neural Information Processing Systems 36 (2023): 63174-63196.
Thank you for the response. I have read it carefully, but my concerns remain, so I will keep my score. The rebuttal does not include new experiments, and the paper has not been revised. The two questions that needed new evidence were set aside: separating instance diversity from search budget is left for future work, and continuous control is considered out of scope. The setting is also narrow and heavily constrained: all environments are discrete-action 2D games, and the method requires hand-designed cells and exact restoration of saved states. The rebuttal acknowledges these requirements but offers no evidence that they can be relaxed. The numbers the rebuttal does give raise a further concern. The matched-budget comparison is described as training on 20,000 instances, yet demonstrations come from only 31 to 4,495 of them. For MonsterLava, 31 instances is far below the scale at which Appendix J reports RL fine-tuning collapsing, and GAGE-RL indeed stays at about 1% there. On novelty, the rebuttal agrees that trajectory optimization is not central, and Appendix F shows it has no significant effect after RL fine-tuning. The remaining parts, namely per-instance Go-Explore, early stopping, BC and KL-regularized PPO, are existing components. The strongest result does not make up for this either: the 89% vs. 28% comparison is not matched in samples, because GAGE is given a larger search budget, justified by wall-clock time.
Response [1/2]
Thank you for taking the time to read our paper and provide your thoughts. In particular we appreciate your kind words regarding the strength and value of our results. We address your concerns regarding novelty and contributions in our overall response, and will discuss your remaining concerns below:
We would like to highlight that although GAGE generates demonstrations through per-instance search, Section 4.1.1 shows that GAGE is more sample efficient than popular RL baselines, achieving higher scores in fewer samples. Appendix G demonstrates that these results hold with only 500 seeds and 200M environment steps. Section 4.1.2 further shows that GAGE performs better with additional samples, while existing RL baselines plateau early in training. These findings support the conclusion that GAGE outperforms PPO and RND with a fixed interaction budget and that this lead grows with additional data.
Although trajectory optimization is a relatively small change, it significantly improves the performance of GAGE-BC in Heist and Jumper and matches performance in Maze and Coinrun. While the difference is not significant after RL finetuning, GAGE-BC is itself a valuable method that outperforms PPO and RND on average while requiring 68% of the CPU-hours and 16% of the GPU-hours of GAGE-RL (Appendix L). Trajectory optimization is not central to our contributions, but we believe it is a valuable addition to the Go-Explore literature.
Limitations: We acknowledge that GAGE has several limitations in our paper. As you state, GAGE currently relies on hand designed states and a deterministic environment simulator. However, modern AI is often trained in simulated environments, so this limitation may not significantly decrease GAGE’s applicability in practice. The reliance on cell representations is a key limitation of the method, which prior work has attempted to address [1][2]. Our work targets an orthogonal limitation of the original Go-Explore, which is its ability to generalize across many environments. We hope that future work will build on GAGE to design cell representations that are applicable to broad distributions of environments.
Environments and Q3: While we agree that testing GAGE in a wider range of environments would be valuable, we believe that this is beyond the scope of our paper. Procgen and MiniHack are established benchmarks for exploration and generalization [3][4][5] and our experiments target particularly challenging sparse-reward environments within these domains. In Procgen, most prior work studies the “easy” distribution, while we target the “hard” distribution that presents substantially more difficult exploration challenges. These environments also test navigation, long horizon planning, and partial observability.
Prior work has demonstrated that Go-Explore is widely applicable to continuous control, robotics, and even text-based domains. Studying the end-to-end sample efficiency of GAGE in such environments would be a valuable direction for future work, but our results establish meaningful improvements in challenging discrete action settings.
Q1a: GAGE searches trajectories one by one, and did not solve all 20,000 of the available environment instances. In practice, given the budgets and stopping conditions used in Section 4.1.1, GAGE solves between 31 (MonsterLava) and 4495 (Maze) instances, depending on the environment, with solve rates ranging from 100% of attempted seeds in all Procgen games to 32% in Monster Lava. This means that 130M steps were spent on unsuccessful searches in MonsterLava, the hardest domain, whereas all search attempts are successful in Procgen. Without budget restrictions, such as for the datasets collected in Section 4.1.2, Go-Explore solves the vast majority of MiniHack instances as well.
Q1b: We did not conduct our experiments on equivalent hardware, so it is not possible to report an exact fair comparison between the methods. Roughly speaking, data collection (800M steps) consumed 12-24 hours on a 12 CPU node, BC required 3-6 hours on an L40s GPU, and RL fine-tuning required 12-24 hours on a GPU node with 8-12 CPUs. PPO and RND (1B steps) required 3-5 days on a GPU node with 8-12 CPUs, with RND typically taking longer. In our experiments, GAGE incurred significantly lower costs than the baselines (Appendix L), largely due to the extremely fast GPU-free data collection process.