acceptodds
Under review as a conference paper at ICLR 2027

GAGE: Training General Agents with Go-Explore Demonstrations

Abstract

Go-Explore is a powerful family of methods for solving hard-exploration problems. These approaches learn by first returning to and then exploring from promising states discovered during training. However, despite dramatically outperforming other exploration techniques in video game, robotics, and text-based reasoning tasks, prior work has not used Go-Explore to train policies that generalize across problem instances. We introduce General Agents via Go-Explore (GAGE), a new method for training general policies in highly variable, sparse-reward environments. GAGE searches for successful trajectories on individual task instances with Go-Explore, distills these demonstrated behaviors into a policy with supervised learning, and then fine-tunes the policy with reinforcement learning. Our experiments on Procgen and MiniHack, two procedurally generated reinforcement learning benchmarks, demonstrate that GAGE generalizes to unseen problem instances while standard reinforcement learning methods struggle. We show that GAGE can match or exceed the sample efficiency of popular RL methods, that its performance reliably improves with additional samples, and that GAGE is robust to increasing reward sparsity. Finally, we show that GAGE can train a single multi-task policy that performs well across several environments. Our work suggests that combining Go-Explore with supervised learning is a promising and underexplored paradigm for training general agents in hard-exploration domains.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Exploration, Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_K3JwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses Go-Explore to collect successful trajectories on individual training levels, then fits a policy to these trajectories with behavior cloning and fine-tunes it with PPO. The main change to Go-Explore is to sample return states from the current successful trajectory after a solution has been found. Experiments use four Procgen games and three MiniHack tasks, with policies evaluated on unseen level seeds. Under matched interaction budgets, GAGE-RL reaches a reported aggregate solve rate of 58%, compared with 28% for PPO and 31% for RND. The authors also examine larger search budgets, different numbers of training instances, and training a shared policy across games.

Critical strengths

The results on Heist and Corridor are worth reporting. PPO and RND struggle with these tasks, and initializing from demonstrations makes a substantial difference. The main comparison also counts search and return steps, so the demonstrations are not treated as free expert data. Testing on held-out seeds is appropriate for the question being asked. I found Appendix F useful because it tests trajectory optimization separately, even though the result is largely negative after RL fine-tuning. These experiments show that the pipeline can help on the selected games. I would regard this as a useful application of existing methods, but not enough of a contribution for acceptance.

Critical weaknesses

The new solution-path sampling rule is a small change, and Appendix F reports no significant improvement from it after RL fine-tuning. With that result, I am left with an existing search procedure followed by standard BC and PPO, evaluated on another set of games. Collecting the demonstrations is allocated 800 million steps in Procgen and 150 million in MiniHack, across 20,000 available training instances. This dependence on extensive per-instance search is difficult to justify as a broadly useful approach. Appendix J shows collapse below 1,000 instances in its setting, so instance diversity matters and obtaining it requires more searches. Yet search is tested only in discrete-action games, with hand-designed cells and reliable return to saved states. There is no evidence here that random-action expansion remains practical in high-dimensional continuous control. Resetting is needed for data collection rather than policy execution, but that still restricts where the required training data can be obtained.

Questions for authors
  1. How many distinct instances actually supply demonstrations under the stated search budgets, and how much of the budget goes to unsuccessful searches? Can the authors report the time and resources for the complete pipeline on comparable hardware, including BC and PPO? Search throughput alone leaves this unclear.

  2. Why does Appendix J report collapse below 1,000 instances while Appendix G obtains gains with 500? Is critic warmup responsible for this difference? It would also help to separate the effect of increasing instance diversity from that of increasing the total search budget.

  3. Could the authors test a continuous-control task in a robot simulator such as MuJoCo, varying action dimension and the accuracy of state restoration? This would require an additional experiment, but would directly test whether the demonstration-generation procedure scales beyond the conditions used here.

3 comments
AuthorsOpenReview ↗

Response [1/2]

Thank you for taking the time to read our paper and provide your thoughts. In particular we appreciate your kind words regarding the strength and value of our results. We address your concerns regarding novelty and contributions in our overall response, and will discuss your remaining concerns below:

We would like to highlight that although GAGE generates demonstrations through per-instance search, Section 4.1.1 shows that GAGE is more sample efficient than popular RL baselines, achieving higher scores in fewer samples. Appendix G demonstrates that these results hold with only 500 seeds and 200M environment steps. Section 4.1.2 further shows that GAGE performs better with additional samples, while existing RL baselines plateau early in training. These findings support the conclusion that GAGE outperforms PPO and RND with a fixed interaction budget and that this lead grows with additional data.

Although trajectory optimization is a relatively small change, it significantly improves the performance of GAGE-BC in Heist and Jumper and matches performance in Maze and Coinrun. While the difference is not significant after RL finetuning, GAGE-BC is itself a valuable method that outperforms PPO and RND on average while requiring 68% of the CPU-hours and 16% of the GPU-hours of GAGE-RL (Appendix L). Trajectory optimization is not central to our contributions, but we believe it is a valuable addition to the Go-Explore literature.

Limitations: We acknowledge that GAGE has several limitations in our paper. As you state, GAGE currently relies on hand designed states and a deterministic environment simulator. However, modern AI is often trained in simulated environments, so this limitation may not significantly decrease GAGE’s applicability in practice. The reliance on cell representations is a key limitation of the method, which prior work has attempted to address [1][2]. Our work targets an orthogonal limitation of the original Go-Explore, which is its ability to generalize across many environments. We hope that future work will build on GAGE to design cell representations that are applicable to broad distributions of environments.

Environments and Q3: While we agree that testing GAGE in a wider range of environments would be valuable, we believe that this is beyond the scope of our paper. Procgen and MiniHack are established benchmarks for exploration and generalization [3][4][5] and our experiments target particularly challenging sparse-reward environments within these domains. In Procgen, most prior work studies the “easy” distribution, while we target the “hard” distribution that presents substantially more difficult exploration challenges. These environments also test navigation, long horizon planning, and partial observability.

Prior work has demonstrated that Go-Explore is widely applicable to continuous control, robotics, and even text-based domains. Studying the end-to-end sample efficiency of GAGE in such environments would be a valuable direction for future work, but our results establish meaningful improvements in challenging discrete action settings.

Q1a: GAGE searches trajectories one by one, and did not solve all 20,000 of the available environment instances. In practice, given the budgets and stopping conditions used in Section 4.1.1, GAGE solves between 31 (MonsterLava) and 4495 (Maze) instances, depending on the environment, with solve rates ranging from 100% of attempted seeds in all Procgen games to 32% in Monster Lava. This means that 130M steps were spent on unsuccessful searches in MonsterLava, the hardest domain, whereas all search attempts are successful in Procgen. Without budget restrictions, such as for the datasets collected in Section 4.1.2, Go-Explore solves the vast majority of MiniHack instances as well.

Q1b: We did not conduct our experiments on equivalent hardware, so it is not possible to report an exact fair comparison between the methods. Roughly speaking, data collection (800M steps) consumed 12-24 hours on a 12 CPU node, BC required 3-6 hours on an L40s GPU, and RL fine-tuning required 12-24 hours on a GPU node with 8-12 CPUs. PPO and RND (1B steps) required 3-5 days on a GPU node with 8-12 CPUs, with RND typically taking longer. In our experiments, GAGE incurred significantly lower costs than the baselines (Appendix L), largely due to the extremely fast GPU-free data collection process.

AuthorsOpenReview ↗

Response [2/2]

Q2a: Thank you for pointing out this discrepancy. Both experiments use the same critic warmup period, but this detail was accidentally omitted from the main text. We will update the main text accordingly. Appendix J and G report different results for a small number of demonstrations, likely due to differing KL constraints. The 500 seed experiments in Appendix G are tuned for that setting and use a stronger KL constraint weight of 1.0, while the 1000 seed experiments in Appendix J use the tuned value of 0.5 from the main experiments in Section 4. We will update the text of both Appendices to note these details and update Appendix N to include the hyperparameters for Appendix G.

Q2b: We study the impact of increasing the search budget in Section 4.1.2, and the impact of increasing the number of instances in Appendix J. Appendix J allows up to 2M search steps per instance, so it does not isolate the effect of increasing instances from increasing search steps. Studying the effect of increasing instances with a fixed search budget would be an interesting future study.

Thank you for taking the time to thoroughly consider our work and helping to identify points of confusion in our experiments. If you feel that we have addressed your questions and concerns, please consider raising your score to support sharing our results with the ICLR community.

[1] Gallouédec, Quentin, and Emmanuel Dellandréa. "Cell-free latent go-explore." International Conference on Machine Learning. PMLR, 2023.

[2] Höftmann, Marc, Jan Robine, and Stefan Harmeling. "Time-myopic go-explore: learning a state representation for the go-explore paradigm." arXiv preprint arXiv:2301.05635 (2023).

[3] Jiang, Yiding, J. Zico Kolter, and Roberta Raileanu. "On the importance of exploration for generalization in reinforcement learning." Advances in Neural Information Processing Systems 36 (2023): 12951-12986.

[4] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.

[5] Zisselman, Ev, et al. "Explore to generalize in zero-shot rl." Advances in Neural Information Processing Systems 36 (2023): 63174-63196.

Thank you for the response. I have read it carefully, but my concerns remain, so I will keep my score. The rebuttal does not include new experiments, and the paper has not been revised. The two questions that needed new evidence were set aside: separating instance diversity from search budget is left for future work, and continuous control is considered out of scope. The setting is also narrow and heavily constrained: all environments are discrete-action 2D games, and the method requires hand-designed cells and exact restoration of saved states. The rebuttal acknowledges these requirements but offers no evidence that they can be relaxed. The numbers the rebuttal does give raise a further concern. The matched-budget comparison is described as training on 20,000 instances, yet demonstrations come from only 31 to 4,495 of them. For MonsterLava, 31 instances is far below the scale at which Appendix J reports RL fine-tuning collapsing, and GAGE-RL indeed stays at about 1% there. On novelty, the rebuttal agrees that trajectory optimization is not central, and Appendix F shows it has no significant effect after RL fine-tuning. The remaining parts, namely per-instance Go-Explore, early stopping, BC and KL-regularized PPO, are existing components. The strongest result does not make up for this either: the 89% vs. 28% comparison is not matched in samples, because GAGE is given a larger search budget, justified by wall-clock time.

Reviewer_ZhTpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAGE, a method to address the exploration problem in sparse reward environments in reinforcement learning. It has multiple stages of training. First, it uses go-explore to explore the environment to reach the goal and collect training data. Then it leverages the training data to do behavior cloning to distill a policy and then further uses reinforcement learning to further fine-tune the policy. Results show improved performance on the tested environment.

Critical strengths
  • the idea is simple yet effective by combining go-explore to collect successful trajectories and then use that to train or finetune a policy
Critical weaknesses
  • I would suggest the authors to clearly distinguish the work from previous work and then make it clear what the real contribution of Gage is. After reading the paper, I think the experimental comparison is unfair and not properly designed. There are several things, first, if we frame Gage as an exploration method, then I think Gage's exploration part is previous work go-explore, thus it is doesn't make sense to compare it with RND and PPO, or even more recent exploration methods such as DEIR [1]. If the main contribution of Gage is proposing a new method of training, I think you should at least discuss some of the related works that also use pretraining/ or collect trajectories and post-training policy techniques, such as the method proposed in [2].

[1] Wan S, Tang Y, Tian Y, et al. Deir: efficient and robust exploration through discriminative-model-based episodic intrinsic rewards[J]. arXiv preprint arXiv:2304.10770, 2023.

[2] Ye Z, Chau P, Emunds R, et al. Boosting deep Reinforcement Learning using pretraining with Logical Options[J]. arXiv preprint arXiv:2603.06565, 2026.

  • Overclaiming. The authors have shown some generalization capability of Gage across the four Procgen tasks. After reading the claim that the policy can generalize, I am expecting a policy that can work cross-domain. Within-task family generalization is good in itself, but please use the term "generalization" properly.
Questions for authors
  • How would the method distinguish a novel state? I read in the paper that the states are currently predefined. It is fine as a limitation and to set automatic novel state discovery as future work, but what is the limit of the proposed method? How large an environment will make Gage/ go-explore crash? and what if there are many novel states there which will not lead to a success?

see weakness. I am happy to discuss with the authors during rebuttal

2 comments
AuthorsOpenReview ↗

Response [1/2]

Thank you for taking the time to review our work and for kindly noting the effectiveness and simplicity of our method. We also appreciate your openness to discussing the concerns that you raised. We discuss each of the points you raised below:

Contributions and Baselines: We discuss the core contributions of GAGE in our overall response. Regarding comparisons to other work, GAGE is a method that learns policies that can perform well across distributions of environment instances without prior data. As such, it addresses the same problem as online RL methods, such as PPO and RND. We evaluate the end-to-end effectiveness of GAGE as a learning procedure by comparing these baselines with equivalent environment interaction budgets, including the Go-Explore search and return steps. Methods that assume access to prior datasets are not directly comparable, because GAGE and online RL methods generate their own data, and because we do not have preexisting datasets to evaluate such methods. We explore alternative approaches to generating trajectories in Appendix E, and find that they have lower solve rates and dramatically higher costs than Go-Explore.

Related Work: Thank you for pointing out H2RL, we were not aware of this recent arXiv preprint and will expand our related works section to include logical options. To briefly discuss the difference here, logical options, as presented in this paper, do not address the challenge of generalization in sparse reward environments studied in our work. They instead focus on learning policies for individual environments as in the original Go-Explore and much of the exploration literature. H2RL assumes access state abstractions, predefined logical options, meaningful subtasks, and the ability to modify the environment. This requires more domain knowledge and more strict assumptions on the environment than GAGE, which requires only state abstractions and a deterministic simulation. These methods make different assumptions and address distinct learning problems.

Accuracy of Claims: We aim to make our discussion of generalization as accurate as possible, and we apologize that the writing was ambiguous. We followed the precedent of prior work such [1], which refer to agents that could be trained to play multiple games as “General Agents” and [2] which describe generalization to unseen test instances of a training environment. To remove any ambiguity, we will rename Section 4.3 from “Generalizing Across Games” to “Multi-Game Training” and clarify the evaluation procedure within that section:

We train each multi-game policy on all four Procgen environments, then evaluate them on unseen instances of those same four training environments. As such, we evaluate generalization to unseen instances of training environments, not generalization to unseen Procgen games.

We appreciate the feedback and will clarify the scope of generalization claims throughout the paper.

AuthorsOpenReview ↗

Response [2/2]

Q1: Go-Explore distinguishes a novel state through a cell representation, which is a mapping from states to some lower-dimensional representation called a cell. The exact cell representations used are described in Appendix B. A single state is saved per cell. Any state which maps to a previously unseen cell is saved, and a state can replace the incumbent if it has a higher return, or an equal return with a shorter trajectory. Importantly, the states themselves are not predefined, only the cell representation is specified in advance, and the archive grows dynamically during exploration.

In Procgen, cells are downsampled images, with the number of collected keys added in Heist. In MiniHack we use select features from the game's observation space, such as the local glyphs around the agent and the agent’s experience level, shared across all games.

The method can succeed even with very simple cell representations but searches more slowly if there are many distracting cells. For example, if each state was mapped to its own cell, the archive would be excessively large and the search process would be less likely to sample interesting, useful states. The scalability depends on the cell representation and the environment. This is the main limitation of the method, and as such has been the focus of several papers building on Go-Explore [3][4]. We address an orthogonal limitation, which is extending Go-Explore beyond singletons to many highly variable environments.

Thank you again for taking the time to review our work and provide constructive feedback. We hope these explanations clarify our contribution and its relation to prior work. If you feel that your concerns have been addressed, please consider raising your score to support sharing our work with the ICLR community.

[1] Bellemare, Marc G., et al. "The arcade learning environment: An evaluation platform for general agents." Journal of artificial intelligence research 47 (2013): 253-279.

[2] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.

[3] Gallouédec, Quentin, and Emmanuel Dellandréa. "Cell-free latent go-explore." International Conference on Machine Learning. PMLR, 2023.

[4] Höftmann, Marc, Jan Robine, and Stefan Harmeling. "Time-myopic go-explore: learning a state representation for the go-explore paradigm." arXiv preprint arXiv:2301.05635 (2023).

Reviewer_6U94OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper is on using Go-Explore-style algorithms for so-called generalist agents. It focuses on applying Go-Explore to procedurally-generated environments rather than individual games.

Critical strengths

The Go-Explore style of algorithm is cute & it is nice to see it revisited for a broader set of problems. It is also nice to see experiments across many different games.

Critical weaknesses

I believe that the contribution is too minimal to be accepted as a standalone paper. The paper also does not articulate any meaningful distinctions from the original Go-Explore paper except for possibly conditioning on the problem instance. However, even then it is uncertain because the setup is not formally stated anywhere in the main body of the paper.

Questions for authors
  1. How is this work different from Go-Explore?
  2. Could the authors please clarify what it means to "generalize across problem instances"?
1 comment

Thank you for taking the time to review our work and for praising the breadth of our experimental results.

We address the differences between GAGE and prior work in our overall response. To summarize, GAGE is the first go-explore-based method to train policies that generalize beyond a single environment. The original Go-Explore uses search and a reverse curriculum [1] to train a policy for a single stochastic environment. GAGE instead uses Go-Explore to generate demonstrations across many problem instances, aggregates them into a dataset, and then trains a policy through BC and RL that generalizes to unseen instances of the training environment. It introduces per-instance search, stopping rules, and trajectory optimization to make this process efficient.

We demonstrate that, despite searching each instance individually, GAGE is surprisingly more sample efficient than PPO and RND in sparse reward generalization benchmarks, and that this advantage remains large as sparsity increases. We will update the background to clarify these differences to Go-Explore. We acknowledge that GAGE utilizes several well known components, but the combination of these algorithms has never been studied, and we show that it is effective and scales favorably with additional data. We believe that GAGE, and Go-Explore more broadly, represents an underutilized approach to exploration, and that our findings would be of interest to the broader RL community at ICLR.

Regarding your question about generalization, we define generalization, the problem setting, and our evaluation procedure in Section 2.2, repeated here for convenience:

We model these benchmarks as underspecified partially observable Markov decision processes (UPOMDPs), where a free parameter θ ∈ Θ determines the fully specified POMDP for an episode. In a maze environment, for example, θ may be the random generation seed, and the resulting POMDP is the specific maze generated by that seed. In our experiments, θ is the level-generation seed and each sampled value of θ defines one problem instance. We train policies on a set of sampled instances and evaluate generalization on held-out values of θ not seen during training.

We hope this clarifies the setting we study, and would welcome any suggestions for making the explanation more clear.

In our experiments, we measure solve rates on test instances of an environment not seen during training. Higher held-out test scores indicate better generalization. We show that GAGE agents can achieve significant returns in these settings, solving at least 67% of unseen instances in every tested Procgen environment, compared to 15% for PPO and 20% for RND.

Given your soundness rating of 1, we would also appreciate clarification on whether you have specific concerns regarding the technical soundness of our approach, beyond the questions of novelty and problem formulation discussed above. If you feel that this response has addressed your concerns, please consider raising your score to support sharing our findings with the ICLR community.

[1] Salimans, Tim, and Richard Chen. "Learning montezuma's revenge from a single demonstration." arXiv preprint arXiv:1812.03381 (2018).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response

Thank you all for taking the time to review our work, and for providing thoughtful feedback. We will address the key concerns raised by the reviewers below:

Novelty: GAGE is the first work to train an agent using Go-Explore to perform well in more than a single environment. Prior work on Go-Explore has exclusively trained policies in singleton environments. In contrast, GAGE performs well on distinct instances of procedurally generated environments, and across multiple distinct environment distributions. GAGE simultaneously addresses sparse reward and generalization challenges. We achieve this generalization by individually solving multiple environment instances and replacing Go-Explore's backwards algorithm with BC and RL.

Contributions: We demonstrate that it is possible (using GAGE) for Go-Explore-based methods to generalize to instances of challenging sparse reward benchmarks not seen during training (Section 4.1). GAGE has many additional advantages over pure RL methods. Standard RL approaches can fail to improve with additional samples, but scaling Go-Explore’s search budget reliably improves performance across the environments tested (Section 4.1.2 and Appendix J). We also show that GAGE is less affected by increasing sparsity than popular RL baselines (Section 4.2). Finally, GAGE can train a single generalist policy on four Procgen games that performs well on held out instances of these games, offering an alternative approach to multi-task learning in these settings (Section 4.3).

Agents trained with GAGE demonstrably generalize to unseen instances of visually complex, hard exploration environments, and this is the first instance of Go-Explore-based agent doing so. Looking forward, these results suggest that collecting data on a wider distribution of environments may allow GAGE to train increasingly general agents, even in extremely sparse reward domains. GAGE’s approach to exploration is orthogonal to more popular approaches like intrinsic motivation or curriculum learning and future work can thus combine GAGE with complementary progress in these areas.

Our strong empirical results demonstrate the advantages of both GAGE as framework and Go-Explore more generally. We believe GAGE warrants further study as it represents an alternative to the many prevailing approaches to exploration, with unique capabilities and limitations. We will update the paper to make the novelty of GAGE and our contributions more salient. We strongly believe that GAGE has value to the ICLR community and the research community more broadly. We would greatly appreciate it if you could raise your score to support sharing our findings with the ICLR community, and more generally if you could advise on how to better communicate what we believe is a clear and strong contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.