- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2(+1)
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces an exploration mechanism that picks goals for the agent to explore to encourage visiting undercovered areas. Their goal selection mechanism also uses a mutual information term that measures the diversity of goals within a batch so that, apart from being under-visited, goals should also be relatively diverse. By discretizing the state space into multiple parts, they measure state visitation using a finite-budget entropy allocation formula. They measure the state visitation of their algorithm and compare it with baselines, and their method shows good performance in terms of state coverage.
Critical strengths
Exploration in online goal-reaching RL is a very important and active research problem in the field. The paper studies this problem by designing an algorithm for picking under-visited and diverse subgoals. The idea is intuitive and easy to understand.
Critical weaknesses
The idea of picking low visitation goals for exploration has been extensively studied in unsupervised RL, e.g., MEGA, skew-fit, CE² (Cluster Edge Exploration), and Go-Explore. This work doesn't add much conceptually to prior works. The use of mutual information to assess diversity among goals is new, but it doesn't make the work distinct enough to be suitable for a conference paper. It could make a very nice workshop paper. Moreover, the authors don't evaluate state coverage with these prior works that have such a close idea. If the work were empirically significantly stronger, I could agree that it is making enough contribution, but the paper lacks such evidence. Moreover, experiments only measure state coverage; it is not clear if visited states are semantically meaningful and useful states and can be used for doing goal-reaching tasks or not.
Apart from the contribution, the writing of the paper is very poor. I suspect AI is extensively used in directly writing sentences because in most parts the writing is made overly complicated to explain a very intuitive idea; Sec. 3.5 is an example of bad writing despite the concept being very simple and intuitive.
[Pitis et al., 2020, Pong et al., 2020, Ecoffet et al., 2021, Diaz-Bone et al., 2025]
Questions for authors
Can you provide more qualitative evidence about the goals you visit? Does the agent learn any useful behavior? Is this algorithm useful for downstream goal-reaching tasks?
8 comments
Response to Reviewer srHg part 2
[W3, Q2, Q3] Downstream usefulness beyond state coverage
The original submission evaluated frozen pretrained behaviors on zero-shot goal reaching and sequential HRL on AntMaze Medium and Large (Sec. 4.5, Table 3, Appendix C.5; original):
| Method | Zero-shot SR, Medium (%) | Zero-shot SR, Large (%) | HRL goals/episode, Medium | HRL goals/episode, Large |
|---|---|---|---|---|
| EGGS | 83.5 | 53.0 | 2.20 | 2.42 |
| TLDR | 83.7 | 48.1 | 2.41 | 2.39 |
| METRA | 20.8 | 14.7 | 0.89 | 1.66 |
These results show that the agent learns behaviors reusable for downstream goal reaching and hierarchical control.
[Q1] Qualitative evidence
Following this request, we visualized exploratory goals selected by EGGS at different training stages in all three AntMaze layouts (Fig. 5; new). Early goals are concentrated near the starting region, whereas later goals extend into more distant regions of each maze.
[W4] Presentation
The core idea of Sec. 3.5 fits in one example: with visit counts (0, 1) and goals, pointwise selection sends all four goals to the least-visited cell, producing (4, 1), whereas the entropy-optimal allocation is (3, 2). Proposition 4 characterizes when a diversity term recovers such allocations; we reorganized Sec. 3.5 around this example and the balancing property.
References
[1] Junik Bae, Kwanyoung Park, and Youngwoon Lee. TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations. In Conference on Robot Learning, 2024.
[5] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum Entropy Gain Exploration for Long Horizon Multi-Goal Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7750–7761, 2020.
[6] Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7783–7792, 2020.
[7] Yuanlin Duan, Guofeng Cui, and He Zhu. Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning. In Advances in Neural Information Processing Systems, 37:61160–61191, 2024.
[8] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First Return, Then Explore. Nature, 590(7847):580–586, 2021.
[9] Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable Unsupervised RL with Metric-Aware Abstraction. In The Twelfth International Conference on Learning Representations, 2024.
[10] Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter, and Andreas Krause. DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning. In Advances in Neural Information Processing Systems 38, 2025.
Thank you for the revisions and the additional experiments.
Despite carefully reading the paper again, I still find the presentation unnecessarily complicated and difficult to follow. Several parts of the motivation and mathematical formulation remain unclear to me, and I think the paper needs substantial editing before being ready for publication.
Questions about the presentation and methodology:
-
Line 187: The derivative is introduced very abruptly, without explaining why it is a useful quantity to consider. What does this derivative calculation actually add to the motivation? Couldn't we simply argue that we want to explore under-visited states, use the visitation density to measure how frequently states are visited, and assign higher scores to states with lower density? I don't understand why the derivative is needed or what makes this derivation particularly principled.
-
Line 229: What exactly are the "feasibility constraints" mentioned here? How are they defined?
-
Equation (2): I find the notation unnecessarily confusing. You first introduce a random goal index , then define the Gaussian distribution conditioned on . I guess that can denote a realization of ? The way you wrote it now, it seems is a realization of . wouldn't it be much clearer to write , where ? This would make the sampling procedure immediately clear.
-
Section 3.4: What exactly does the exploratory policy do after reaching a selected goal? Is it a random policy, a separately trained exploration policy, or something else? This seems particularly important since the motivation for selecting diverse goals is to reduce the overlap between the subsequent exploration trajectories.
-
Section 3.5: Despite the revisions, I still find this section unnecessarily complicated. As I understand it, the main idea is relatively simple: we want to select goals in under-visited regions while spreading the selected goals across different regions to avoid redundant exploration. However, this intuition is buried under several optimization formulations and technical terminology. For instance, sentences such as "We use an idealized model that treats each allocation as one visitation unit placed directly in a selected cell, thereby collapsing Go transit..." are very difficult to follow without first explaining the basic idea. The discussion of water-filling and the different allocation problems also makes the section harder to read than necessary.
I also find the mutual-information formulation more complicated than the underlying intuition seems to require. If the goal is to select under-visited and diverse goals, why not motivate the method directly in those terms? A balanced allocation across cells does not necessarily correspond to maximizing Gaussian mutual information in the learned representation.
-
Section 3.5, definition of cells: How exactly are the cells defined? Are they constructed based on similarity in the learned representation space, or are they arbitrary partitions of the state space? I couldn't find a clear definition in the main text. Given how much the method relies on these cells, I find this omission particularly frustrating. A simple visual example showing how the cells are defined, how goals are assigned to them, and how the allocation procedure works would make this section much easier to understand. More importantly, why don't you discuss the connection between the notion of balanced visitation goals and the maximization of the MI introduced before? I don't think they are always necessarily the same.
Response to Reviewer srHg part 1
We thank the reviewer for the detailed feedback. Major revisions are highlighted in blue in the revised PDF. In this response, original marks results in the submitted version and new marks results added in this revision.
Revision summary. We added a finite-budget result to Sec. 3.5 and Appendix A.2 (Proposition 4; new): in the idealized finite-cell model, with a unique least-visited cell and K≥d+2 (where d is the visitation-count gap to the second-least-visited cell), there exists a positive categorical diversity weight that strictly improves post-allocation visitation entropy over frozen pointwise selection. The original balancing results establish asymptotic convergence with or without diversity; the new result identifies when diversity improves a single allocation batch. It motivates the design of EGGS rather than providing a guarantee for the learned algorithm. We also reorganized the theoretical exposition and clarified the presentation of the method and experiments. We also corrected several implementation descriptions in Appendix B to match the code used for all reported results (e.g., actor activations, simulator version, and replay handling); no reported result changes.
[W1] Novelty relative to prior work
Prior approaches including MEGA [5], Skew-Fit [6], and CE² [7] have developed strategies for selecting exploratory goals. EGGS specifically addresses the joint allocation of multiple exploratory goals to rollouts collected under a shared learner snapshot, combining pointwise under-coverage utility with set-level Gaussian distinguishability. Independently selected high-utility goals can generate overlapping trajectories, potentially reducing marginal coverage per collection batch. We clarified this distinction in Sec. 2.2 and added the finite-budget analysis summarized above (Sec. 3.5, Proposition 4; new). MI's contribution is examined through fixed-K ablations (Tables 9 and 10; original), trajectory-level diagnostics (Table 2; original), end-to-end selector comparisons (Table 17; new), and environment-dependent MI effects (Table 11; new).
[W2] Empirical evidence and comparison with prior methods
Following this comment, we added MEGA [5] and Skew-Fit [6] to the controlled 2-D benchmark (Table 1; new rows). MEGA uses its released 2-D maze configuration without retuning; Skew-Fit is adapted to state observations using a VAE trained on standardized (x,y) coordinates, with hyperparameters selected on Square-C as for EGGS and the other tuned baselines (Sec. 4.2, Appendix C.1). Including the added MEGA and Skew-Fit baselines, EGGS achieves the highest mean coverage AUC in all six controlled 2-D environments and the highest mean final normalized visitation entropy in five of six (Table 1). On Square-Large, EGGS exceeds the next-best method, ExDM, by 8.2 percentage points in mean coverage AUC (54.7% vs. 46.5%).
CE² [7] uses a learned world model, while our quantitative comparisons focus on model-free unsupervised GCRL methods; we discuss the relationship to CE² in Sec. 2.2. Go-Explore [8] introduced the return-then-explore collection principle used by EGGS. Within this framework, we compare collection and goal-selection variants (Explore-only, single-goal, and Top-K; Table 9, original; Table 17, new), alongside TLDR [1] as an existing Go-Explore-style unsupervised GCRL baseline (Table 1; original). The methodological relationship is discussed in Sec. 2.3. DISCOVER [10], which selects goals toward a specified target task rather than for reward-free exploration, is discussed in Sec. 2.2.