acceptodds
Under review as a conference paper at ICLR 2027

EGGS: Entropy-Guided Joint Goal Selection for Long-Horizon Unsupervised Goal-Conditioned Reinforcement Learning

Abstract

We consider unsupervised goal-conditioned reinforcement learning, where the agent aims to learn a policy that can reach diverse goals without task-specific rewards. To this end, existing methods select goals using visitation novelty or predicted exploration value to guide the agent toward new states, but when several goals are used in the same collection batch, they may be close to one another, leading to overlapping exploration across rollouts. We introduce EGGS (Entropy-Guided Joint Goal Selection), which jointly selects goals to reduce redundancy across exploration rollouts. EGGS combines a visitation novelty score motivated by visitation entropy with a mutual-information term that measures distinguishability among the selected goals in a learned temporal representation. Each selected goal guides a Go-Explore rollout, which first moves toward the goal and then continues exploration from the resulting state. Compared with selecting goals using visitation novelty alone, EGGS selects more distinguishable goals and collects trajectories with less overlap while retaining high novelty scores. In controlled 2-D environments, EGGS achieves the highest mean coverage AUC in all six tasks and the highest mean final normalized visitation entropy in five of six, demonstrating rapid coverage and broadly distributed visitation. Across AntMaze and locomotion benchmarks, EGGS attains the highest mean coverage AUC in four of six environments, and its pretrained behaviors support zero-shot goal reaching and sequential multi-goal control.

Primary area:
reinforcement learning
Keywords:
reward-free reinforcement learning, exploration, goal-conditioned reinforcement learning, maximum-entropy exploration, unsupervised reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_srHgOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces an exploration mechanism that picks goals for the agent to explore to encourage visiting undercovered areas. Their goal selection mechanism also uses a mutual information term that measures the diversity of goals within a batch so that, apart from being under-visited, goals should also be relatively diverse. By discretizing the state space into multiple parts, they measure state visitation using a finite-budget entropy allocation formula. They measure the state visitation of their algorithm and compare it with baselines, and their method shows good performance in terms of state coverage.

Critical strengths

Exploration in online goal-reaching RL is a very important and active research problem in the field. The paper studies this problem by designing an algorithm for picking under-visited and diverse subgoals. The idea is intuitive and easy to understand.

Critical weaknesses

The idea of picking low visitation goals for exploration has been extensively studied in unsupervised RL, e.g., MEGA, skew-fit, CE² (Cluster Edge Exploration), and Go-Explore. This work doesn't add much conceptually to prior works. The use of mutual information to assess diversity among goals is new, but it doesn't make the work distinct enough to be suitable for a conference paper. It could make a very nice workshop paper. Moreover, the authors don't evaluate state coverage with these prior works that have such a close idea. If the work were empirically significantly stronger, I could agree that it is making enough contribution, but the paper lacks such evidence. Moreover, experiments only measure state coverage; it is not clear if visited states are semantically meaningful and useful states and can be used for doing goal-reaching tasks or not.

Apart from the contribution, the writing of the paper is very poor. I suspect AI is extensively used in directly writing sentences because in most parts the writing is made overly complicated to explain a very intuitive idea; Sec. 3.5 is an example of bad writing despite the concept being very simple and intuitive.

[Pitis et al., 2020, Pong et al., 2020, Ecoffet et al., 2021, Diaz-Bone et al., 2025]

Questions for authors

Can you provide more qualitative evidence about the goals you visit? Does the agent learn any useful behavior? Is this algorithm useful for downstream goal-reaching tasks?

8 comments
AuthorsOpenReview ↗

Response to Reviewer srHg part 1

We thank the reviewer for the detailed feedback. Major revisions are highlighted in blue in the revised PDF. In this response, original marks results in the submitted version and new marks results added in this revision.

Revision summary. We added a finite-budget result to Sec. 3.5 and Appendix A.2 (Proposition 4; new): in the idealized finite-cell model, with a unique least-visited cell and K≥d+2K\ge d+2 (where dd is the visitation-count gap to the second-least-visited cell), there exists a positive categorical diversity weight that strictly improves post-allocation visitation entropy over frozen pointwise selection. The original balancing results establish asymptotic convergence with or without diversity; the new result identifies when diversity improves a single allocation batch. It motivates the design of EGGS rather than providing a guarantee for the learned algorithm. We also reorganized the theoretical exposition and clarified the presentation of the method and experiments. We also corrected several implementation descriptions in Appendix B to match the code used for all reported results (e.g., actor activations, simulator version, and replay handling); no reported result changes.

[W1] Novelty relative to prior work

Prior approaches including MEGA [5], Skew-Fit [6], and CE² [7] have developed strategies for selecting exploratory goals. EGGS specifically addresses the joint allocation of multiple exploratory goals to rollouts collected under a shared learner snapshot, combining pointwise under-coverage utility with set-level Gaussian distinguishability. Independently selected high-utility goals can generate overlapping trajectories, potentially reducing marginal coverage per collection batch. We clarified this distinction in Sec. 2.2 and added the finite-budget analysis summarized above (Sec. 3.5, Proposition 4; new). MI's contribution is examined through fixed-KK ablations (Tables 9 and 10; original), trajectory-level diagnostics (Table 2; original), end-to-end selector comparisons (Table 17; new), and environment-dependent MI effects (Table 11; new).

[W2] Empirical evidence and comparison with prior methods

Following this comment, we added MEGA [5] and Skew-Fit [6] to the controlled 2-D benchmark (Table 1; new rows). MEGA uses its released 2-D maze configuration without retuning; Skew-Fit is adapted to state observations using a VAE trained on standardized (x,y)(x,y) coordinates, with hyperparameters selected on Square-C as for EGGS and the other tuned baselines (Sec. 4.2, Appendix C.1). Including the added MEGA and Skew-Fit baselines, EGGS achieves the highest mean coverage AUC in all six controlled 2-D environments and the highest mean final normalized visitation entropy in five of six (Table 1). On Square-Large, EGGS exceeds the next-best method, ExDM, by 8.2 percentage points in mean coverage AUC (54.7% vs. 46.5%).

CE² [7] uses a learned world model, while our quantitative comparisons focus on model-free unsupervised GCRL methods; we discuss the relationship to CE² in Sec. 2.2. Go-Explore [8] introduced the return-then-explore collection principle used by EGGS. Within this framework, we compare collection and goal-selection variants (Explore-only, single-goal, and Top-K; Table 9, original; Table 17, new), alongside TLDR [1] as an existing Go-Explore-style unsupervised GCRL baseline (Table 1; original). The methodological relationship is discussed in Sec. 2.3. DISCOVER [10], which selects goals toward a specified target task rather than for reward-free exploration, is discussed in Sec. 2.2.

AuthorsOpenReview ↗

Response to Reviewer srHg part 2

[W3, Q2, Q3] Downstream usefulness beyond state coverage

The original submission evaluated frozen pretrained behaviors on zero-shot goal reaching and sequential HRL on AntMaze Medium and Large (Sec. 4.5, Table 3, Appendix C.5; original):

MethodZero-shot SR, Medium (%)Zero-shot SR, Large (%)HRL goals/episode, MediumHRL goals/episode, Large
EGGS83.553.02.202.42
TLDR83.748.12.412.39
METRA20.814.70.891.66

These results show that the agent learns behaviors reusable for downstream goal reaching and hierarchical control.

[Q1] Qualitative evidence

Following this request, we visualized exploratory goals selected by EGGS at different training stages in all three AntMaze layouts (Fig. 5; new). Early goals are concentrated near the starting region, whereas later goals extend into more distant regions of each maze.

[W4] Presentation

The core idea of Sec. 3.5 fits in one example: with visit counts (0, 1) and K=4K=4 goals, pointwise selection sends all four goals to the least-visited cell, producing (4, 1), whereas the entropy-optimal allocation is (3, 2). Proposition 4 characterizes when a diversity term recovers such allocations; we reorganized Sec. 3.5 around this example and the balancing property.

References

[1] Junik Bae, Kwanyoung Park, and Youngwoon Lee. TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations. In Conference on Robot Learning, 2024.

[5] Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum Entropy Gain Exploration for Long Horizon Multi-Goal Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7750–7761, 2020.

[6] Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-Fit: State-Covering Self-Supervised Reinforcement Learning. In Proceedings of the 37th International Conference on Machine Learning, PMLR 119:7783–7792, 2020.

[7] Yuanlin Duan, Guofeng Cui, and He Zhu. Exploring the Edges of Latent State Clusters for Goal-Conditioned Reinforcement Learning. In Advances in Neural Information Processing Systems, 37:61160–61191, 2024.

[8] Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First Return, Then Explore. Nature, 590(7847):580–586, 2021.

[9] Seohong Park, Oleh Rybkin, and Sergey Levine. METRA: Scalable Unsupervised RL with Metric-Aware Abstraction. In The Twelfth International Conference on Learning Representations, 2024.

[10] Leander Diaz-Bone, Marco Bagatella, Jonas Hübotter, and Andreas Krause. DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning. In Advances in Neural Information Processing Systems 38, 2025.

Thank you for the revisions and the additional experiments.

Despite carefully reading the paper again, I still find the presentation unnecessarily complicated and difficult to follow. Several parts of the motivation and mathematical formulation remain unclear to me, and I think the paper needs substantial editing before being ready for publication.

Questions about the presentation and methodology:

  1. Line 187: The derivative is introduced very abruptly, without explaining why it is a useful quantity to consider. What does this derivative calculation actually add to the motivation? Couldn't we simply argue that we want to explore under-visited states, use the visitation density mt(x)m_t(x) to measure how frequently states are visited, and assign higher scores to states with lower density? I don't understand why the derivative is needed or what makes this derivation particularly principled.

  2. Line 229: What exactly are the "feasibility constraints" mentioned here? How are they defined?

  3. Equation (2): I find the notation unnecessarily confusing. You first introduce a random goal index CC, then define the Gaussian distribution conditioned on CC. I guess that ii can denote a realization of CC? The way you wrote it now, it seems ii is a realization of x~\tilde x. wouldn't it be much clearer to write X~=ϕt(gC)+ϵ\tilde X=\phi_t(g_C)+\epsilon, where ϵ∼N(0,σI2I)\epsilon\sim\mathcal N(0,\sigma_I^2I)? This would make the sampling procedure immediately clear.

  4. Section 3.4: What exactly does the exploratory policy do after reaching a selected goal? Is it a random policy, a separately trained exploration policy, or something else? This seems particularly important since the motivation for selecting diverse goals is to reduce the overlap between the subsequent exploration trajectories.

  5. Section 3.5: Despite the revisions, I still find this section unnecessarily complicated. As I understand it, the main idea is relatively simple: we want to select goals in under-visited regions while spreading the selected goals across different regions to avoid redundant exploration. However, this intuition is buried under several optimization formulations and technical terminology. For instance, sentences such as "We use an idealized model that treats each allocation as one visitation unit placed directly in a selected cell, thereby collapsing Go transit..." are very difficult to follow without first explaining the basic idea. The discussion of water-filling and the different allocation problems also makes the section harder to read than necessary.

    I also find the mutual-information formulation more complicated than the underlying intuition seems to require. If the goal is to select under-visited and diverse goals, why not motivate the method directly in those terms? A balanced allocation across cells does not necessarily correspond to maximizing Gaussian mutual information in the learned representation.

  6. Section 3.5, definition of cells: How exactly are the cells defined? Are they constructed based on similarity in the learned representation space, or are they arbitrary partitions of the state space? I couldn't find a clear definition in the main text. Given how much the method relies on these cells, I find this omission particularly frustrating. A simple visual example showing how the cells are defined, how goals are assigned to them, and how the allocation procedure works would make this section much easier to understand. More importantly, why don't you discuss the connection between the notion of balanced visitation goals and the maximization of the MI introduced before? I don't think they are always necessarily the same.

Reviewer_yPLvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EGGS, a method for improving exploration in unsupervised goal-conditioned reinforcement learning by jointly selecting exploratory goals rather than considering each goal independently. The central idea is that individually promising goals may lead to overlapping trajectories and redundant exploration when selected together. EGGS addresses this by combining a visitation-based novelty score with a mutual-information objective that encourages distinguishable goals in a learned temporal representation. The authors motivate their approach through visitation entropy and evaluate it across 2D navigation, AntMaze, and locomotion environments.

Critical strengths
  • The paper addresses a clear and well-motivated problem: independently selecting exploratory goals can lead to redundant exploration when multiple trajectories are collected together.

  • The proposed method is relatively simple and principled, combining visitation novelty with a mutual-information objective without introducing unnecessary architectural complexity.

  • The theoretical analysis provides a useful justification for balancing novelty and diversity, rather than relying entirely on empirical intuition.

  • The analysis in Table 2 provides convincing evidence that EGGS reduces trajectory overlap while retaining high novelty scores, directly supporting the proposed mechanism.

Critical weaknesses
  • The ablation studies show that the mutual-information term provides inconsistent improvements in exploration performance. In particular, it does not improve mean coverage AUC on AntMaze Large, making its contribution to the overall performance gains less clear.

  • The downstream goal-reaching results in Table 3 are mixed, with EGGS performing similarly to or slightly worse than TLDR in several settings. This limits the evidence that improved exploration produces more useful goal-conditioned policies.

  • While Table 2 demonstrates reduced trajectory overlap, it does not establish whether these trajectories actually discover more previously unexplored states. The reported number of distinct visited cells includes states already encountered during training.

Questions for authors

My only concern is with regards to exploration efficiency: could the authors report the number of previously undiscovered cells reached during the matched-snapshot experiments in Table 2? This would help establish whether reduced trajectory overlap translates into genuinely improved exploration. For instance, we could be reducing trajectory overlap in some parts of the trajectory while visiting the same states for another part of it.

4 comments
AuthorsOpenReview ↗

Response to Reviewer yPLv

We thank the reviewer for the thoughtful suggestion. Major revisions are highlighted in blue in the revised PDF. In this response, original marks results in the submitted version and new marks results added in this revision or reported in this response.

Revision summary. We added a finite-budget result to Sec. 3.5 and Appendix A.2 (Proposition 4; new): in the idealized finite-cell model, with a unique least-visited cell and K≥d+2K\ge d+2 (where dd is the visitation-count gap to the second-least-visited cell), there exists a positive categorical diversity weight that strictly improves post-allocation visitation entropy over frozen pointwise selection. The original balancing results establish asymptotic convergence with or without diversity; the new result identifies when diversity improves a single allocation batch. It motivates the design of EGGS rather than providing a guarantee for the learned algorithm. We also reorganized the theoretical exposition and clarified the presentation of the method and experiments. We also corrected several implementation descriptions in Appendix B to match the code used for all reported results (e.g., actor activations, simulator version, and replay handling); no reported result changes.

[W1] Contribution of MI on AntMaze Large

MI reduced trajectory overlap from 0.231 to 0.134 while retaining near-top under-coverage utility in the matched AntMaze Large diagnostic (Table 2; original). Across the tested MI settings, mean coverage AUC ranged from 37.7% to 39.4%, around the no-MI mean of 38.6% (Table 9; original). In the controlled 2-D mazes, the MI gain tended to be larger where high-utility candidates formed more spatial clusters (Table 11; new). On Square-Large, all four tested positive distinguishability weights improved mean coverage AUC by 1.00–2.79 percentage points (Table 12; new).

[W2] Downstream performance

The original submission evaluated frozen pretrained behaviors on zero-shot goal reaching and sequential HRL (Table 3, Appendix C.5; original):

MethodZero-shot SR, Medium (%)Zero-shot SR, Large (%)HRL goals/episode, MediumHRL goals/episode, Large
EGGS83.553.02.202.42
TLDR83.748.12.412.39
METRA20.814.70.891.66

EGGS achieves downstream performance comparable to TLDR and substantially above METRA. Its primary contribution lies in exploration under matched interaction budgets: EGGS achieves the highest mean coverage AUC among the compared methods in all six controlled 2-D environments and four of six AntMaze and locomotion environments (Tables 1 and 7). These downstream evaluations further demonstrate that the learned goal-conditioned behaviors remain reusable for zero-shot goal reaching and hierarchical control.

[W3, Q1] Previously undiscovered cells and exploration efficiency

Following this suggestion, we measured historical new-cell discovery against the cumulative set of cells visited up to each frozen checkpoint, independently of replay retention (new analysis reported in this response). The visited-cell set was held fixed across selectors and diagnostic repetitions at each checkpoint. We report this new measure on the 6M diagnostic batches of Table 2 and at an additional earlier 3M checkpoint (for the three seeds whose checkpoints were available), using the same novelty definition and 10 trials × 3 repetitions per seed.

CheckpointMetricTop-KEGGSFPS
3MTrajectory overlap0.250 ± 0.0570.138 ± 0.0110.112 ± 0.003
Historical new cells / batch0.000 ± 0.0000.044 ± 0.0510.011 ± 0.019
6MTrajectory overlap0.2310.1340.111
Historical new cells / batch0.153 ± 0.0990.133 ± 0.0750.060 ± 0.060

Values are mean ± SD where available; the 6M trajectory-overlap entries are the mean values reported in Table 2 (original).

Under frozen policies, historical new-cell discovery was rare at both checkpoints. EGGS reduced mean trajectory overlap at both stages. At 3M, EGGS discovered more historically new cells on average than Top-K (0.044 vs. 0.000 cells per batch); at 6M, their mean discovery counts were similar (paired EGGS–Top-K difference: −0.020 ± 0.080 cells per batch). FPS achieved the lowest mean overlap at both checkpoints but discovered fewer historically new cells than EGGS.

Because the diagnostic collects only one batch with frozen policies, we also examined full training on Square-Large (Table 17; new). Compared with Top-K, EGGS achieved lower trajectory overlap (0.033 vs. 0.043), more new cells per cycle (8.49 vs. 8.06), and higher coverage AUC (54.7% vs. 52.0%). FPS attained the lowest overlap (0.020) but discovered the fewest new cells per cycle (7.21) and had the lowest AUC (49.2%). Together, these results motivate considering both under-coverage utility and goal-set diversity.

Thank you again for your thoughtful questions. They helped us improve the paper. If our responses have addressed your concerns, we would appreciate it if you would consider updating your score. We are happy to answer any remaining questions.

Dear Authors,

Thank you for your additional experiments. I'm generally happy with the results although the actual conclusiveness of the ablations is rather subtle. Even though I won't raise my score I am happy to give this paper an accept. I think the authors have thoroughly explored the claims they propose and the paper makes an interesting contribution combining Go-Explore and information theory.

I do somewhat agree with reviewer srHg, in terms of the complexity of the analysis. While the finite-cell analysis is an interesting perspective it gives the reader not that much insight and value other than justifying the own method. I believe in those cases it would be better to show the clear ablations instead as proof of the working method rather than an abstract setting that lacks generality. I would personally move the finite cell analysis to the appendix.

Dear Reviewer yPLv,

Thank you for your support and for the thoughtful discussion throughout the review. Since Reviewer srHg raised a similar point about the finite-cell analysis, in the final version we will shorten Sec. 3.5 to its key intuition and result, move the detailed analysis to the appendix, and use the space to bring the full algorithm and key ablation results into the main text.

Reviewer_MdmCOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a new method called EGGS for reward-free exploration that combines visitation novelty (under-coverage) with a mutual-information term that measures distinguishability among selected goals in a learned temporal representation. They integrate this objective into a Go-Explore-style mechanism. They show that EGGS reduces redundancy among selected goals while retaining high novelty utility, balancing the two objectives. They conduct empirical experiments on controlled 2D environments, AntMaze, and locomotion tasks, showing that EGGS improves mean coverage AUC and normalized visitation entropy in most cases.

Critical strengths
  • The paper is easy to follow, clear, and well written.

  • The motivation to avoid redundancy among exploratory goals is strong and relevant to the field. The proposed methodology addresses the trade-off between novelty and diversity, with theoretical motivation provided through the finite-cell analysis. This is a nice contribution, supported by empirical exploration results across different environments and comparisons with several baseline methods (TLDR, METRA, RND, DIAYN, CIC, etc.), using metrics such as coverage AUC and normalized visitation entropy.

Critical weaknesses
  • The paper's central methodological contribution is the joint novelty–distinguishability objective. However, the ablation on AntMaze Large does not demonstrate a clear benefit from adding the Gaussian MI component. This raises the question of how much of EGGS's improvement comes from the specific Gaussian MI formulation.

  • The empirical results are somewhat mixed in certain environments, especially in AntMaze and locomotion tasks. Moreover, the downstream results on sequential HRL and zero-shot goal-reaching tasks are less convincing compared to some other methods. This raises the question of whether better exploration coverage consistently translates into better downstream performance.

Questions for authors
  1. Contribution of Gaussian MI: The AntMaze Large ablation does not show an improvement from adding the MI term. How important is Gaussian MI joint selection for the overall gains? How does EGGS compare with simpler joint selectors, such as novelty combined with pairwise-distance penalties or farthest-point sampling?

  2. Sensitivity to the Gaussian noise scale: How robust is the method to the choice of Gaussian noise scale σI\sigma_I? Is the performance sensitive to this hyperparameter?

  3. Computational overhead: What is the computational overhead introduced by the joint goal-selection procedure, particularly the Gaussian MI estimation?

  4. Effect of the learned representation: The learned temporal representation is used to estimate visitation under-coverage, measure distinguishability among selected goals, and define progress toward a selected goal. How sensitive are these components to the quality or choice of the representation? Why was TLDR's representation chosen specifically, and how would using a different representation affect the method?

3 comments
AuthorsOpenReview ↗

Response to Reviewer MdmC part 1

We thank the reviewer for the constructive feedback. Major revisions are highlighted in blue in the revised PDF. In this response, original marks results in the submitted version and new marks results added in this revision.

Revision summary. We added a finite-budget result to Sec. 3.5 and Appendix A.2 (Proposition 4; new): in the idealized finite-cell model, with a unique least-visited cell and K≥d+2K\ge d+2 (where dd is the visitation-count gap to the second-least-visited cell), there exists a positive categorical diversity weight that strictly improves post-allocation visitation entropy over frozen pointwise selection. The original balancing results establish asymptotic convergence with or without diversity; the new result identifies when diversity improves a single allocation batch. It motivates the design of EGGS rather than providing a guarantee for the learned algorithm. We also reorganized the theoretical exposition and clarified the presentation of the method and experiments. We also corrected several implementation descriptions in Appendix B to match the code used for all reported results (e.g., actor activations, simulator version, and replay handling); no reported result changes.

[W1, Q1(a)] Contribution of Gaussian MI

Gaussian MI is designed to reduce redundancy among goals selected for the same batch while preserving under-coverage utility. In the matched AntMaze Large diagnostic, EGGS retained nearly the same mean utility percentile as pointwise Top-K (99.09 vs. 99.66) while reducing trajectory overlap from 0.231 to 0.134 (Table 2; original).

On AntMaze Large (5M interactions, three seeds), the tested MI settings yielded mean coverage AUC between 37.7% and 39.4%, around the no-MI mean of 38.6% (Table 9; original). In the controlled 2-D mazes, the mean MI gain tended to be larger where high-utility candidates formed more spatial clusters (Table 11; new). On Square-Large, all four tested positive distinguishability weights improved mean coverage AUC over pointwise selection by 1.00–2.79 percentage points across ten paired seeds (Table 12; new).

Regarding the share of MI in the overall gains: without MI, EGGS already exceeds the strongest baseline in all six controlled 2-D environments (Table 10 vs. Table 1). On Square-Large, its 8.2-point lead over ExDM (54.7% vs. 46.5%) comprises 5.5 points from under-coverage goal selection with Go-Explore collection (51.96% without MI) and a further 2.7 points from MI.

[W2] Downstream performance

The original submission evaluated frozen pretrained behaviors on zero-shot goal reaching and sequential HRL on AntMaze Medium and Large (Sec. 4.5, Table 3, Appendix C.5; original):

MethodZero-shot SR, Medium (%)Zero-shot SR, Large (%)HRL goals/episode, MediumHRL goals/episode, Large
EGGS83.553.02.202.42
TLDR83.748.12.412.39
METRA20.814.70.891.66

EGGS achieves downstream performance comparable to TLDR and substantially above METRA. Its primary contribution lies in exploration under matched interaction budgets: EGGS achieves the highest mean coverage AUC among the compared methods in all six controlled 2-D environments and four of six AntMaze and locomotion environments (Tables 1 and 7). These downstream evaluations further demonstrate that the learned goal-conditioned behaviors remain reusable for zero-shot goal reaching and hierarchical control.

AuthorsOpenReview ↗

Response to Reviewer MdmC part 2

[Q1(b)] Comparison with simpler joint selectors

FPS, a simple diversification strategy, separates goals further than EGGS but at a substantial utility cost: in the matched AntMaze Large diagnostic, it reduces trajectory overlap to 0.111 (EGGS: 0.134), while its mean utility percentile is 82.3 (EGGS: 99.1; Table 2; original). Under full training on Square-Large, EGGS attained higher coverage AUC than FPS (54.7% vs. 49.2%; higher in nine of ten paired seeds; Table 17; new).

An analytically related utility-aware pairwise alternative, which we did not separately evaluate, is

Jpair(Z)=1K∑i=1Ku(zi)−λPK2∑i,j=1Kexp⁡(−∥zi−zj∥24σI2).J_{\mathrm{pair}}(Z)=\frac{1}{K}\sum_{i=1}^{K}u(z_i)-\frac{\lambda_P}{K^2}\sum_{i,j=1}^{K}\exp\left(-\frac{\lVert z_i-z_j\rVert^2}{4\sigma_I^2}\right).

For fixed KK and σI\sigma_I, the pairwise kernel sum is a decreasing function of the Rényi-2 differential entropy of the Gaussian mixture 1K∑iN(zi,σI2I)\frac{1}{K}\sum_i\mathcal{N}(z_i,\sigma_I^2I). In contrast, Gaussian MI is equivalent, up to a constant, to the Shannon differential entropy of the same mixture for uniformly sampled goal indices. Both criteria favor goal distinguishability at scale σI\sigma_I. We use Gaussian MI because, in the well-separated-cell limit, it recovers the categorical entropy H(b/K)H(b/K) of the selected goals' cell assignments (Appendix A.2; original).

[Q2] Sensitivity to the Gaussian noise scale

On AntMaze Large, σI∈{15,30,60}\sigma_I\in\lbrace 15,30,60\rbrace produced coverage AUC between 37.7% and 39.4% (Table 9; original). On Square-Large, σI∈{10,20,40}\sigma_I\in\lbrace 10,20,40\rbrace produced AUC between 53.3% and 54.7%, all above the no-MI mean of 52.0% (Table 13; new). Performance did not depend strongly on a particular value of σI\sigma_I within these tested fourfold ranges.

[Q3] Computational overhead

On five frozen AntMaze Large checkpoints (RTX 3090; Appendix C.4; new), selector time averaged 0.07 ms for Top-K, 1.1 ms for FPS, and 236 ms for EGGS, mostly from Gaussian MI evaluation (177 ms). Because selection runs once per collection cycle, this adds approximately 0.24 s to a 7.19 s collection-and-learning cycle (3.4%) under serial collection; relative overhead may increase with faster or parallel collection.

[Q4] Effect and choice of the learned representation

We tested identity coordinates and ALLO's [2] Laplacian representation in place of TLDR's [1] default representation on Square-Large. Only representation outputs were rescaled; all hyperparameters, including σI=20\sigma_I=20, remained unchanged (ten paired seeds; Table 14; new).

RepresentationCoverage AUC (%)
TLDR (default)54.7
Identity coordinates53.3
ALLO (Laplacian)50.4
Strongest baseline: ExDM [3]46.5

Both alternative representations exceeded the strongest baseline in this environment.

We chose TLDR's representation because its Euclidean embedding ϕ(s)\phi(s) reflects temporal distance, allowing us to define visitation density, Gaussian-mixture MI, and Go progress within a shared latent space. The ALLO result shows that another transition-aware Euclidean representation can also be used.

References

[1] Junik Bae, Kwanyoung Park, and Youngwoon Lee. TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations. In Conference on Robot Learning, 2024.

[2] Diego Gomez, Michael Bowling, and Marlos C. Machado. Proper Laplacian Representation Learning. In International Conference on Learning Representations, 2024.

[3] Chengyang Ying, Huayu Chen, Xinning Zhou, Zhongkai Hao, Hang Su, and Jun Zhu. Exploratory Diffusion Model for Unsupervised Reinforcement Learning. In The Fourteenth International Conference on Learning Representations, 2026.

Thank you again for your thoughtful questions. They helped us improve the paper. If our responses have addressed your concerns, we would appreciate it if you would consider updating your score. We are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.