acceptodds
Under review as a conference paper at ICLR 2027

Coverage Maximizing Exploration

Abstract

We propose batched goals exploration (BGE), an adaptive exploration algorithm that simultaneously targets multiple novel regions in the state space without any online RL updates. This is in contrast to standard RL exploration algorithms that adapt to new reward information by repeatedly updating the exploration policy online with RL objectives, which can be expensive and unstable. BGE avoids online RL updates by decoupling behavior optimization from high-level online exploration. In particular, BGE first pre-trains a goal-conditioned agent on a prior offline dataset, then online optimizes a batch of goals such that the expected state occupancy achieved by rolling out the pre-trained goal-conditioned agent conditioned on these goals maximizes the coverage over potential high-reward states. Across challenging long-horizon, sparse-reward tasks across locomotion and manipulation, BGE is able to leverage unlabeled prior data to achieve better online sample-efficiency compared to prior methods, while only using a fraction of the computational cost online.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, exploration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HbveOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies unlabeled offline-to-online RL where an agent initially has access to trajectories without reward labels and then must learn the reward function in an online phase. They do this with an algorithm that separates goal-reaching from online exploration by pretraining a goal-conditioned policy and adaptively selecting goals. During the offline pretraining phase, they learn a goal-conditioned policy as well as an occupancy estimator that predicts the visitation of a state when the policy is conditioned on a goal. They use this in online exploration to avoid selecting goals from a batch that have overlapping trajectories while optimizing for their coverage of potentially rewarding states, which are optimistic with added RND bonuses. They evaluate the method on OGBench navigation and manipulation tasks with unknown reward goals.

Critical strengths
  • Separating the offline goal-conditioned policy learning from online exploration is well principled to avoid frequent updates under changing exploration objectives. And the experiments show that the pretrained goal-reaching abilities can be used effectively for online reward discovery.
  • Using a learned occupancy estimator to maximize the combined coverage of multiple goal-conditioned trajectories is an interesting approach to reducing redundant exploration that also makes sense.
Critical weaknesses
  • Although the proposed method outperforms prior online RL-based exploration methods, Figure 3 shows minor improvements over SingleGoal which already avoids online RL updates and Figure 4 shows that increasing the goal-set size does not consistently improve performance. Considering this, the experiments do not establish when or how much batching improves exploration beyond the adaptive single-goal selection. This is important since the joint batch optimization appears to be the main algorithmic contribution.
  • The method also relies on sufficient offline data coverage, having a competent pretrained goal-conditioned policy, and an oracle goal representation. In the navigation tasks, exploration involves discovering an unknown rewarding location in an environment where the goal-reaching policy has already been pretrained. If the rewarding location were already known and reachable by the pretrained policy, the exploration stage would not be necessary. Additionally, the experiments only evaluate on one task per environment, leaving it unclear how well the approach generalizes to different tasks requiring behaviors that are not covered in the offline data.
Questions for authors
  • How significant are the improvements of BGE over SingleGoal and when does batching provide a clear advantage?
  • How accurately does the occupancy estimator predict the actual trajectory coverage and how sensitive is BGE to estimation errors?
  • Does BGE generalize to different rewards and datasets with more limited offline coverage?
  • Is there any evidence that existing methods actually unlearn previous exploration behaviors, rather than simply requiring additional updates to adapt to changing objectives?
2 comments

Thank you for taking the time to review our paper and for providing constructive feedback! In this response, we address when and by how much batching helps beyond adaptive single-goal selection (W1, Q1) by running a goal-batch-size sweep on 5 cube-double tasks and find that batching helps most when there are independent regions of the space to explore. In cube-double, for example, a batch can target the exploration of both cubes at once. We also show that our method is beneficial even with limited offline coverage (W2, Q3) by running experiments on antmaze-giant that remove offline data near the goal. Finally, we show how sensitive BGE is to estimation errors in the occupancy network (Q2) by running BGE with noise added to the occupancy network, and clarify what we mean by unlearning previous exploration behaviors (Q4).

When and by how much does batching provide a clear advantage over SingleGoal? (W1, Q1)

We agree that figures 3 and 4 alone do not show when batching matters. How much batching matters depends on the domain. Figure 4 uses humanoidmaze-giant, where one well-placed goal already covers most of the state space due to the interconnectivity of the maze. When the agent starts in the bottom-left corner, there are several similarly long paths to the far upper-right goal, and together those paths cover most of the maze. Thus, increasing the number of goals adds little to no new coverage and introduces sampling noise, which is why the goal-set size has an inconsistent effect in figure 4.

Rather, batching helps most when the potential reward lies in several spatially disjoint regions. Cube-double is a better test of this because it has multiple spatially disjoint regions of potential reward. We ran a goal-batch-size sweep over 5 cube-double tasks (5 seeds each), and calculate the area under the success curve over 200k environment steps:

cube-doubleSingleGoal: AUCBGE (ours, 10 goals): AUC
Task 138.3±4.242.4±5.4
Task 215.8±8.932.4±6.0
Task 310.6±9.422.7±3.5
Task 42.5±1.76.0±1.3
Task 527.5±14.035.0±3.7

BGE matches or beats SingleGoal on every task. We have added these results and discussion to the results section of the paper, as well as explicitly stated that batching helps most when rewards lie in several disjoint spaces, and that in a highly connected maze, a single coverage-optimized goal is often enough. See figure 5, section 7 question 2, and appendix H question 2.

Does BGE generalize to datasets with limited offline coverage? (W2, Q3)

(See Appendix H) To test the sensitivity of BGE to the coverage of the offline dataset and ability to explore in regions with limited coverage from the offline dataset, we remove the coverage of the offline dataset near the goal. We run experiments removing 90% and 99% of the data within a radius of 10 units of the goal, roughly removing 3% of the total data. We then run experiments removing 90% and 99% of the data within 40 units of the goal, roughly removing 56% and 62% of the total data, respectively. We find that the success of the pre-trained goal-conditioned agent when conditioned on the true goal is highly sensitive to the amount of data we remove.

GroupSuccess
Remove 99% of Data within Radius 40 of Goal0.0 ± 0.0
Remove 90% of Data within Radius 40 of Goal13.1 ± 5.0
Remove 99% of Data within Radius 10 of Goal47.4 ± 2.9
Remove 90% of Data within Radius 10 of Goal54.4 ± 3.2

Then, we run Batched Goals Exploration on the imbalanced dataset to test how our algorithm works on data that does not have good coverage near the goal, keeping the quality of the goal-conditioned agent constant and (2) the original dataset, to test how our algorithm works as the pre-trained goal-conditioned agent varies.

  1. BGE On Imbalanced Dataset |Env Step|Remove 99% of Data within Radius 40 of Goal|Remove 90% of Data within Radius 40 of Goal|Remove 99% of Data within Radius 10 of Goal|Remove 90% of Data within Radius 10 of Goal| |-|-|-|-|-| |200k|0.0 ± 0.0|9.2 ± 1.1|42.5 ± 11.0|55.7 ± 7.2|
  1. BGE On Main Dataset |Env Step|Remove 99% of Data within Radius 40 of Goal|Remove 90% of Data within Radius 40 of Goal|Remove 99% of Data within Radius 10 of Goal|Remove 90% of Data within Radius 10 of Goal| |-|-|-|-|-| |200k|0.0 ± 0.0|12.5 ± 3.8|46.7 ± 5.1|54.7 ± 6.4|

For both when we run our algorithm on the imbalanced dataset after pre-training on the imbalance dataset and when we run our algorithm on the main dataset with varying qualities of goal-conditioned agent, our agents achieve similar levels of success as the pre-trained goal-conditioned agent. This suggests that exploration is not the bottleneck to increasing accuracy, and that our method is robust to different qualities of pre-trained goal-conditioned agents. However, because we are not doing any RL finetuning, it is limited by the quality of the goal-conditioned agent, meaning it can’t surpass the capabilities of the pre-trained goal-conditioned agent. We agree that this is a limitation and have discussed this in Appendix G. We have expanded upon this discussion and added the results to Appendix H question 1.

Regarding “if the rewarding location were known and reachable, exploration would be unnecessary,” the pre-trained agent knows how to reach states, but not which state has high reward. In our setting, the agent gets no reward information offline, so the agent must interact with the environment to find the high-reward states. Batched goals exploration decides which goals to condition the agent on so that the reward is found in as few online samples as possible.

How sensitive is BGE to occupancy estimation errors? (Q2)

humanoidmaze-giant

NoiseSuccess (env_step=192k)
0 (ours)91.0±5.5
0.50.0±0.0
10.0±0.0
20.0±0.0
40.0±0.0

cube-double

NoiseSuccess (env_step=200k)
0 (ours)56.0±9.0
0.554.0±8.5
153.0±7.5
246.0±7.0
449.0±5.5

To test how sensitive BGE is to the accuracy of the occupancy estimator, we add noise σ\sigma to the outputs of the pre-trained occupancy network during goal selection on cube-double (task 2) and humanoidmaze-giant (task 1), each across 4 seeds. On humanoidmaze-giant, BGE is highly sensitive to the quality of the occupancy network; it struggles to succeed even when σ=0.5\sigma=0.5. On cube-double, the results are less sensitive to the quality of the occupancy network. This is because in cube-double, every task goal has each cube resting on a surface, either the table or another cube. No true goal has a cube in the air, so the states a rollout passes through while carrying a cube therefore never contain the reward. The only reward-relevant states a rollout reaches are the final resting positions of the cube. This minimizes the effect of the noise in the occupancy network, as the occupancy of the intermediate states matters less (because they do not hold the true goal if a cube is in the air). See full results and discussion in appendix H question 3.

Is there any evidence that existing methods unlearn previous exploration behaviors? (Q4)

Thank you for asking this! We’d like to clarify that by “unlearn,” we do mean what you stated regarding “requiring additional updates to adapt to changing objectives.” It requires additional updates to adapt to shifting objectives, but as we show in figure 2, this adaptation is limited and we cannot continuously increase the UTD ratio. More specifically, methods like explore [1] and SUPE [2] keep a single policy that creates a moving target. As visited regions lose their novelty bonus, the policy and critic must be updated to steer toward new novel regions. Each shift in the exploration objective costs gradient updates, so exploration is limited by how fast the agent can adapt. Raising the UTD ratio to adapt faster eventually stops helping (figure 2) and marks training less stable [3] [4]. In BGE, with a goal-conditioned agent frozen after pre-training, changing the exploration objective only requires changing which goals we condition on. We have clarified this in the introduction and section 4.

[1] Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024 [2] Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine. Leveraging skills from unlabeled prior data for efficient online exploration. In International Conference on Machine Learning (ICML), 2025. [3] Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828–16847. PMLR, 2022. [4] Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36:37142–37159, 2023

Reviewer_VuCtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Batched Goals Exploration (BGE) for unlabeled offline-to-online reinforcement learning. BGE pretrains a goal-conditioned policy and an occupancy estimator using reward-free offline data, then selects batches of goals to encourage coverage of potentially rewarding regions. The policy remains frozen online, while reward and novelty models are updated from new interactions. Experiments on four OGBench configurations compare BGE with ExPLORe, SUPE, and two goal-selection baselines, reporting improvements in online sample efficiency and reduced online computation.

Critical strengths
  1. The paper addresses a meaningful problem: using reward-free prior experience to reduce the interaction and computation required for sparse-reward exploration. Separating offline behavior learning from online goal selection offers a useful approach to this problem.

  2. The comparison includes relevant prior methods and baselines sharing the same pretrained goal-conditioned policy. Reporting both learning curves and first-success statistics helps distinguish task-solving performance from initial reward discovery.

Critical weaknesses
  1. Limited experimental coverage. Evaluation covers only the default task in four environment configurations, while sensitivity and efficiency analyses focus on a single task. This provides insufficient evidence of robustness across different goals and offline data coverage conditions.

  2. Insufficient validation of the core contribution. The ablations do not adequately isolate the benefits of joint coverage optimization. A comparison with independently selecting the same number of goals, along with measurements of actual state coverage and trajectory overlap, is missing.

  3. Incomplete reproducibility details. The occupancy estimator’s discount factor, goal-search parameters, and some training settings are unspecified. Inconsistent reward descriptions further hinder reproducibility.

Questions for authors
  1. Experimental coverage: Could you supplement the evaluation with additional goal configurations and datasets with reduced state-space coverage? Please also extend the goal-set sensitivity and runtime analyses to a manipulation task, to assess whether the findings generalize beyond humanoidmaze-giant.

  2. Validation of joint coverage optimization: Could you compare joint coverage optimization against independently selecting the same number of goals, using identical pretrained policies, interaction budgets, and model-update schedules? Please report actual state coverage and trajectory overlap to verify whether reduced redundant exploration explains the performance gains.

  3. Reproducibility: Could you specify the occupancy estimator’s discount factor, goal-search parameters �0��0�, pretraining duration, and reward/RND update schedules? Please also reconcile the reward descriptions in Appendix B.3 and Figure 5 and confirm that all methods used the same reward convention.

2 comments

Thank you for your thoughtful review and constructive feedback! In this response, we show the advantages of joint coverage optimization with new cube-double results comparing joint coverage optimization to independently selecting goals (W2, Q2). Next, we extend our results to increase experimental coverage by showing how BGE behaves when the offline dataset has less coverage by removing the offline data near the goal on antmaze-giant, as well as running it across 5 tasks on cube-double, a manipulation task (W1, Q1). We also increase experimental coverage by extending efficiency analysis to cube-double, a manipulation task (W1) and specify reproducibility details (Q3).

How much does joint-optimization help in comparison to selecting the same number of goals independently? (W2, Q2)

Thank you for this suggestion! Selecting the same number of goals independently, scoring each goal by its own coverage and taking the top-k, would behave very similarly to our SingleGoal ablation. When goals are scored in isolation, the highest-scoring goals concentrate in the same high-UCB region and induce nearly the same occupancy, so the k rollouts largely revisit the same states. In effect, this reduces to rolling out the single best goal k times, which is what SingleGoal does. We acknowledge that one could inject noise into the goal sampler to diversify independently selected goals. However, the noise is not informed by what the other goals already cover, so resulting goals would still mostly target the same regions spread out arbitrarily rather than toward complimentary high-UCB regions.

The goal-set-size sweep on cube-double (see response to Hbve) isolates this effect. Across 5 tasks, increasing the number of goals from 1 to 10 increases the area under the success curve across 200k environment steps from 0.189 to 0.277 (averaged across 5 tasks). See figure 5, section 7 question 2, and appendix H question 2 for results and more discussion.

How does reduced state-space coverage affect BGE? (W1, Q1)

To test how well BGE works when the coverage of the offline dataset near the goal is limited or nonexistent, we remove the coverage of the offline dataset near the goal. We run experiments removing 90% and 99% of the data within a radius of 10 units of the goal, roughly removing 3% of the total data. We then run experiments removing 90% and 99% of the data within 40 units of the goal, roughly removing 56% and 62% of the total data, respectively.

GroupSuccess
Remove 99% of Data within Radius 40 of Goal0.0 ± 0.0
Remove 90% of Data within Radius 40 of Goal13.1 ± 5.0
Remove 99% of Data within Radius 10 of Goal47.4 ± 2.9
Remove 90% of Data within Radius 10 of Goal54.4 ± 3.2

Then, we run Batched Goals Exploration on the imbalanced dataset to test how our algorithm works on data that does not have good coverage near the goal, keeping the quality of the goal-conditioned agent constant and (2) the original dataset, to test how our algorithm works as the pre-trained goal-conditioned agent varies.

  1. BGE On Imbalanced Dataset |Env Step|Remove 99% of Data within Radius 40 of Goal|Remove 90% of Data within Radius 40 of Goal|Remove 99% of Data within Radius 10 of Goal|Remove 90% of Data within Radius 10 of Goal| |-|-|-|-|-| |200k|0.0 ± 0.0|9.2 ± 1.1|42.5 ± 11.0|55.7 ± 7.2|

  2. BGE On Main Dataset |Env Step|Remove 99% of Data within Radius 40 of Goal|Remove 90% of Data within Radius 40 of Goal|Remove 99% of Data within Radius 10 of Goal|Remove 90% of Data within Radius 10 of Goal| |-|-|-|-|-| |200k|0.0 ± 0.0|12.5 ± 3.8|46.7 ± 5.1|54.7 ± 6.4|

For both when we run our algorithm on the imbalanced dataset after pre-training on the imbalanced dataset and when we run our algorithm on the main dataset with varying qualities of goal-conditioned agent, our agents achieve similar levels of success as the pre-trained goal-conditioned agent. This suggests that exploration is not the bottleneck to increasing accuracy, and that our method is robust to different qualities of pre-trained goal-conditioned agents. However, because we are not doing any RL finetuning, it is limited by the quality of the goal-conditioned agent, meaning it can’t surpass the capabilities of the pre-trained goal-conditioned agent. We have added these results and this discussion to figure 7 and appendix H question 1 of our paper.

Efficiency analyses for manipulation task (W1)

Thank you for the suggestion! We have added the runtime and deployment analysis for cube-double to the paper (see table 2). On cube-double, BGE’s online training takes about 2 hours, SUPE takes about 3 hours, and ExPLORe takes 2 hours. The speedup is a lot smaller than on humanoidmaze-giant, but BGE remains faster than SUPE. While the runtime is similar to ExPLORe’s, ExPLORe does not succeed on any seed (0/5) within the 200k environment steps, while BGE reaches its first success on all 5 seeds (figure 4).

Reproducibility details (W3, Q3)

Thank you for pointing these out. We have added an updated table of hyperparameters to the appendix, including the occupancy estimator discount factor, pre-training steps, goal proposal size, number of test states, goal reselection frequency, and reward model and RND update schedule. See Appendix E.

Reward convention (Q3)

Thank you for pointing this out! To clarify, appendix B.3 describes cube-double rewards of -2, -1, and 0 (zero, one, or both cubes placed correctly, respectively). Thus, consistent with both appendix B.3 and figure 5, the environment begins with -2 reward and gets +1 for every cube correctly placed. All methods, including BGE, ExPLORe, SUPE, and the ablations use the same environment wrapper with the -2/-1/0 convention. We have added to the figure 6 (previously figure 5) caption to make this clear.

Thank you so much for your review! Please let us know if you have any additional questions regarding our paper.

Reviewer_WZHROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a novel approach to exploration in offline-to-online GCRL settings. It proposes a relatively compute-efficient method for online exploration based on a online sub-goal selection mechanism.

Critical strengths

The paper is easy to read. The bandit-style goal selection mechanism for exploration is appealing and could complement existing state-of-the-art exploration algorithms.

Critical weaknesses

The paper would be stronger with comparisons to other state-of-the-art exploration methods for GCRL. Some of the design choices seem poorly motivated without corresponding ablation studies.

Questions for authors

Q1.Can you substantiate the claims in paragraph 2 of Intro with appropriate citations? eg., "and when it visits one part of the state space and reduces its novelty, a standard exploration algorithm needs to unlearn how to go to this region and relearn how to visit a different region that is still underexplored...": I believe this is not always the case. For instance, "Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL" (Bastankhah et. al., 2025) shows that CRL learns to explore without such "un-learning".

Q2. I found Eq. 8 confusing. Specifically, doesn't 1[V (Ψ(s), g+) > V (g+, g+)] almost always give 0? Also, do you train 2 value functions; one with psi representations and another in the raw state space?

Q3. Can you also add comparisons to Single-goal contrastive RL and "Exploratory Diffusion Model for Unsupervised Reinforcement Learning" ? I believe these are strong baselines which are also based on estimating the occupancy measure.

Q4. Can you add more ablations that shed light on the design choices behind adaptive σ and test state filtering?

2 comments

Thank you for taking the time to review our paper and we appreciate your positive review! In this response, strengthen our paper with comparisons to other SOTA methods (W1, Q3), we motivate our design choices for adaptive sigma and test-state filtering help through ablations on the cube-double domain (Q4), and clarify what we mean by “unlearning and relearning” (Q1) and eq 8 and the value functions we train (Q2).

How does BGE compare to SGCRL and ExDM? (Q3)

Thank you for the suggestion! Both methods target a different problem setting than ours. We explain the differences and adapted ExDM to fit our problem setting to compare to as a baseline.

SGCRL operates under the setting where the agent is given the true goal observation. This is different from our setting, in which our agent is only given unlabeled, reward-free offline data. During the online exploration, our agent has no goal observation and must find high reward states through exploration. Without the goal observation, SGCRL has nothing to condition on, so we cannot apply the method directly, as that would be giving SGCRL information our method does not see.

The ExDM baseline is designed for online reward-free pre-training, where it learns exploratory behavior by interacting with the environment with no prior dataset. In contrast, we operate under the unlabeled offline-to-online setting, where the agent begins with a fixed unlabeled (reward-free) dataset, then must find a high reward policy online. To make the two comparable and to compare the “occupancy-based measure” part of ExDM to our method, we adapted ExDM to fit our problem setting. To do this, we keep ExDM’s primary contribution, its score-based intrinsic reward. We placed this reward in the ExPLORe pipeline in place of its UCB novelty bonus, and we call this baseline ExPLORe w/ ExDM bonus. More details of the implementation in Appendix C.4. We have added these results to the main results in figure 3.

At 200k environment steps:

EnvironmentBGE (ours)ExPLORe w/ ExDM bonus
antmaze-large88.0±6.0100.0±0.0
antmaze-giant68.0±22.564.0±13.0
humanoidmaze-giant91.0±5.50.0±0.0
cube-double53.0±9.00.0±0.0

How sensitive is our method with respect to the filtering threshold and adaptive sigma? (Q4)

Cube-double domain at 200k environment steps:

Adaptive (ours)=0=0.2=0.4=0.6=1.0
56.0±9.046.0±14.059.0±10.023.0±22.539.0±13.061.0±21.0

σ\sigma represents the probability threshold of the occupancy network output that allows us to consider a state as occupied/“covered,” given the agent is conditioned on a given goal. If, according to our occupancy networks, a state has a higher probability of being occupied than sigma, we consider it “covered.” In practice, we set sigma to be the probability the agent will reach the goal if it is conditioned on the goal itself (section 5.3). That is, we approximate the states the goal-conditioned agent will cover when conditioned on g+ to be all of the states it is more likely to reach than g+ itself. This is because the states on the path to g+ should have a higher occupancy value than g+ itself, as our occupancy uses a discounted estimator where it assigns lower value to states that are further away from the goal. This adaptive sigma helps our method be more robust to the sensitivity of the occupancy networks, and allows for us to not have to finetune sigma for every new environment. A hyperparameter sweep over sigma on cube-double shows that performance varies with respect to sigma, and our adaptive sigma matches the performance of the best-performing sigma values. See appendix H question 4 and figure 10.

Do previous exploration methods really require “unlearning and relearning”? (Q1)

Thank you for suggesting we look at Bastankhah et al. (2026). We agree that our wording was too strong, and **we revised our introduction and section 4 to be more precise. **

Our claim applies to exploration without a known target. Online exploration methods in our setting, such as ExPLORe [1] and SUPE [2], keeps a single policy that is trained online to maximize a non-stationary exploration bonus (i.e., RND). As visited regions lose their novelty, the critic and policy must be updated to steer the agent toward regions that are still novel. “Unlearning” refers to these extra gradient updates needed to track a moving objective, not to the policy forgetting how to reach a region. These updates limit how quickly the exploration behavior can adapt. From figure 2, we see that we are unable to continuously raise the UTD ratio to increase sample efficiency, as high UTD ratios are known to make training less stable (Nikishin et al, 2022; 2023).

In contrast, Bastankhah et al. (2026) refers to SGCRL, which operates under the setting given the true goal observation. Thus, learning pushes visited, unsuccessful states away from the goal and then strengthens the successful path once the goal is found. The policy is not trained toward a novelty target that is later withdrawn, explaining why SGCRL does not require “unlearning and relearning” in the way we describe. Meanwhile, BGE operates under the setting that the agent is not given the true goal observation, requiring the agent to quickly explore by maximizing the coverage of visited states.

Does 1[V (Ψ(s), g+) > V (g+, g+)] almost always give 0? (Q2)

No, 1[V (Ψ(s), g+) > V (g+, g+)] does not almost always give 0. Concretely, V(s, g^+) reflects the (discounted) probability of visiting state ss at any point while the policy is conditioned on g+g^+. Thus, every state along the path to the goal contributes to the coverage, not just the goal itself. This means for a perfect goal-conditioned policy, the coverage of states other than the goal would not all go to zero. Instead, the occupancy value would be high on all the states along its trajectory from the initial state to the goal, with the states closest to the initial state being the highest and the states closest to the goal state being the lowest due to the discount factor. Then, with our adaptive sigma, every state along the trajectory would have a higher V(s,g+)V(s, g^+) value than the goal itself, V(g+,g+)V(g^+, g^+). This would favor goals that would cause the agent to visit many states with a high UCB reward.

How essential is test-state filtering? (Q4)

Thank you for bringing this up! We replace test-state filtering with using the augmented reward as described in equation 6 with β=1\beta=1. We find that our algorithm performs similarly, meaning we could likely replace test-state filtering with the original augmented reward. More results and discussion in appendix H question 5 and figure 11.

Do you train two value functions? (Q2)

We train a single occupancy estimator, Q(s,a,ϕ(s+),g+)Q(s, a, \phi(s^+), g^+), distilled into V(ϕ(s),g)V(\phi(s), g). No second value function is trained on the raw states.

Thank you again for your review. Please also let us know if you have any additional questions/concerns/suggestions regarding our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.