- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies unlabeled offline-to-online RL where an agent initially has access to trajectories without reward labels and then must learn the reward function in an online phase. They do this with an algorithm that separates goal-reaching from online exploration by pretraining a goal-conditioned policy and adaptively selecting goals. During the offline pretraining phase, they learn a goal-conditioned policy as well as an occupancy estimator that predicts the visitation of a state when the policy is conditioned on a goal. They use this in online exploration to avoid selecting goals from a batch that have overlapping trajectories while optimizing for their coverage of potentially rewarding states, which are optimistic with added RND bonuses. They evaluate the method on OGBench navigation and manipulation tasks with unknown reward goals.
Critical strengths
- Separating the offline goal-conditioned policy learning from online exploration is well principled to avoid frequent updates under changing exploration objectives. And the experiments show that the pretrained goal-reaching abilities can be used effectively for online reward discovery.
- Using a learned occupancy estimator to maximize the combined coverage of multiple goal-conditioned trajectories is an interesting approach to reducing redundant exploration that also makes sense.
Critical weaknesses
- Although the proposed method outperforms prior online RL-based exploration methods, Figure 3 shows minor improvements over SingleGoal which already avoids online RL updates and Figure 4 shows that increasing the goal-set size does not consistently improve performance. Considering this, the experiments do not establish when or how much batching improves exploration beyond the adaptive single-goal selection. This is important since the joint batch optimization appears to be the main algorithmic contribution.
- The method also relies on sufficient offline data coverage, having a competent pretrained goal-conditioned policy, and an oracle goal representation. In the navigation tasks, exploration involves discovering an unknown rewarding location in an environment where the goal-reaching policy has already been pretrained. If the rewarding location were already known and reachable by the pretrained policy, the exploration stage would not be necessary. Additionally, the experiments only evaluate on one task per environment, leaving it unclear how well the approach generalizes to different tasks requiring behaviors that are not covered in the offline data.
Questions for authors
- How significant are the improvements of BGE over SingleGoal and when does batching provide a clear advantage?
- How accurately does the occupancy estimator predict the actual trajectory coverage and how sensitive is BGE to estimation errors?
- Does BGE generalize to different rewards and datasets with more limited offline coverage?
- Is there any evidence that existing methods actually unlearn previous exploration behaviors, rather than simply requiring additional updates to adapt to changing objectives?
2 comments
- BGE On Main Dataset |Env Step|Remove 99% of Data within Radius 40 of Goal|Remove 90% of Data within Radius 40 of Goal|Remove 99% of Data within Radius 10 of Goal|Remove 90% of Data within Radius 10 of Goal| |-|-|-|-|-| |200k|0.0 ± 0.0|12.5 ± 3.8|46.7 ± 5.1|54.7 ± 6.4|
For both when we run our algorithm on the imbalanced dataset after pre-training on the imbalance dataset and when we run our algorithm on the main dataset with varying qualities of goal-conditioned agent, our agents achieve similar levels of success as the pre-trained goal-conditioned agent. This suggests that exploration is not the bottleneck to increasing accuracy, and that our method is robust to different qualities of pre-trained goal-conditioned agents. However, because we are not doing any RL finetuning, it is limited by the quality of the goal-conditioned agent, meaning it can’t surpass the capabilities of the pre-trained goal-conditioned agent. We agree that this is a limitation and have discussed this in Appendix G. We have expanded upon this discussion and added the results to Appendix H question 1.
Regarding “if the rewarding location were known and reachable, exploration would be unnecessary,” the pre-trained agent knows how to reach states, but not which state has high reward. In our setting, the agent gets no reward information offline, so the agent must interact with the environment to find the high-reward states. Batched goals exploration decides which goals to condition the agent on so that the reward is found in as few online samples as possible.
How sensitive is BGE to occupancy estimation errors? (Q2)
humanoidmaze-giant
| Noise | Success (env_step=192k) |
|---|---|
| 0 (ours) | 91.0±5.5 |
| 0.5 | 0.0±0.0 |
| 1 | 0.0±0.0 |
| 2 | 0.0±0.0 |
| 4 | 0.0±0.0 |
cube-double
| Noise | Success (env_step=200k) |
|---|---|
| 0 (ours) | 56.0±9.0 |
| 0.5 | 54.0±8.5 |
| 1 | 53.0±7.5 |
| 2 | 46.0±7.0 |
| 4 | 49.0±5.5 |
To test how sensitive BGE is to the accuracy of the occupancy estimator, we add noise to the outputs of the pre-trained occupancy network during goal selection on cube-double (task 2) and humanoidmaze-giant (task 1), each across 4 seeds. On humanoidmaze-giant, BGE is highly sensitive to the quality of the occupancy network; it struggles to succeed even when . On cube-double, the results are less sensitive to the quality of the occupancy network. This is because in cube-double, every task goal has each cube resting on a surface, either the table or another cube. No true goal has a cube in the air, so the states a rollout passes through while carrying a cube therefore never contain the reward. The only reward-relevant states a rollout reaches are the final resting positions of the cube. This minimizes the effect of the noise in the occupancy network, as the occupancy of the intermediate states matters less (because they do not hold the true goal if a cube is in the air). See full results and discussion in appendix H question 3.
Is there any evidence that existing methods unlearn previous exploration behaviors? (Q4)
Thank you for asking this! We’d like to clarify that by “unlearn,” we do mean what you stated regarding “requiring additional updates to adapt to changing objectives.” It requires additional updates to adapt to shifting objectives, but as we show in figure 2, this adaptation is limited and we cannot continuously increase the UTD ratio. More specifically, methods like explore [1] and SUPE [2] keep a single policy that creates a moving target. As visited regions lose their novelty bonus, the policy and critic must be updated to steer toward new novel regions. Each shift in the exploration objective costs gradient updates, so exploration is limited by how fast the agent can adapt. Raising the UTD ratio to adapt faster eventually stops helping (figure 2) and marks training less stable [3] [4]. In BGE, with a goal-conditioned agent frozen after pre-training, changing the exploration objective only requires changing which goals we condition on. We have clarified this in the introduction and section 4.
[1] Qiyang Li, Jason Zhang, Dibya Ghosh, Amy Zhang, and Sergey Levine. Accelerating exploration with unlabeled prior data. Advances in Neural Information Processing Systems, 36, 2024 [2] Max Wilcoxson, Qiyang Li, Kevin Frans, and Sergey Levine. Leveraging skills from unlabeled prior data for efficient online exploration. In International Conference on Machine Learning (ICML), 2025. [3] Evgenii Nikishin, Max Schwarzer, Pierluca D’Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828–16847. PMLR, 2022. [4] Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Clare Lyle, Razvan Pascanu, Will Dabney, and André Barreto. Deep reinforcement learning with plasticity injection. Advances in Neural Information Processing Systems, 36:37142–37159, 2023
Thank you for taking the time to review our paper and for providing constructive feedback! In this response, we address when and by how much batching helps beyond adaptive single-goal selection (W1, Q1) by running a goal-batch-size sweep on 5 cube-double tasks and find that batching helps most when there are independent regions of the space to explore. In cube-double, for example, a batch can target the exploration of both cubes at once. We also show that our method is beneficial even with limited offline coverage (W2, Q3) by running experiments on antmaze-giant that remove offline data near the goal. Finally, we show how sensitive BGE is to estimation errors in the occupancy network (Q2) by running BGE with noise added to the occupancy network, and clarify what we mean by unlearning previous exploration behaviors (Q4).
When and by how much does batching provide a clear advantage over SingleGoal? (W1, Q1)
We agree that figures 3 and 4 alone do not show when batching matters. How much batching matters depends on the domain. Figure 4 uses humanoidmaze-giant, where one well-placed goal already covers most of the state space due to the interconnectivity of the maze. When the agent starts in the bottom-left corner, there are several similarly long paths to the far upper-right goal, and together those paths cover most of the maze. Thus, increasing the number of goals adds little to no new coverage and introduces sampling noise, which is why the goal-set size has an inconsistent effect in figure 4.
Rather, batching helps most when the potential reward lies in several spatially disjoint regions. Cube-double is a better test of this because it has multiple spatially disjoint regions of potential reward. We ran a goal-batch-size sweep over 5 cube-double tasks (5 seeds each), and calculate the area under the success curve over 200k environment steps:
BGE matches or beats SingleGoal on every task. We have added these results and discussion to the results section of the paper, as well as explicitly stated that batching helps most when rewards lie in several disjoint spaces, and that in a highly connected maze, a single coverage-optimized goal is often enough. See figure 5, section 7 question 2, and appendix H question 2.
Does BGE generalize to datasets with limited offline coverage? (W2, Q3)
(See Appendix H) To test the sensitivity of BGE to the coverage of the offline dataset and ability to explore in regions with limited coverage from the offline dataset, we remove the coverage of the offline dataset near the goal. We run experiments removing 90% and 99% of the data within a radius of 10 units of the goal, roughly removing 3% of the total data. We then run experiments removing 90% and 99% of the data within 40 units of the goal, roughly removing 56% and 62% of the total data, respectively. We find that the success of the pre-trained goal-conditioned agent when conditioned on the true goal is highly sensitive to the amount of data we remove.
Then, we run Batched Goals Exploration on the imbalanced dataset to test how our algorithm works on data that does not have good coverage near the goal, keeping the quality of the goal-conditioned agent constant and (2) the original dataset, to test how our algorithm works as the pre-trained goal-conditioned agent varies.