- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes a method to estimate empowerment (maximum mutual information between a skill and the discounted future state) purely from unlabeled offline data, without environment interaction. The key idea: learn the discounted state-occupancy measure of a skill-conditioned policy via TD backups, train the policy to maximize effective empowerment from that occupancy measure, then read off the empowerment estimate. The pretrained estimator is then used as an auxiliary bonus for online exploration on sparse-reward, goal-conditioned tasks.
Critical strengths
- The insight that empowerment estimators all require online rollouts, and that naive offline adaptation fails, is clearly articulated and demonstrated.
- Routing everything through the Bellman recursion of the occupancy measure is elegant and carefully explained.
Critical weaknesses
- The main question, "does the estimator recover true empowerment?", is answered only qualitatively (Figs. 1–2) on environments whose structure is argued to dictate ground truth. There is no numerical comparison to any ground truth (even in low-dimensional/toy settings where empowerment is computable, e.g. tabular mazes), no ablation on dataset size/coverage.
- Fig. 4 shows clear gains on only ~5 of 9 task/dataset combinations; on antmaze-medium-explore, antsoccer-stitch, and pointmaze-teleport-navigate the bonus is roughly neutral, and the paper doesn't analyze why.
- The four baselines are all hand-adapted versions of existing online estimators, and the paper itself argues they are expected to fail. A missing comparison is against a more reasonable proxy: e.g., successor-representation-based exploration value, disagreement/ensemble-based state value, or simple count/density baselines (which the paper mentions as related work but never evaluates).
Questions for authors
- The estimator operates on raw states (or proprioceptive state). Do you expect it to work from pixels, and if so, what would need to change?
- You motivate the work broadly ("robotics and coding"), but all experiments are goal-conditioned locomotion/manipulation. Do you expect the estimator to transfer to non-goal-conditioned settings, and is the downstream bonus applicable there?
3 comments
Official Author Response to Reviewer rWj6 (2/2)
“The estimator operates on raw states (or proprioceptive state). Do you expect it to work from pixels, and if so, what would need to change?”
While our method is general, it is not clear how difficult this would be to scale to pixel-based environments. In general, prior work has shown that scaling mutual information based estimators (not just empowerment) to pixel-based methods is very challenging (Poole et al., 2019; Song & Ermon, 2020). We expect that extending this to video likely requires learning some lower-dimensional encoding of the image, but leave this to future work, given that scaling mutual information estimation to video is in itself an open research problem.
“Do you expect the estimator to transfer to non-goal-conditioned settings, and is the downstream bonus applicable there?”
Yes, the central contribution of the work, the offline empowerment estimator is very general and not tied to goal-conditioned exploration. Prior work has shown applications of empowerment such as assisting humans (Du et al., 2020; Myers et al., 2024), multi-agent behaviour (Franzmeyer et al., 2022; Guckelsberger et al., 2016) and representation learning (Choi et al., 2021; Bharadhwaj et al., 2022) to name a few ; We chose the goal-conditioned setting as this setting is an especially challenging exploration problem and thus is a useful test bed for exploration methods (Park et al., 2025). While we did not directly test applications of the bonus to other settings, we believe the insights about exploration are very general, and our evaluation setup is in line with prior works (Li et al., 2023; Wilcoxson et al., 2025).
We once again thank the reviewer for their feedback. We plan to add these clarifications to the camera-ready version to improve the work.
References:
- Du et al. “AvE: Assistance via Empowerment.” NeurIPS 2020.
- Myers et al. “Learning to Assist Humans without Inferring Rewards.” NeurIPS 2024.
- Franzmeyer et al. “Learning Altruistic Behaviours in RL without External Rewards.” ICLR 2022.
- Guckelsberger et al. “Intrinsically Motivated General Companion NPCs via Coupled Empowerment Maximisation.” IEEE CIG 2016.
- Bharadhwaj et al. “Information Prioritization through Empowerment in Visual Model-Based RL.” ICLR 2022.
- Choi et al. “Variational Empowerment as Representation Learning for Goal-Based RL.” ICML 2021.
- Klyubin et al. “Empowerment: A Universal Agent-Centric Measure of Control.” IEEE CEC 2005.
- Mohamed & Jimenez Rezende. “Variational Information Maximisation for Intrinsically Motivated RL.” NeurIPS 2015.
- Gregor et al. “Variational Intrinsic Control.” ICLR 2017.
- Poole et al. “On Variational Bounds of Mutual Information.” ICML 2019.
- Song & Ermon. “Understanding the Limitations of Variational Mutual Information Estimators.” ICLR 2020.
- Park et al. “OGBench: Benchmarking Offline Goal-Conditioned RL.” ICLR 2025.
- Li et al. “Accelerating Exploration with Unlabeled Prior Data.” NeurIPS 2023.
- Wilcoxson et al. “Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration.” ICML 2025.
Thank you for providing more detailed explanation. Most of my concerns are addressed now, so I raised my score from 2 to 3.
Official Author Response to Reviewer rWj6 (1/2)
Thank you to the reviewer for their constructive feedback and their interest in our novel offline empowerment estimator. We have also uploaded a revised version of the paper with all changes in blue text for the reviewer's reference. Below we clarify the reviewer’s main concerns. Additionally, we kindly request the reviewer to revise their score if their concerns have been sufficiently addressed, otherwise we welcome any further questions or comments.
“The main question, "does the estimator recover true empowerment?", is answered only qualitatively”
We have added a check for this in Section 6.3 (Table 1, Fig. 4; details in Appendix A.7). Specifically, we discretize the antmaze-medium-navigate, pointmaze-teleport-navigate, and antsoccer-arena-navigate datasets into grids with five actions (stay and four moves), with transition probabilities counted from the data. In these tabular MDPs, the occupancy of any policy can be computed exactly, so we compute ground-truth empowerment by searching over skill policies. We train our estimator and baselines on this, and find that our estimator does an overall qualitatively better job of recovering the region-level structure (see Table 1, Fig. 4) compared to the baselines. Note that this is despite the fact that our baselines should do especially well on discrete environments where distribution-shift and diversity collapse (i.e. DADS) is less of an issue; a central contribution of our method is that it works for arbtirary/continuous environments. We also find that the skill-conditioned policy recovered from our method is highly empowered. Computing its empowerment in the tabular MDP yields an average rank correlation of 82% with ground truth (“Our skills” in Table 1), which is higher than the baselines.
“Fig. 4 shows clear gains on only ~5 of 9 task/dataset combinations...and the paper doesn't analyze why.”
We would highlight that on the environments where there is no gain from the empowerment bonus, none of the exploration methods tested give any gain over baseline CRL. This suggests that in those particular environments, exploration is not as much of a bottleneck, and perhaps the performance is dominated by the quality of the offline data, given that most of those environments are stitch and explore variants. The most concrete example of this antsoccer, where we find that exploration primarily helps in getting the ant to the ball more quickly. It turns out however that the main bottleneck is actually fine-grained, temporally-correlated control of the ball for which exploration is not as helpful.
“A missing comparison is against a more reasonable proxy: e.g., successor-representation-based exploration value, disagreement/ensemble-based state value, or simple count/density baselines”
We note that empowerment is in itself a long-studied, fundamental quantity in reinforcement learning (Klyubin et al., 2005; Mohamed & Jimenez Rezende, 2015; Gregor et al., 2017), and not inherently tied to exploration. Consequently, it does not make sense to use count/density or successor functions as empowerment baselines, as these are different quantities. The central contribution of the work is an offline empowerment estimator; as an application of it, we show it is useful for exploration value, but our contributions are not tied to exploration.
We have however added a comparison to the state occupancy in Section 6.3 (Fig. 4; details in Appendix A.7). Specifically, we discretize antmaze-medium-navigate and pointmaze-teleport-navigate into grids with five actions (stay and four moves), with transition probabilities counted from the data. Fig. 4 shows that while the state occupancy (N(s)) of the offline data is (unsurprisingly) a reasonable proxy, it is sensitive to the offline data distribution in two adversarial ways:
We also added a count-based exploration baseline to our online experiments, Distributional Random Network Distillation (DRND) (Yang et al., 2024), which estimates a pseudo-count bonus approximating 1/N(s) (Section 5.2; details in Appendix A.3.2). We evaluate both CRL + DRND and CRL + DRND + Empowerment on all tasks (Section 6.4, Fig. 5). DRND performs comparably to RND, and adding our empowerment bonus on top of DRND further improves performance on multiple tasks.
These experiments all demonstrate that empowerment is complementary to density/count-based novelty, rather than being a substitute.
(Continued in the next comment.)