acceptodds
Under review as a conference paper at ICLR 2027

Measuring Empowerment from Offline Data for Intrinsically Motivated Exploration

Abstract

Large, unlabeled datasets of prior experience have become increasingly available for agentic tasks in domains such as robotics and coding. A common recipe for these tasks is to train a policy using the prior data, and then further improve it with online rollouts. But while much work has studied how to use both the offline and online data for policy extraction, far less work has been done to understand how the prior data can be used to efficiently explore during the online learning phase. Previous work has attempted to formalize this idea by defining the empowerment, a measure of how many future states an agent can realize; states with higher empowerment require more exploration as there are more futures to explore. However, existing methods for estimating the empowerment require online policy rollouts in the environment. We propose the first method for estimating the empowerment from unlabeled prior data, without any environment interaction. By jointly learning an empowerment-maximizing policy together with its discounted state-occupancy, our estimator can be trained on unlabeled trajectories generated from any unknown policy. We demonstrate that our empowerment estimator qualitatively gives intuitive empowerment measures and show that the pretrained estimator can be used to improve exploration in the online goal-conditioned reinforcement learning setting.

Primary area:
reinforcement learning
Keywords:
Empowerment, Reinforcement Learning, Skill Discovery

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_rWj6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a method to estimate empowerment (maximum mutual information between a skill and the discounted future state) purely from unlabeled offline data, without environment interaction. The key idea: learn the discounted state-occupancy measure of a skill-conditioned policy via TD backups, train the policy to maximize effective empowerment from that occupancy measure, then read off the empowerment estimate. The pretrained estimator is then used as an auxiliary bonus for online exploration on sparse-reward, goal-conditioned tasks.

Critical strengths
  • The insight that empowerment estimators all require online rollouts, and that naive offline adaptation fails, is clearly articulated and demonstrated.
  • Routing everything through the Bellman recursion of the occupancy measure is elegant and carefully explained.
Critical weaknesses
  • The main question, "does the estimator recover true empowerment?", is answered only qualitatively (Figs. 1–2) on environments whose structure is argued to dictate ground truth. There is no numerical comparison to any ground truth (even in low-dimensional/toy settings where empowerment is computable, e.g. tabular mazes), no ablation on dataset size/coverage.
  • Fig. 4 shows clear gains on only ~5 of 9 task/dataset combinations; on antmaze-medium-explore, antsoccer-stitch, and pointmaze-teleport-navigate the bonus is roughly neutral, and the paper doesn't analyze why.
  • The four baselines are all hand-adapted versions of existing online estimators, and the paper itself argues they are expected to fail. A missing comparison is against a more reasonable proxy: e.g., successor-representation-based exploration value, disagreement/ensemble-based state value, or simple count/density baselines (which the paper mentions as related work but never evaluates).
Questions for authors
  • The estimator operates on raw states (or proprioceptive state). Do you expect it to work from pixels, and if so, what would need to change?
  • You motivate the work broadly ("robotics and coding"), but all experiments are goal-conditioned locomotion/manipulation. Do you expect the estimator to transfer to non-goal-conditioned settings, and is the downstream bonus applicable there?
3 comments
AuthorsOpenReview ↗

Official Author Response to Reviewer rWj6 (1/2)

Thank you to the reviewer for their constructive feedback and their interest in our novel offline empowerment estimator. We have also uploaded a revised version of the paper with all changes in blue text for the reviewer's reference. Below we clarify the reviewer’s main concerns. Additionally, we kindly request the reviewer to revise their score if their concerns have been sufficiently addressed, otherwise we welcome any further questions or comments.

“The main question, "does the estimator recover true empowerment?", is answered only qualitatively”

We have added a check for this in Section 6.3 (Table 1, Fig. 4; details in Appendix A.7). Specifically, we discretize the antmaze-medium-navigate, pointmaze-teleport-navigate, and antsoccer-arena-navigate datasets into grids with five actions (stay and four moves), with transition probabilities counted from the data. In these tabular MDPs, the occupancy of any policy can be computed exactly, so we compute ground-truth empowerment by searching over skill policies. We train our estimator and baselines on this, and find that our estimator does an overall qualitatively better job of recovering the region-level structure (see Table 1, Fig. 4) compared to the baselines. Note that this is despite the fact that our baselines should do especially well on discrete environments where distribution-shift and diversity collapse (i.e. DADS) is less of an issue; a central contribution of our method is that it works for arbtirary/continuous environments. We also find that the skill-conditioned policy recovered from our method is highly empowered. Computing its empowerment in the tabular MDP yields an average rank correlation of 82% with ground truth (“Our skills” in Table 1), which is higher than the baselines.

“Fig. 4 shows clear gains on only ~5 of 9 task/dataset combinations...and the paper doesn't analyze why.”

We would highlight that on the environments where there is no gain from the empowerment bonus, none of the exploration methods tested give any gain over baseline CRL. This suggests that in those particular environments, exploration is not as much of a bottleneck, and perhaps the performance is dominated by the quality of the offline data, given that most of those environments are stitch and explore variants. The most concrete example of this antsoccer, where we find that exploration primarily helps in getting the ant to the ball more quickly. It turns out however that the main bottleneck is actually fine-grained, temporally-correlated control of the ball for which exploration is not as helpful.

“A missing comparison is against a more reasonable proxy: e.g., successor-representation-based exploration value, disagreement/ensemble-based state value, or simple count/density baselines”

We note that empowerment is in itself a long-studied, fundamental quantity in reinforcement learning (Klyubin et al., 2005; Mohamed & Jimenez Rezende, 2015; Gregor et al., 2017), and not inherently tied to exploration. Consequently, it does not make sense to use count/density or successor functions as empowerment baselines, as these are different quantities. The central contribution of the work is an offline empowerment estimator; as an application of it, we show it is useful for exploration value, but our contributions are not tied to exploration.

We have however added a comparison to the state occupancy in Section 6.3 (Fig. 4; details in Appendix A.7). Specifically, we discretize antmaze-medium-navigate and pointmaze-teleport-navigate into grids with five actions (stay and four moves), with transition probabilities counted from the data. Fig. 4 shows that while the state occupancy (N(s)\sqrt{N(s)}) of the offline data is (unsurprisingly) a reasonable proxy, it is sensitive to the offline data distribution in two adversarial ways:

  • Corners and edges are infrequently visited by the offline data collection policy so these states may be assigned artificially low, or even zero exploration value
  • Hallways are assigned too high of a value; for example, in pointmaze, the long hallways at the edges of the maze have many trajectories, so they're assigned a comparable exploration value to the center of the maze, despite being non-central and also dead ends.

We also added a count-based exploration baseline to our online experiments, Distributional Random Network Distillation (DRND) (Yang et al., 2024), which estimates a pseudo-count bonus approximating 1/N(s)1/\sqrt{N(s)} (Section 5.2; details in Appendix A.3.2). We evaluate both CRL + DRND and CRL + DRND + Empowerment on all tasks (Section 6.4, Fig. 5). DRND performs comparably to RND, and adding our empowerment bonus on top of DRND further improves performance on multiple tasks.

These experiments all demonstrate that empowerment is complementary to density/count-based novelty, rather than being a substitute.

(Continued in the next comment.)

AuthorsOpenReview ↗

Official Author Response to Reviewer rWj6 (2/2)

“The estimator operates on raw states (or proprioceptive state). Do you expect it to work from pixels, and if so, what would need to change?”

While our method is general, it is not clear how difficult this would be to scale to pixel-based environments. In general, prior work has shown that scaling mutual information based estimators (not just empowerment) to pixel-based methods is very challenging (Poole et al., 2019; Song & Ermon, 2020). We expect that extending this to video likely requires learning some lower-dimensional encoding of the image, but leave this to future work, given that scaling mutual information estimation to video is in itself an open research problem.

“Do you expect the estimator to transfer to non-goal-conditioned settings, and is the downstream bonus applicable there?”

Yes, the central contribution of the work, the offline empowerment estimator is very general and not tied to goal-conditioned exploration. Prior work has shown applications of empowerment such as assisting humans (Du et al., 2020; Myers et al., 2024), multi-agent behaviour (Franzmeyer et al., 2022; Guckelsberger et al., 2016) and representation learning (Choi et al., 2021; Bharadhwaj et al., 2022) to name a few ; We chose the goal-conditioned setting as this setting is an especially challenging exploration problem and thus is a useful test bed for exploration methods (Park et al., 2025). While we did not directly test applications of the bonus to other settings, we believe the insights about exploration are very general, and our evaluation setup is in line with prior works (Li et al., 2023; Wilcoxson et al., 2025).

We once again thank the reviewer for their feedback. We plan to add these clarifications to the camera-ready version to improve the work.

References:

  • Du et al. “AvE: Assistance via Empowerment.” NeurIPS 2020.
  • Myers et al. “Learning to Assist Humans without Inferring Rewards.” NeurIPS 2024.
  • Franzmeyer et al. “Learning Altruistic Behaviours in RL without External Rewards.” ICLR 2022.
  • Guckelsberger et al. “Intrinsically Motivated General Companion NPCs via Coupled Empowerment Maximisation.” IEEE CIG 2016.
  • Bharadhwaj et al. “Information Prioritization through Empowerment in Visual Model-Based RL.” ICLR 2022.
  • Choi et al. “Variational Empowerment as Representation Learning for Goal-Based RL.” ICML 2021.
  • Klyubin et al. “Empowerment: A Universal Agent-Centric Measure of Control.” IEEE CEC 2005.
  • Mohamed & Jimenez Rezende. “Variational Information Maximisation for Intrinsically Motivated RL.” NeurIPS 2015.
  • Gregor et al. “Variational Intrinsic Control.” ICLR 2017.
  • Poole et al. “On Variational Bounds of Mutual Information.” ICML 2019.
  • Song & Ermon. “Understanding the Limitations of Variational Mutual Information Estimators.” ICLR 2020.
  • Park et al. “OGBench: Benchmarking Offline Goal-Conditioned RL.” ICLR 2025.
  • Li et al. “Accelerating Exploration with Unlabeled Prior Data.” NeurIPS 2023.
  • Wilcoxson et al. “Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration.” ICML 2025.

Thank you for providing more detailed explanation. Most of my concerns are addressed now, so I raised my score from 2 to 3.

Reviewer_31dEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns an empowerment estimate from offline data and uses it to guide online exploration. It jointly trains a skill policy and models of future states. The online results show improvements in several settings.

Critical strengths

+Using offline data to identify states with more opportunities for control is a useful idea. +The experiments cover several environments and dataset types, with five seeds and confidence intervals. The gains when combined with RND and SUPE are encouraging.

Critical weaknesses

-I am not convinced by the step from Eq. (10) to Gaussian sampling in Eq. (15). Eq. (10) scores real future states by their distance to a skill center. Two centers can give the same scores to every real future state, while Gaussian samples around those centers are still different. The estimated mutual information could then reflect a difference in the sampled embeddings rather than a difference in the skills’ actual outcomes. Please explain what makes the Gaussian samples represent the encoded future-state distribution. -The paper needs a more direct check of the estimate. The heatmaps look reasonable, and the online improvements show that the signal is useful. However, these results do not tell us how accurately it measures the defined empowerment. One useful check would be to fix the learned skill policy and compare its actual future outcomes with the predicted distributions. A small environment with known dynamics could also help. This would not require finding the globally optimal empowerment in the continuous tasks.

Questions for authors

Eqs. (7) and (8) train Q using two kinds of samples: a randomly selected dataset state and the observed next state. Please explain why combining these two losses with equal weight learns the future-state distribution in Eq. (3). For example, if Q and V already predicted the true distributions, would the expected training updates leave them unchanged? Please also clarify how these targets are used in the negative-log loss in Appendix A.5.

3 comments
AuthorsOpenReview ↗

Official Author Response to Reviewer 31dE (1/2)

Thank you to the reviewer for their constructive feedback and their interest in our novel offline empowerment estimator. We have also uploaded a revised version of the paper with all changes in blue text for the reviewer's reference. Below we clarify the reviewer’s main concerns. Additionally, we kindly request the reviewer to revise their score if their concerns have been sufficiently addressed, otherwise we welcome any further questions or comments.

“Please explain what makes the Gaussian samples represent the encoded future-state distribution”

You are correct that Eq. 10 alone does not guarantee that the Gaussian samples follow the distribution of encoded future states, but empirically we found that this is ok in practice. We have added detailed results in Appendix A.6., but we summarize the main verification below. The difference between two skills’ log-scores is linear in the encoding as log⁡Vi(ψ)−log⁡Vj(ψ)=(2/d)⟨ϕi−ϕj,ψ⟩+const\log V^i(\psi) - \log V^j(\psi) = (2/d)\langle\phi_i - \phi_j, \psi\rangle + \text{const}, hence two skills score all real future states identically exactly when real encodings don’t vary across the direction separating their centers. Consequently, we can measure the variance of real dataset states along the difference between two skill centers.

Test 1: do skills differ along directions in which real states don’t vary? We took a principal component analysis of real encodings, and found the directions along which real future states vary; in general, we find that the top 28-101 of the 256 latent directions account for 99.9% of the variation. Only 1-2% of the differences between skill centers lie in the remaining directions that are orthogonal to variation in actual state. We also found that removing this orthogonal component and recomputing Eq. 15 with the same Gaussian changes the empowerment estimate by at most 1.5% on every dataset (Table 8), demonstrating that mere differences in sampled embeddings are not driving the empowerment estimate.

Test 2: Does the estimate change if we use real states? If we replace the Gaussian samples with random dataset states in the empowerment estimate, we find that the estimator still has 0.78-0.91 rank correlation with our empowerment estimate (Table 8, Fig. 10), and the estimate only differs by 0.04-0.10 nats per state after affine rescaling. While this formulation is not theoretically correct (hence the need for our Gaussian model), it demonstrates that the estimator does not change much when using real states vs sampled states.

“The paper needs a more direct check of the [empowerment] estimate”

We have added a check for this in Section 6.3 (Table 1, Fig. 4; details in Appendix A.7). Specifically, we discretize the antmaze-medium-navigate, pointmaze-teleport-navigate, and antsoccer-arena-navigate datasets into grids with five actions (stay and four moves), with transition probabilities counted from the data. In these tabular MDPs, the occupancy measure of any policy can be computed exactly, so we compute ground-truth empowerment by searching over skill policies. We train our estimator and baselines on this, and find that our estimator does a quantitatively better job of recovering the region-level structure (see Table 1, Fig. 4) compared to the baselines. Note that this is despite the fact that the offline baselines considered should do especially well on discrete environments where distribution-shift and diversity collapse is less of an issue; a central contribution of our method is that it works for arbtirary/continuous environments. We also find that the skill-conditioned policy recovered from our method is highly empowered. Taking our learned skill policy and computing its future-state distribution in the tabular MDP yields an average rank correlation of 82% with ground truth across environments (“Our skills” in Table 1), which is much higher than the baselines.

As you suggested, we also fixed our learned skill policy and compared its true empowerment with our estimator's predictions in the modified discrete environments. On AntMaze, Ant-Soccer, PointMaze-Teleport, the two agree reasonably well: regional Spearman correlation is 0.90, 0.88, 0.73 respectively, and mean absolute error is around 0.10 nats for both.

(Continued in the next comment.)

AuthorsOpenReview ↗

Official Author Response to Reviewer 31dE (2/2)

“Eqs. (7) and (8) train Q using two kinds of samples: a randomly selected dataset state and the observed next state. Please explain why combining these two losses with equal weight learns the future-state distribution in Eq. (3). For example, if Q and V already predicted the true distributions, would the expected training updates leave them unchanged? Please also clarify how these targets are used in the negative-log loss in Appendix A.5.”

Thank you, we had a minor typo in our exposition that may help to clarify your question. Eq. (8) should instead use s+=ss^+=s for the self-term, with target (1−γ)+γVˉz(s∣s′)(1-\gamma)+\gamma\bar V^z(s\mid s'), so we learn the occupancy that includes the current state: pz(s+∣s,a)=(1−γ)1(s+=s)+γ Es′[pz(s+∣s′)]p^z(s^+\mid s,a)=(1-\gamma)\mathbf 1(s^+=s)+\gamma\,\mathbb E_{s'}[p^z(s^+\mid s')]. We have corrected Eqs. (1), (3) and (8) in the new manuscript.

Explanation of Appendix A.5. For a prediction QQ and a target TT, Eq. (18) uses y^=−log⁡Q\hat y=-\log Q and y=−log⁡Ty=-\log T, so the unclipped loss is κT/Q+log⁡Q\kappa T/Q+\log Q. The targets are T=Vˉ(s+∣s′)T=\bar V(s^+\mid s'), κ=γ\kappa=\gamma (Eq. 7); T=(1−γ)+γVˉ(s∣s′)T=(1-\gamma)+\gamma\bar V(s\mid s'), κ=1\kappa=1 (Eq. 8); and T=Qˉ(s+∣s,a′)T=\bar Q(s^+\mid s,a'), κ=1\kappa=1 (Eq. 9) (Table 7). Its expected minimizer is Q=κ E[T]Q=\kappa\,\mathbb E[T], so single samples of s′s' and a′a' are unbiased.

Note that like other RL methods combining bootstrapping, function approximation, and off-policy learning, we do not have formal convergence guarantees to the fixed point under general function approximation, but we can show that our theory is consistent in discrete environments (Proposition A.1 in Appendix A.5):

Why the two losses learn the right quantity. Consider discrete states. For any future state xx, the recursion we want QQ to satisfy is Q(x∣s,a)=(1−γ) 1(x=s)+γ Es′[V(x∣s′)].Q(x\mid s,a)=(1-\gamma)\,\mathbf 1(x=s)+\gamma\,\mathbb E_{s'}[V(x\mid s')]. This has two cases, and each loss handles one:

  • If x≠sx\neq s, the right-hand side is γ Es′[V(x∣s′)]\gamma\,\mathbb E_{s'}[V(x\mid s')]. Eq. (7) trains QQ toward this target at randomly drawn dataset states. For this analysis, we exclude draws where s+=ss^+=s from Eq. (7).
  • If x=sx=s, the right-hand side is (1−γ)+γ Es′[V(s∣s′)](1-\gamma)+\gamma\,\mathbb E_{s'}[V(s\mid s')]. Eq. (8) trains QQ toward this target at ss itself.

Why equal weights learn the occupancy. Each value Q(x∣s,a)Q(x\mid s,a) is trained by exactly one of the two losses: Eq. (7) for x≠sx\neq s and Eq. (8) for x=sx=s. The weight on a loss scales its gradient but does not change the value it converges to, so any positive weights give the same solution (up to a constant that cancels in the empowerment estimate), and we use equal weights for simplicity. The loss weights also do not need to match the coefficients 1−γ1-\gamma and γ\gamma in the recursion, because those coefficients are already part of the regression targets in Eqs. (7) and (8). Together, the two losses therefore perform the backup above at every xx. For a fixed policy and fixed xx, this is a standard Bellman backup with reward (1−γ)1(s=x)(1-\gamma)\mathbf 1(s=x) and discount γ\gamma. It has a unique solution, which is the discounted future-state occupancy of xx, so this is the only quantity the backups can converge to.

Would correct Q and V stay unchanged? Yes. If QQ and VV are correct, the target in Eq. (7) equals QQ for every x≠sx\neq s, the target in Eq. (8) equals QQ at x=sx=s, and the target in Eq. (9) equals VV. The expected updates are therefore zero, so the true occupancies are a fixed point of training. We also confirmed this numerically on a small tabular MDP with stochastic dynamics to an error of 10−1210^{-12}.

We once again thank the reviewer for their feedback. We plan to add these clarifications to the camera-ready version to improve the work.

Thank you for the clarification. I accept the correction to Eq. (8). The new fixed-policy comparisons directly address my request for quantitative validation and strengthen the empirical support for the estimator. The PCA analysis also helps address the failure case I raised, although the Gaussian sampling step remains an empirically supported approximation rather than a consequence of Eq. (10).

Reviewer_Fo4LOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides a measure of effective empowerment learnt from offline data. The offline empowerment estimate is shown to be empirically useful when used as an exploration bonus on subsequent online learning tasks and in qualitatively measuring where the agent has controllability.

Critical strengths

The full formulation of the effective empowerment measure and losses from features for offline data is a well justified contribution. In Figure 2, the analysis of the empowerment measure over agent interactions in the Ant-Soccer and Cube domains shows, at places, clear alignment of the empowerment measure. Moreover, the empowerment measures high and low where we would expect it to (for instance, higher when approaching a ball, lower after the ball has been kicked). The paper is also well presented and in general, the writing is clear throughout.

Critical weaknesses

The performance gains from using empowerment as an exploration bonus in the online setting overall shows an improvement against included baselines without the bonus, however, in many domains the performance differences between methods with the empowerment bonus and without are small.

Questions for authors

I have questions for the authors which has to do with the effectiveness of the empowerment measure as an exploration bonus relative to related research rather than the ability to estimate empowerment (which I consider to have been done well, hench the initial accept rating).

Firstly, could the authors include results for comparative baselines using a more recent exploration bonus measure of pseudo-novelty than random network distillation (RND) such as distributional random network distillation (DRND [1])? or provide a reason for not including this.

My next related question is only specific to the AntMaze and the PointMaze-Teleport domains. In the AntMaze domain, the empowerment measure in Figure 1 is, as I would expect, higher in the center of the domain where the agent is expected to be able to move around in many unique diverse ways and less in the corners where unique diverse movements is expected to be comparatively limited. If we discretized states in the Ant domains into cells and took the state occupancy of the antmaze-medium-navigate-v0 data and pointmaze-teleport-navigate-v0 data, does it look like the empowerment mappings? and if so, given N(s) is the number of times the agent has been in state s, would using a better measure of 1/sqrt(N(s)) (using DRND instead of RND) make the gaps close between the empowerment bonus and a pseudo-novelty bonus in the online experiments of antmaze-medium?

[1] Kai Yang, Jian Tao, Jiafei Lyu, Xiu Li, Exploration and Anti-Exploration with Distributional Random Network Distillation, Proceedings of the 41st International Conference on Machine Learning, PMLR 235:56397-56421, 2024.

3 comments
AuthorsOpenReview ↗

Official Author Response to Reviewer Fo4L

Thank you to the reviewer for their constructive feedback and their interest in our novel offline empowerment estimator. We have also uploaded a revised version of the paper with all changes in blue text for the reviewer's reference. Below we clarify the reviewer’s main concerns. Additionally, we kindly request the reviewer to revise their score if their concerns have been sufficiently addressed, otherwise we welcome any further questions or comments.

“...in many domains the performance differences between methods with the empowerment bonus and without are small.”

The reviewer is correct in that the empowerment estimator does not always give a big performance gain across all tasks. However, the important point is that adding the empowerment bonus does not hurt performance on any environments (Section 6.4, Fig. 5), and the empowerment can also be combined with existing exploration methods to strictly improve performance. We would also like to highlight that a central contribution of the work is the first offline empowerment estimator; exploration is one demonstrated application of this, but prior work has found other applications of empowerment such as assisting humans (Du et al., 2020; Myers et al., 2024), multi-agent behaviour (Franzmeyer et al., 2022; Guckelsberger et al., 2016) and representation learning (Choi et al., 2021; Bharadhwaj et al., 2022) to name a few. We hope future work will find other applications of our core contributions.

“Could the authors include results…using a more recent exploration bonus measure of pseudo-novelty …such as distributional random network distillation”

Yes, thank you for this suggestion, we now have included results with DRND (Yang et al., 2024) in the manuscript (Section 6.4, Fig. 5; details in Appendix A.3.2); the results are generally the same as RND, where we again find that combining empowerment with DRND can help get some improvement over just DRND (and accordingly, these gains are persistent on antmaze-medium).

“If we discretized states in the Ant domains into cells and took the state occupancy of the antmaze-medium-navigate-v0 data and pointmaze-teleport-navigate-v0 data, does it look like the empowerment mappings?”

We've added some results for this in Section 6.3 (Fig. 4; details in Appendix A.7). Specifically, we discretize antmaze-medium-navigate and pointmaze-teleport-navigate into grids with five actions (stay and four moves), with transition probabilities counted from the data. Fig. 4 shows that while the state occupancy (N(s)\sqrt{N(s)}) of the offline data is (unsurprisingly) a reasonable proxy, it is highly sensitive to the distribution of the offline data in two adversarial ways:

  • Corners and edges are infrequently visited by the offline data collection policy so these states may be assigned artificially low, or even zero exploration value
  • Hallways may be assigned too high of an exploration value; for example, in pointmaze, the long hallway stretches at the edges of the maze happen to have many trajectories in the offline data, so they are assigned a comparable exploration value to the center of the maze despite being non-central and also a dead end

Evidently empowerment and visitation density while being correlated, are two fundamentally very different things. Empirically, we also found above that DRND does not help much in most environments, and combining it with empowerment still helps, further supporting our hypotheses.

We once again thank the reviewer for their feedback. We plan to keep these clarifications in the camera-ready version to improve the work.

References:

  • Du et al. “AvE: Assistance via Empowerment.” NeurIPS 2020.
  • Myers et al. “Learning to Assist Humans without Inferring Rewards.” NeurIPS 2024.
  • Franzmeyer et al. “Learning Altruistic Behaviours in RL without External Rewards.” ICLR 2022.
  • Guckelsberger et al. “Intrinsically Motivated General Companion NPCs via Coupled Empowerment Maximisation.” IEEE CIG 2016.
  • Bharadhwaj et al. “Information Prioritization through Empowerment in Visual Model-Based RL.” ICLR 2022.
  • Choi et al. “Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning.” ICML 2021.
  • Yang et al. “Exploration and Anti-Exploration with Distributional Random Network Distillation.” ICML 2024.

Dear Reviewer Fo4L,

Thank you once again for your insightful feedback on our submission. As the discussion phase is coming to a close, we would like to follow up and ask if our rebuttal has sufficiently addressed your questions. If there are any outstanding questions or further clarifications, we are happy to answer them.

Thank you for your detailed rebuttal. In light of my concerns being well addressed and the clear contribution of the research, I have raised my score from 3 to 4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Summary of the Additional Rebuttal Experiments

We thank all reviewers for their constructive feedback and their interest in our novel offline empowerment estimator. We have uploaded a revised version of the paper with all changes in blue text for the reviewers' reference. We would like to summarize the main additional experiments below, all of which further support our work's conclusions:

  1. Direct evaluation against ground-truth empowerment: We have added a check for this in Section 6.3 (Table 1, Fig. 4; details in Appendix A.7). Specifically, we discretize the antmaze-medium-navigate, pointmaze-teleport-navigate, and antsoccer-arena-navigate datasets into grids with five actions (stay and four moves), with transition probabilities counted from the data. In these tabular MDPs, the occupancy measure of any policy can be computed exactly, so we compute ground-truth empowerment by searching over skill policies. We train our estimator and baselines on this, and find that our estimator does a quantitatively better job of recovering the region-level structure compared to the baselines. We also find that the skill-conditioned policy recovered from our method is highly empowered, yielding an average rank correlation of 82% with ground truth across environments, which is much higher than the baseline methods.

  2. Validating the Gaussian representation: We have added detailed results in Appendix A.6 to verify that our Gaussian samples provide a reasonable approximation for estimating empowerment. We find that only 1-2% of the differences between skill centers lie in directions orthogonal to variation in actual states, and removing this component changes the empowerment estimate by at most 1.5% on every dataset. Additionally, if we replace the Gaussian samples with random dataset states in the empowerment estimate, we find that the estimator still has high rank correlation with our empowerment estimate, and the estimate only differs by 0.04-0.10 nats per state after affine rescaling. These results demonstrate that mere differences in sampled embeddings are not driving the empowerment estimate, and the estimator is learning some meaningful structure.

  3. Additional exploration experiments and analysis: We have also included results with DRND (Yang et al., 2024). These are very similar to our RND results, and once again support our contributions by showing that empowerment bonuses help improve performance on online RL tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.