acceptodds
Under review as a conference paper at ICLR 2027

MIM-DICE: Unsupervised Skill Discovery from Arbitrary Off-policy Experience

Abstract

Unsupervised reinforcement learning (URL) aims to acquire reusable behaviors without external supervision for rapid adaptation to downstream tasks. Within URL, skill learning achieves this by discovering skills whose induced state distributions are mutually distinct yet collectively provide broad state coverage. A common way to induce these properties is to maximize the mutual information (MI) between skills and policy-induced states. However, existing methods entail a state distribution mismatch between the off-policy replay data and the current policy. This leads to two critical limitations in practice. First, as we empirically show, their biased estimation of MI impedes effective MI maximization. Second, they strictly rely on replay data collected by skill-conditioned policies, precluding skill learning from arbitrary offline datasets. We introduce Mutual Information Maximization Distribution Correction Estimation (MIM-DICE), a DICE-based method that corrects this distribution mismatch, enabling off-policy unbiased and effective MI maximization. Furthermore, by eliminating the reliance on skill-conditioned data, MIM-DICE enables skill learning directly from arbitrary offline datasets. In online off-policy experiments on tabular environments and Pendulum, MIM-DICE better maximizes MI under the current policy than prior methods. Across six MAZE domains, it learns diverse skills directly from arbitrary offline datasets, achieving higher MI and inducing more clearly separated state-visitation regions. Finally, MIM-DICE outperforms a broad range of state-of-the-art baselines on the URLB benchmark suite.

Primary area:
reinforcement learning
Keywords:
off-policy learning, unsupervised skill discovery, mutual information maximization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BPEaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MIM-DICE, a method for unsupervised skill discovery, It formulates skill learning as MI maximization over skill-conditioned occupancy measures, regularized to remain close to the replay distribution. To handle the nonconcavity of the objective, it introduces an auxiliary surrogates and a minorization–maximization procedure, then solved using DICE-style dualization. Experiments evaluate policy-induced mutual information in tabular MDPs and Pendulum, offline skill discovery in MAZE, and downstream performance on URLB.

Critical strengths
  1. The MM and DICE derivations form a coherent approach to optimizing skill-conditioned occupancies under flow constraints.
  2. The evaluation spans tabular settings, continuous control task, offline RL, and downstream adaptation.
Critical weaknesses

The main concern about this paper is that the proposed motivation is not fully clear. I find the occupancy-based formulation and its derivation internally coherent: optimizing a DICE-style objective with respect to the current policy’s occupancy is a reasonable way to address the stated optimization problem. My concern is why replay–policy mismatch motivates a DICE-based solution specifically? This mismatch is common in off-policy RL and may also affect the paper’s MI-based baselines, including CIC, BeCL, and CeSD. But they doesn't training badly. The paper compares against these methods, but I wonder why the mismatch is particularly consequential for them or explain why the proposed correction is developed only within a DICE framework.

Furthermore, assuming the mismatch does affect the performance of policy improvement in URL, the experiments do not isolate the benefit of occupancy correction. The reported comparisons cannot determine whether MIM-DICE’s gains come from correcting replay–policy mismatch only, not its DICE objective, or implementation differences. A direct ablation study with a vanilla DICE could help. This would test whether the gains are specifically attributable to the proposed correction and help this paper more trustworthy.

Questions for authors
  1. Question: The skills trajectories seem not distinguish enough on the visualization of 2D Maze (table 1)? What about compared with DIAYN and DADS?

  2. typo: Appendix C.1 seems contains an unresolved equation reference: “Substituting Eq. equation ?? gives”?

  3. typo: "an policy" on 324 line

4 comments

[W1: Why the replay-policy mismatch motivates a DICE-based correction]

(1) Why the mismatch matters more for MIM than for standard off-policy RL. In standard off-policy RL, the reward is a fixed function given by the environment and does not depend on the occupancy of the policy, so the mismatch between the policy and the data affects only how values are estimated. In mutual information maximization (MIM), the reward itself is defined by the state occupancies of the current policy, whereas existing off-policy MIM methods compute it with the occupancies of the replay buffer:

  • rπ(s,z)=−log⁡dˉπ(s)+log⁡dˉzπ(s)r^{\pi}(s,z) = -\log \bar d^{\pi}(s) + \log \bar d^{\pi}_z(s) (the true reward for MI maximization),
  • rB(s,z)=−log⁡dˉB(s)+log⁡dˉzB(s)r^{\mathcal{B}}(s,z) = -\log \bar d^{\mathcal{B}}(s) + \log \bar d^{\mathcal{B}}_z(s) (the reward used by existing off-policy MIM methods).

Replacing rπr^{\pi} with rBr^{\mathcal{B}} changes the objective itself, and Section 3.1 gives the exact gap.

(2) Why DICE. MI is the expectation of rπr^{\pi} under dzd_z, but rπr^{\pi} itself depends on the policy occupancies, so the objective is a nonlinear function of the occupancies dz(s,a)d_z(s,a). DICE optimizes directly over occupancies under the Bellman flow constraints, so this dependence is handled exactly. Moreover, with the ratio wz(s,a)=dz(s,a)/dB(s,a)w_z(s,a) = d_z(s,a)/d^{\mathcal{B}}(s,a) and Lagrangian duality, the problem involves only expectations under the data distribution dBd^{\mathcal{B}} (Eq. (15)), so it can be estimated from off-policy samples without knowing the behavior policy. Other off-policy corrections do not provide both properties. Value-based methods assume a reward that does not change with the policy, and importance sampling requires the behavior policy and has variance that grows exponentially with the horizon (curse of horizon).

(3) Why the mismatch is consequential for CIC, BeCL, and CeSD. These methods perform well on downstream tasks, but their policy-induced MI is low when trained with off-policy data. On Pendulum they reach 0.13, 0.37, and 0.02, against 0.94 for MIM-DICE (Table 1a). On tabular MDPs, training with the replay-based reward yields lower policy-induced MI than training with the true reward, and the replay-buffer MI differs substantially from the policy-induced MI (Figure 1); all prior methods also remain below MIM-DICE in policy-induced MI (Figure 2a). The mismatch therefore limits how well they maximize MI, even when their downstream scores are high.

[W2. Isolating the benefit of occupancy correction with a vanilla DICE ablation]

Following the reviewer’s suggestion, we ran a direct ablation with vanilla DICE (new, Table 2). It shows that the gains of MIM-DICE come from the occupancy correction, and not from the DICE objective or from implementation differences.

We applied vanilla DICE (OptiDICE) to APS, DIAYN, and BeCL while keeping their original intrinsic rewards. These rewards are computed from the replay buffer, like rBr^{\mathcal{B}} in W1, and not from the occupancies of the current policy as in the true reward rπr^{\pi}.

MethodOriginal MIDICE variant MI
MIM-DICE0.94 ± 0.05
APS0.60 ± 0.120.59 ± 0.05
DIAYN0.66 ± 0.080.52 ± 0.05
BeCL0.37 ± 0.090.37 ± 0.07

Table 2 (new). Policy-induced MI on Pendulum for the original methods and their intrinsic-reward DICE variants.

The variants are optimized with DICE, and their MI does not improve over the original methods, so the DICE objective alone does not explain the gains. They also use the same policy extraction and implementation as MIM-DICE, and MIM-DICE still outperforms them under the same divergence and coefficient (Table 7), which rules out implementation differences. The only remaining difference is occupancy correction. The variants optimize rewards computed from the replay buffer, while MIM-DICE corrects the occupancy through the ratio wz(s,a)=dz(s,a)dB(s,a)w_z(s,a) = \frac{d_z(s,a)}{d^{\mathcal{B}}(s,a)} and optimizes the policy-induced MI itself.

[Q1. Skill distinguishability on MAZE and comparison with DIAYN and DADS]

In Figure 2c, the color of each point encodes the two-dimensional continuous skill (z0,z1)(z_0, z_1) through the color map shown at the top left. When all skills visit the same states, the colors mix into a single muddy color, as in the BUFFER column and in the baselines. When different skills occupy different regions, the regions show distinct colors, as in the MIM-DICE column. Since the skills are continuous, the color changes gradually between neighboring regions, which is expected and does not mean that the skills are not distinguishable.

Following the reviewer’s question, we additionally evaluated DIAYN and DADS on MAZE (new, Table 5, details in Appendix B.4).

Data CoverageMIM-DICE (Ours)APSCICDIAYNDADS
50%2.14 ± 0.290.43 ± 0.070.10 ± 0.020.05 ± 0.010.05 ± 0.01
70%2.09 ± 0.160.30 ± 0.080.07 ± 0.010.04 ± 0.010.06 ± 0.01
100%2.10 ± 0.190.39 ± 0.060.06 ± 0.010.04 ± 0.010.03 ± 0.00

Table 5. Offline MAZE results across dataset-coverage levels.

Among the evaluated methods, MIM-DICE is the only method that effectively increases policy-induced MI. We have also revised Appendix G to include MAZE visualizations for DIAYN and DADS (Figures 4–6).

[Q2 & Q3. Typos]

Thank you for pointing these out. We fixed the unresolved reference in Appendix C.1 and the typo “an policy”.

Thank you for the response and efforts in addressing my questions and concerns. I will maintain my current rating.

Thank you for following up on our response. Your comments were very helpful in improving the paper, and we are glad we had the chance to address them.

Reviewer_uGCrOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIM-DICE, a DICE-based method that corrects the distribution mismatch between the off-policy replay data and the current policy, enabling off-policy unbiased and effective MI maximization.

Critical strengths
  1. The claim of this paper is supported by both theoretical analysis and experimental results.

  2. This paper is of high writing quality, introducing the significance of MIM-DICE.

Critical weaknesses
  1. The experiments are relatively simple in scenes and settings. The environments in this paper are either tabular environments or simple continuous control (Pendulum). These experiments have not demonstrated whether the proposed method can be applied to complex MDPs with complicated dynamics and occupancies.

  2. The authors claim that the tabular environments are chosen since policy occupancies and MI can be computed directly (line: 364). The proposed method might be weak in the face of those environments whose policy occupancies and MI are difficult to compute.

Questions for authors
  1. In the appendix, the authors claim that MIM-DICE uses a 64-dimensional continuous skill variable. How is "64" chosen? Whether have the authors evaluated MIM-DICE on other hyper-parameters to find the best combinations of parameters?

  2. For those environments where policy occupancies and MI cannot be computed exactly, how are they estimated? It may need some explanations in detail.

2 comments

[W1: Experiments beyond tabular MDPs and Pendulum]

Please note that our experiments are not limited to tabular MDPs and Pendulum. We evaluate MIM-DICE in three settings of increasing complexity.

(1) Controlled diagnosis (tabular MDPs and Pendulum, Figures 2a and 2b, Tables 1a and 2). We use small domains here on purpose. The policy-induced MI of every method can be computed exactly (tabular) or estimated by discretizing a two-dimensional state space (Pendulum), which lets us verify directly whether each method maximizes it. MIM-DICE achieves the highest policy-induced MI in both domains.

(2) Fully offline skill discovery (six MAZE layouts at three dataset-coverage levels, Figure 2c, Tables 1b and 5, Appendix G). This setting tests whether skills can be learned from a fixed dataset that is collected independently of skills and has no skill labels, with continuous states, actions, and skills. MIM-DICE achieves a policy-induced MI of about 2.1, while the best baseline (APS) reaches at most 0.43.

(3) Downstream adaptation on URLB (12 tasks on Walker, Quadruped, and Jaco, Figure 3, Table 4). This setting tests whether skills learned without rewards are useful for downstream tasks in high-dimensional continuous control, including 3D quadruped locomotion and a 6-DOF arm with a three-finger gripper. MIM-DICE achieves the best aggregate performance.

This suite follows the protocol of recent work. BeCL and CeSD are evaluated on mazes and URLB, and SEMDICE on tabular MDPs, MountainCar, and URLB. In the revision we also added results under stochastic dynamics (Appendix E.1, Table 13).

[W2,Q2. Reliance on exact occupancy and MI computation]

Please note that tabular MDPs are only one of our experimental settings. We include them because the policy-induced MI can be measured exactly there, which makes them an ideal testbed for verifying whether each method maximizes it. MIM-DICE itself does not need to compute policy occupancies or MI directly for training. It optimizes the sample-based objective of Eq. (15) using only transitions (s,a,s′)(s,a,s') from the dataset, so the difficulty of computing occupancies or MI does not limit where it can be applied. The MAZE and URLB results show this for problems with continuous states and actions (Tables 1b, 4, and 5, Figure 3).

In such environments, MI is estimated only for evaluation, not for training (new, Appendix B.2). On Pendulum and MAZE, we roll out the learned policy for each evaluation skill, discretize each state dimension into 10 bins, and compute I^π(S;Z)=∑zp(z)∑sp^(s∣z)log⁡p^(s∣z)p^(s)\hat I_\pi(S;Z) = \sum_z p(z) \sum_s \hat p(s | z) \log \frac{\hat p(s | z)}{\hat p(s)}, where p^(s∣z)\hat p(s | z) is the empirical visitation frequency of bin ss under skill zz, p^(s)=∑zp(z)p^(s∣z)\hat p(s) = \sum_z p(z) \hat p(s | z), and bins with zero count are omitted. All methods are evaluated with the same bins, evaluation skills, and number of steps, so the bias of this estimator affects them equally. On tabular MDPs, no estimation is needed: we solve the discounted occupancy equation for each skill-conditioned policy and compute MI exactly. On URLB, we follow the standard benchmark protocol and evaluate downstream performance.

[Q1. Choice of the skill dimension and other hyperparameters]

We set the skill dimension to 64 following CIC, which uses 64-dimensional continuous skills on URLB. We did not tune it for MIM-DICE. This setting applies only to URLB. The other experiments use 8 discrete skills (tabular MDPs), 4 discrete skills (Pendulum), and two-dimensional continuous skills (MAZE). Following the reviewer’s question, we additionally evaluated 32 and 96 dimensions on Jaco (new, Table 9).

Jaco task32 dimensions64 dimensions96 dimensions
Reach Top Left69.66 ± 17.4479.60 ± 9.1086.43 ± 8.94
Reach Top Right71.18 ± 9.7384.60 ± 5.6059.37 ± 7.62
Reach Bottom Left71.67 ± 5.7282.90 ± 5.9080.48 ± 9.49
Reach Bottom Right72.51 ± 4.0676.70 ± 8.4083.27 ± 5.39
Average71.26 ± 5.1180.96 ± 3.8177.39 ± 1.78

Table 9. Jaco downstream scores under different skill dimensions

The table reports expert-normalized scores. The average is highest with 64 dimensions, but the gap to 96 dimensions is within one standard error, and the best dimension varies by task. MIM-DICE is therefore not very sensitive to the skill dimension in this range.

For the other hyperparameters, we did not search for the best combination. Most are inherited from prior work without tuning. The DDPG agent uses the standard URLB settings (Table 3), and the density estimator uses the settings of APT and SEMDICE (Appendix B.1). For the hyperparameters specific to MIM-DICE, we evaluated other values on Pendulum: the choice of ff-divergence and the coefficient α\alpha (Table 7), and, following the reviewer’s question, the number of neighbors kk of the density estimator (new, Table 8).

Choice of ff-divergence (Table 7):

KLPearson χ2\chi^2Soft-χ2\chi^2
MI0.581 ± 0.1560.957 ± 0.0330.943 ± 0.048

Regularization coefficient α\alpha with Soft-χ2\chi^2 (Table 7):

α\alpha0.10.20.51.0
MI0.943 ± 0.0480.938 ± 0.0710.903 ± 0.0700.352 ± 0.112

Number of neighbors kk of the density estimator (new, Table 8):

kk26122464
MI1.047 ± 0.0461.082 ± 0.0210.929 ± 0.0941.078 ± 0.0311.122 ± 0.016

MIM-DICE performs well over a range of values. Pearson χ2\chi^2 and Soft-χ2\chi^2 give similar MI, MI remains high for α≤0.5\alpha \le 0.5, and it changes little across kk.

Reviewer_djfCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIM-DICE for unsupervised skill discovery from skill-unlabeled off-policy data. It combines minorization–maximization and DICE-based occupancy correction to better align the optimized objective with current-policy mutual information. Experiments cover tabular MDPs, Pendulum, MAZE, and URLB.

Critical strengths
  1. The paper identifies a meaningful replay–policy distribution mismatch and provides a clear mathematical motivation for correcting it.
  2. Discovering continuous skills from offline transitions without skill labels is practically useful.
  3. The method achieves strong policy-induced MI and skill separation on Pendulum and MAZE, with competitive aggregate URLB results.
Critical weaknesses
  1. Technical consistency: Eq. (13)/(15) and Eq. (62) appear to use different coefficients for the dual variable μ(s). The authors should clarify which residual is implemented and whether policy extraction matches the derivation. The MM monotonicity argument also does not directly guarantee convergence for approximate neural updates.
  2. Novelty and comparisons: The method builds on established DICE and MM techniques. Relevant recent work, including SD3, HUSD, GenDa, and DiMS, is missing from the discussion and comparable benchmarks; its distinction from DOI and SEMDICE should be sharper.
  3. Robustness and ablations: The broad claim of learning from arbitrary off-policy data is not adequately tested under poor action coverage, inaccurate density estimation, or stochastic dynamics. Key ablations isolating occupancy correction, policy extraction, and skill selection, as well as computational-cost comparisons, are lacking.
  4. Uneven downstream gains: Strong aggregate URLB performance masks significant gaps on Quadruped tasks: Walk scores 26.9 versus 69.7 for ExDM, and Jump scores 49.9 versus 69.7 for CeSD. The paper does not sufficiently explain why higher MI fails to translate into stronger control performance in these settings.
Questions for authors
  1. Which residual expression is used in the implementation, and how does the discrepancy affect occupancy-ratio recovery?
  2. How sensitive is the method to limited dataset support, density-estimation errors, and stochastic transitions?
  3. Can the authors provide component ablations and matched-compute comparisons, and explain the weaker Quadruped results?
  4. How does the method compare with recent related skill-discovery approaches under compatible protocols?
4 comments

We sincerely thank the reviewer for the careful and detailed review. We address the raised weaknesses and questions below. Equation, table, and line numbers refer to the revised manuscript.

[W1-1,Q1: Residual used in the implementation and occupancy-ratio recovery]

Eqs. (13), (15), and (66) (Eq. (62) in the submitted version) use the same coefficient for μ(s)\mu(s), which is p(z)p(z) in all three. There is no discrepancy between the derivation and the implementation.

Eq. (13), argument of f∗f^{\ast}:

p(z)(log⁡qzk(s)+μ(s))−λz(s)+γ∑s′P(s′∣s,a),λz(s′)α,p(z)\frac{{\color{red}{p(z)}}\big(\log q_z^k(s) + {\color{blue}{\mu(s)}}\big) - \lambda_z(s) + \gamma \sum_{s'} P(s' \mid s,a)\\,\lambda_z(s')}{\alpha\\,p(z)}

Eq. (15), argument of f∗f^{\ast}:

p(z)(log⁡qz(s)+μ(s))−λz(s)+γ,λz(s′)α,p(z)\frac{{\color{red}{p(z)}}\big(\log q_z(s) + {\color{blue}{\mu(s)}}\big) - \lambda_z(s) + \gamma\\,\lambda_z(s')}{\alpha\\,p(z)}

Eq. (66), single-sample residual:

e^λ,μ,q(s,a,s′,z)=p(z)(log⁡qz(s)+μ(s))−λz(s)+γ,λz(s′)\hat e_{\lambda,\mu,q}(s,a,s',z) = {\color{red}{p(z)}}\big(\log q_z(s) + {\color{blue}{\mu(s)}}\big) - \lambda_z(s) + \gamma\\,\lambda_z(s')

Eq. (66) is the only residual expression in our implementation, and it enters the training objective Eq. (15) directly. Policy extraction evaluates the ratio at actions sampled from the policy, for which no next state s′s' is available. We therefore train the residual model eξ(s,a,z)e_\xi(s,a,z), a neural network that does not take s′s' as input, by regressing it onto Eq. (66) (Eq. (67)). Since a squared-error regression recovers the conditional mean of its target, eξ(s,a,z)e_\xi(s,a,z) approximates Es′∼P(⋅∣s,a)[e^λ,μ,q(s,a,s′,z)]\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\big[\hat e_{\lambda,\mu,q}(s,a,s',z)\big], which is exactly the numerator of Eq. (13). The ratio wzξ(s,a)=∇f∗!(eξ(s,a,z)α,p(z))w^\xi_z(s,a) = \nabla f^{\ast}\\!\left(\frac{e_\xi(s,a,z)}{\alpha\\,p(z)}\right) (Eq. (68)) is therefore the optimal ratio wz⋆(s,a)w^\star_z(s,a) of Eq. (58) evaluated with the learned residual. Occupancy-ratio recovery follows the derivation without modification.

Policy extraction matches the derivation: it minimizes the I-projection objective of Eq. (65) with the learned ratio wzξ(s,a)w^\xi_z(s,a) (Eq. (69)). Following SEMDICE (Lee et al., 2025), we set ηBC=0\eta_{\mathrm{BC}} = 0 on URLB, which removes the KL term toward the behavior policy, since under online training the behavior policy is an older version of the current policy.

[W1-2: MM monotonicity under approximate neural updates]

The monotonicity result in Section 3.3 holds for exact MM updates, and we do not claim guarantees beyond that setting. As is standard in deep RL, the exact analysis is the principle from which the practical algorithm is derived, and the neural implementation approximates these updates. SAC (Haarnoja et al., 2018), for example, is derived from soft policy iteration, whose convergence is proved for exact updates, and is implemented with neural approximations.

In practice, the neural implementation of MIM-DICE maximizes MI effectively, reaching the highest policy-induced MI among the compared methods on Pendulum and MAZE (Table 1).

[W2,Q4: Novelty and comparisons with related methods]

DICE and MM are established techniques, but neither one alone solves mutual information maximization (MIM). DICE dualization requires a concave objective in the occupancies dd, but the MI objective is not concave: the term −H(S∣Z)-H(S \mid Z) is convex in dd, which makes the objective a difference of convex functions (Section 3.3). Our contribution is a formulation that addresses this challenge. At each MM iteration we replace −H(S∣Z)-H(S \mid Z) with a lower bound that is linear in the occupancies, which reduces MIM to a sequence of concave problems that DICE can solve from off-policy data. We show that the dual of each problem can be estimated from arbitrary offline data (Proposition 1, Appendix A.3). In contrast, prior MIM methods estimate their reward from per-skill replay data, so every transition must be labeled with its skill. Consequently, when optimized on fixed offline datasets, MIM-DICE significantly outperforms the baselines in policy-induced MI (Tables 1b and 5, Figure 2a,c)

Following the reviewer’s suggestion, we added SD3 as a URLB baseline (Figure 3, Table 4) and discuss SD3, HUSD, GenDa, and DiMS in Section 4 of the revision. On the URLB benchmark, MIM-DICE outperforms SD3 on average (66.53 vs 59.68 in Table R1). HUSD and DiMS have no public implementation, and their papers omit details needed to reproduce them (e.g., skill distributions and sampling procedures). GenDa does not report URLB results.

DomainMIM-DICESD3
Walker74.8570.97
Quadruped43.7855.50
Jaco80.9652.55
Overall66.5359.68

Table R1. Average expert-normalized URLB scores per domain (from Table 4 in the revision).

Among DICE-based methods, DOI differs from MIM-DICE in three ways. First, the setting: DOI learns skills under an imitation constraint and requires expert demonstrations, while MIM-DICE uses neither demonstrations nor rewards. Second, the surrogate: the two MI decompositions are equal in value, but each method optimizes a different variational bound. DOI bounds H(Z)−H(Z∣S)H(Z) - H(Z \mid S) with a skill discriminator q(z∣s)q(z \mid s) over a finite set of skills, while MIM-DICE handles H(S)H(S) exactly through the dual variable μ\mu and bounds only −H(S∣Z)-H(S \mid Z) with q(s∣z)q(s \mid z), which also covers continuous skills. Third, the empirical behavior: on Pendulum and URLB the DOI discriminator collapses to the uniform distribution (Appendix D.3), a failure mode also reported in the DOI paper. In contrast, MIM-DICE shows no such collapse. Prior work also reports better exploration and downstream performance for methods based on H(S)−H(S∣Z)H(S) - H(S \mid Z) (APS, Liu & Abbeel, 2021; CIC, Laskin et al., 2022).

SEMDICE maximizes the state entropy H(S)H(S) of a single policy. This objective is concave in the occupancy, so one DICE dualization suffices and no MM is needed. MIM-DICE learns a family of skill-conditioned occupancies that are coupled through their marginal, and adds the term −H(S∣Z)-H(S \mid Z), which makes the objective nonconcave and requires the MM construction described above.

[W3-1,Q2: Sensitivity to limited dataset support, density-estimation errors, and stochastic transitions]

MIM-DICE inherits the ability of DICE methods to learn from data collected by any policy, and unlike prior MIM methods (e.g., DIAYN, APS, CIC), it does not require skill labels in the data. The offline experiments demonstrate this: in Figure 2a (offline), the datasets are collected by a uniformly random policy, and in Table 5, the MAZE datasets are collected independently of any skill. In both cases MIM-DICE significantly outperforms the baselines in policy-induced MI. As with any DICE method, the learned occupancies dz=wz dBd_z = w_z\, d^{\mathcal{B}} stay within the support of the data, so the coverage of the data determines which skills can be learned.

For the three factors raised by the reviewer, we ran additional experiments on action coverage, density estimation, and stochastic transitions, which are included in the revision. MIM-DICE achieves the highest policy-induced MI in every setting. Its MI is not sensitive to the density estimator or to stochastic transitions, and under limited support it drops noticeably only at the lowest action coverage (25%).

(1) Limited dataset support: action coverage (new, Table 14, Appendix F) and state coverage (Table 5). Following the reviewer's comment on poor action coverage, we added an experiment that varies the action coverage of the dataset. We use tabular MDPs with 20 states, 8 actions, and 8 skills, where the policy-induced MI can be computed exactly. At each state, the dataset contains only 2, 4, 6, or 8 of the 8 actions (25% to 100% action coverage) and is collected by a uniform policy over these actions. Skill labels are assigned at random, independently of the data. All methods are trained on the same fixed dataset and are restricted to the observed actions, for which transition statistics are available.

Action coverageMIM-DICEAPSDIAYNVISRCIC
2 actions (25%)0.351 ± 0.0290.092 ± 0.0080.046 ± 0.0030.091 ± 0.0080.049 ± 0.005
4 actions (50%)0.552 ± 0.0210.059 ± 0.0040.031 ± 0.0020.059 ± 0.0040.057 ± 0.003
6 actions (75%)0.543 ± 0.0220.047 ± 0.0030.025 ± 0.0010.047 ± 0.0030.064 ± 0.004
8 actions (100%)0.499 ± 0.0250.041 ± 0.0020.022 ± 0.0010.041 ± 0.0020.067 ± 0.003

Table 14. Exact policy-induced MI (nats) on offline tabular MDPs under different action coverage (mean ± SE over 10 seeds).

MIM-DICE achieves the highest MI at every coverage level, at least 3.8 times that of the best baseline (at most 0.092). Its MI is similar from 50% to 100% coverage (0.50 to 0.55) and decreases to 0.351 at 25%. This decrease is expected, since with only two actions per state fewer distinct state distributions can be realized from the data. Even in this case MIM-DICE learns distinguishable skills, while the baselines remain below 0.1 at every level.

For state coverage, we restrict the MAZE datasets to cover 100%, 70%, or 50% of the state space (Table 5). MIM-DICE reaches the same MI (about 2.1) at all three levels and stays far above the baselines (at most 0.43).

(2) Density-estimation errors (new, Table 8). MIM-DICE uses a kNN estimate of the data density, and we change this estimate by varying the number of neighbors kk on Pendulum. The MI of MIM-DICE changes little and is the highest at every kk.

Methodk=2k=2k=6k=6k=12k=12k=24k=24k=64k=64
MIM-DICE1.047 ± 0.0461.082 ± 0.0210.929 ± 0.0941.078 ± 0.0311.122 ± 0.016
APS0.722 ± 0.0480.571 ± 0.0570.540 ± 0.0730.588 ± 0.0580.483 ± 0.101
CIC0.252 ± 0.0530.177 ± 0.0720.106 ± 0.0260.058 ± 0.0220.073 ± 0.038

(3) Stochastic transitions (new, Table 13). We add Gaussian noise to the actions in Pendulum and use the extension of MIM-DICE for stochastic dynamics (Appendix E). MIM-DICE reaches the highest MI and state coverage, and its MI is close to the 0.94 of the deterministic setting.

MethodMIState coverage (%)
MIM-DICE0.968 ± 0.04791.60 ± 0.40
DIAYN0.732 ± 0.05272.20 ± 2.58
APS0.666 ± 0.05186.60 ± 3.34
BeCL0.271 ± 0.11860.00 ± 9.64

[W3-2,Q3: Component ablations and matched-compute comparisons]

(1) Occupancy correction. Three results isolate the effect of occupancy correction, and each shows that occupancy correction is essential for maximizing policy-induced MI from off-policy data.

  • Figure 1. On tabular MDPs, we train with two versions of the intrinsic reward for MI maximization, keeping everything else equal:

    • rπ(s,z)=−log⁡dˉπ(s)+log⁡dˉzπ(s)r^{\pi}(s,z) = -\log \bar d^{\pi}(s) + \log \bar d^{\pi}_z(s), computed with the state distributions of the current policy (the true MI objective),
    • rB(s,z)=−log⁡dˉB(s)+log⁡dˉzB(s)r^{\mathcal{B}}(s,z) = -\log \bar d^{\mathcal{B}}(s) + \log \bar d^{\mathcal{B}}_z(s), computed with the state distributions of the replay buffer (as in existing off-policy MIM methods).

    Training with rBr^{\mathcal{B}} yields significantly lower policy MI than training with rπr^{\pi}. Occupancy correction addresses this gap: MIM-DICE optimizes the ratio wz(s,a)=dz(s,a)/dB(s,a)w_z(s,a) = d_z(s,a) / d^{\mathcal{B}}(s,a), so the MI objective is evaluated on the policy occupancy dzd_z while sampling only from the buffer. In the same tabular setting, MIM-DICE achieves the highest policy MI among the off-policy methods (Figure 2a).

  • Table 5. On the offline MAZE datasets, where skills are assigned independently of states, MIM-DICE significantly outperforms the baselines in policy-induced MI. In this setting dˉzB=dˉB\bar d^{\mathcal{B}}_z = \bar d^{\mathcal{B}}, so the replay-buffer reward above is zero for every state and skill, and methods that estimate their reward from the dataset receive no signal to separate skills. MIM-DICE optimizes the skill-conditioned occupancies directly and does not rely on this reward.

  • Table 2 (new). On Pendulum, MIM-DICE significantly outperforms DICE variants of APS, DIAYN, and BeCL. These variants replace the original policy optimization with DICE but keep the intrinsic rewards computed from the replay buffer, and their MI does not improve over the original methods. This implies that using DICE to optimize an existing intrinsic reward is not sufficient, and that occupancy correction of the MI objective itself is essential.

MethodOriginal MIDICE variant MI
MIM-DICE0.94 ± 0.05
APS0.60 ± 0.120.59 ± 0.05
DIAYN0.66 ± 0.080.52 ± 0.05
BeCL0.37 ± 0.090.37 ± 0.07

Table 2. Policy-induced MI on Pendulum for the original methods and their intrinsic-reward DICE variants

(2) Policy extraction (Table 2). Table 2 also isolates policy extraction. The DICE variants combine the (biased) replay-based intrinsic rewards of APS, DIAYN, and BeCL with the DICE-based policy optimization and the policy extraction of MIM-DICE. Their MI does not improve, so policy extraction is not the source of the improvement of MIM-DICE.

(3) Skill selection (new, Table 10). Skill selection is needed only when a downstream reward is given, which is the case in URLB fine-tuning. The tabular, Pendulum, and MAZE experiments evaluate policy-induced MI without a downstream task, so they involve no skill selection. On URLB, replacing our regression-based selection with CEM lowers the average score only slightly, so the downstream performance is not very sensitive to the specific selection method.

DomainRegression (ours)CEM
Walker74.8572.78
Quadruped43.7843.12
Jaco80.9674.84
Overall66.5363.58

Table 10. Comparison between skill-selection methods in fine-tuning.

(4) Matched compute (new, Table 11). MIM-DICE uses the same DDPG backbone as the baselines and is evaluated in the same URLB codebase with the same number of environment interactions (Table 3). We additionally measured wall-clock time on Pendulum. For 100K training steps, MIM-DICE is on par with APS, DIAYN, CIC, SMM, and CeSD, and takes less than half the time of BeCL.

MethodEnd-to-endTraining loop
MIM-DICE20m 50s16m 19s
APS21m 14s16m 42s
DIAYN21m 02s16m 33s
CIC21m 11s16m 39s
SMM21m 08s16m 41s
CeSD21m 05s16m 34s
BeCL54m 19s49m 48s

Table 11. Wall-clock runtime for 100K Pendulum training steps.

[W4,Q3. Uneven downstream gains on Quadruped]

MIM-DICE underperforms ExDM and CeSD on Quadruped, and we do not claim that higher MI yields stronger control on every task. Two factors explain the gap.

(1) The MI objective is task-agnostic. Maximizing MI only requires skills to be distinguishable from one another, and distinguishable skills are not necessarily useful for a particular task. MIM-DICE optimizes the same objective and therefore shares this limitation.

On Quadruped, every method that scores above MIM-DICE uses an exploration-driven reward: ExDM (67.4), CeSD (60.8), SD3 (55.5), APT (50.9), CIC (50.5), and RND (47.8). CIC belongs to this group in practice, since its intrinsic reward is the entropy term H(τ)H(\tau) alone, with the contrastive term used only for representation learning. Consequently, CIC’s policy-induced MI is low in our experiments (0.13 on Pendulum, Table 1a). Among the methods whose reward depends on the skill (APS, DIAYN, BeCL, DOI), MIM-DICE is the best on Quadruped (43.8 against 31 to 33). On Jaco the order reverses (MIM-DICE 81.0, ExDM 37.0, CeSD 36.5). No method is uniformly strong across domains, which is why we report the per-task results (Table 4) together with the RLiable aggregate (Figure 3).

(2) Skill selection (new, Table R2). MIM-DICE must identify a task-relevant skill in a 64-dimensional continuous skill space from early fine-tuning interactions. As a diagnostic, we kept the pretrained models and the fine-tuning budget fixed and changed only the Quadruped selection schedule to a single regression at 5K steps.

QuadrupedDefault scheduleSingle regression at 5KBest baseline
Walk26.87 ± 5.4342.49 ± 8.8169.7 (ExDM)
Jump49.95 ± 6.6953.26 ± 5.0769.7 (CeSD)
Run29.69 ± 4.0938.54 ± 3.8646.3 (CeSD)
Stand68.61 ± 7.4767.94 ± 5.4789.2 (ExDM)
Average43.78 ± 3.2850.56 ± 2.2967.4 (ExDM)

Table R2. Quadruped scores under the default selection schedule and a single regression at 5K steps

With identical pretrained skills, the Quadruped average rises from 43.78 to 50.56, mostly on Walk and Run, so skill selection accounts for part of the gap. This diagnostic is reported separately, and Table 4 and Figure 3 keep the uniform schedule for all domains.

The Quadruped gap and the role of skill selection are discussed in the Limitation paragraph, and the selection-schedule diagnostic is reported in Appendix D.8 (Table 12).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.