- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MIM-DICE, a method for unsupervised skill discovery, It formulates skill learning as MI maximization over skill-conditioned occupancy measures, regularized to remain close to the replay distribution. To handle the nonconcavity of the objective, it introduces an auxiliary surrogates and a minorization–maximization procedure, then solved using DICE-style dualization. Experiments evaluate policy-induced mutual information in tabular MDPs and Pendulum, offline skill discovery in MAZE, and downstream performance on URLB.
Critical strengths
- The MM and DICE derivations form a coherent approach to optimizing skill-conditioned occupancies under flow constraints.
- The evaluation spans tabular settings, continuous control task, offline RL, and downstream adaptation.
Critical weaknesses
The main concern about this paper is that the proposed motivation is not fully clear. I find the occupancy-based formulation and its derivation internally coherent: optimizing a DICE-style objective with respect to the current policy’s occupancy is a reasonable way to address the stated optimization problem. My concern is why replay–policy mismatch motivates a DICE-based solution specifically? This mismatch is common in off-policy RL and may also affect the paper’s MI-based baselines, including CIC, BeCL, and CeSD. But they doesn't training badly. The paper compares against these methods, but I wonder why the mismatch is particularly consequential for them or explain why the proposed correction is developed only within a DICE framework.
Furthermore, assuming the mismatch does affect the performance of policy improvement in URL, the experiments do not isolate the benefit of occupancy correction. The reported comparisons cannot determine whether MIM-DICE’s gains come from correcting replay–policy mismatch only, not its DICE objective, or implementation differences. A direct ablation study with a vanilla DICE could help. This would test whether the gains are specifically attributable to the proposed correction and help this paper more trustworthy.
Questions for authors
-
Question: The skills trajectories seem not distinguish enough on the visualization of 2D Maze (table 1)? What about compared with DIAYN and DADS?
-
typo: Appendix C.1 seems contains an unresolved equation reference: “Substituting Eq. equation ?? gives”?
-
typo: "an policy" on 324 line
4 comments
Following the reviewer’s question, we additionally evaluated DIAYN and DADS on MAZE (new, Table 5, details in Appendix B.4).
| Data Coverage | MIM-DICE (Ours) | APS | CIC | DIAYN | DADS |
|---|---|---|---|---|---|
| 50% | 2.14 ± 0.29 | 0.43 ± 0.07 | 0.10 ± 0.02 | 0.05 ± 0.01 | 0.05 ± 0.01 |
| 70% | 2.09 ± 0.16 | 0.30 ± 0.08 | 0.07 ± 0.01 | 0.04 ± 0.01 | 0.06 ± 0.01 |
| 100% | 2.10 ± 0.19 | 0.39 ± 0.06 | 0.06 ± 0.01 | 0.04 ± 0.01 | 0.03 ± 0.00 |
Table 5. Offline MAZE results across dataset-coverage levels.
Among the evaluated methods, MIM-DICE is the only method that effectively increases policy-induced MI. We have also revised Appendix G to include MAZE visualizations for DIAYN and DADS (Figures 4–6).
[Q2 & Q3. Typos]
Thank you for pointing these out. We fixed the unresolved reference in Appendix C.1 and the typo “an policy”.
Thank you for the response and efforts in addressing my questions and concerns. I will maintain my current rating.
Thank you for following up on our response. Your comments were very helpful in improving the paper, and we are glad we had the chance to address them.
[W1: Why the replay-policy mismatch motivates a DICE-based correction]
(1) Why the mismatch matters more for MIM than for standard off-policy RL. In standard off-policy RL, the reward is a fixed function given by the environment and does not depend on the occupancy of the policy, so the mismatch between the policy and the data affects only how values are estimated. In mutual information maximization (MIM), the reward itself is defined by the state occupancies of the current policy, whereas existing off-policy MIM methods compute it with the occupancies of the replay buffer:
Replacing rπ with rB changes the objective itself, and Section 3.1 gives the exact gap.
(2) Why DICE. MI is the expectation of rπ under dz, but rπ itself depends on the policy occupancies, so the objective is a nonlinear function of the occupancies dz(s,a). DICE optimizes directly over occupancies under the Bellman flow constraints, so this dependence is handled exactly. Moreover, with the ratio wz(s,a)=dz(s,a)/dB(s,a) and Lagrangian duality, the problem involves only expectations under the data distribution dB (Eq. (15)), so it can be estimated from off-policy samples without knowing the behavior policy. Other off-policy corrections do not provide both properties. Value-based methods assume a reward that does not change with the policy, and importance sampling requires the behavior policy and has variance that grows exponentially with the horizon (curse of horizon).
(3) Why the mismatch is consequential for CIC, BeCL, and CeSD. These methods perform well on downstream tasks, but their policy-induced MI is low when trained with off-policy data. On Pendulum they reach 0.13, 0.37, and 0.02, against 0.94 for MIM-DICE (Table 1a). On tabular MDPs, training with the replay-based reward yields lower policy-induced MI than training with the true reward, and the replay-buffer MI differs substantially from the policy-induced MI (Figure 1); all prior methods also remain below MIM-DICE in policy-induced MI (Figure 2a). The mismatch therefore limits how well they maximize MI, even when their downstream scores are high.
[W2. Isolating the benefit of occupancy correction with a vanilla DICE ablation]
Following the reviewer’s suggestion, we ran a direct ablation with vanilla DICE (new, Table 2). It shows that the gains of MIM-DICE come from the occupancy correction, and not from the DICE objective or from implementation differences.
We applied vanilla DICE (OptiDICE) to APS, DIAYN, and BeCL while keeping their original intrinsic rewards. These rewards are computed from the replay buffer, like rB in W1, and not from the occupancies of the current policy as in the true reward rπ.
Table 2 (new). Policy-induced MI on Pendulum for the original methods and their intrinsic-reward DICE variants.
The variants are optimized with DICE, and their MI does not improve over the original methods, so the DICE objective alone does not explain the gains. They also use the same policy extraction and implementation as MIM-DICE, and MIM-DICE still outperforms them under the same divergence and coefficient (Table 7), which rules out implementation differences. The only remaining difference is occupancy correction. The variants optimize rewards computed from the replay buffer, while MIM-DICE corrects the occupancy through the ratio wz(s,a)=dB(s,a)dz(s,a) and optimizes the policy-induced MI itself.
[Q1. Skill distinguishability on MAZE and comparison with DIAYN and DADS]
In Figure 2c, the color of each point encodes the two-dimensional continuous skill (z0,z1) through the color map shown at the top left. When all skills visit the same states, the colors mix into a single muddy color, as in the BUFFER column and in the baselines. When different skills occupy different regions, the regions show distinct colors, as in the MIM-DICE column. Since the skills are continuous, the color changes gradually between neighboring regions, which is expected and does not mean that the skills are not distinguishable.