- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper designs a method for automatically obtaining reward functions for useful behaviours such as standing or walking in Mujoco simulated robots. The main contribution of the paper is to obtain a Markovian reward such that maximizing that reward corresponds to maximizing the positive LE of open-loop dynamics; this intuitively encourages the agent to go to states where the dynamics will take the agent to further states in the future for small changes in the current state, basically areas of the environment where dynamic and active behaviours are possible. They find a forward form of the backward LE formula and show its equivalence and show how to obtain a Markovian reward that maximizes CIP. In their experiments, they first verify that the value function indeed estimates the CIP properly. They show that their method can perform well in downstream tasks and show emergent behaviours with their intrinsic reward + forward velocity reward.
Critical strengths
1- The theoretical analysis of the paper seems sound and useful. I am not a control expert, but to me it seems their contribution in converting a backward formulation into a forward formulation for CIP that allows for direct RL reward maximization is significant.
2- The problem the paper tries to address is a very critical problem in unsupervised RL: the design of rewards that lead to emergent behaviour, and their view is pretty unique in that, in most prior approaches, intrinsic reward based on novelty is used; But their work uses a very new reward which is based on the interestingness of the dynamics.
3- The paper is well-written and easy to understand for the most part (except for sec 3)
Critical weaknesses
1- The paper requires explicit access to the dynamics to compute the Jacobian matrix. This is a very big drawback, as in most real experimental setups we only get to learn the environment through interactions and not through explicit dynamic formulas. Can you show if a trained forward model of the environment would perform well?
2- The experimental setup is somewhat limited. Can you try with a humanoid or with manipulation tasks?
I will be happy to accept the paper if the above concerns are solved.
Questions for authors
Can you rewrite sec 3 of the paper to give more context to a reader who is familiar with unsupervised RL but not with control? The rest of the paper is very well-written, but I had a serious problem understanding Sec 3. Also, notations are not well defined in that section.
5 comments
Thanks for the rebuttal and for the new experiments. The new experiments are convincing, and the fact that the method works with learned dynamics is great. The fact that the agent learns to interact with the block in an unsupervised manner is very interesting. Could you please qualitatively compare your method in the Franka robotic arm setting with methods based on mutual information skill learning such as [Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning, Zheng et al.] or [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction, Park et al.]? I understand you have compared your method with DIAYN, which is an MI maximization method in the paper, but the referenced papers are usually significantly better than DIAYN in exploration. I think comparing your method with those baselines rather than the random policy would provide a much better way to evaluate the method. One clarifying question: if you only use the object position as the state for the FRANKA arm, then how does it learn to find the object in the first place? Because it won't receive any reward before seeing the object, right? Could you share a video showing the agent's behavior alongside how the F-CIP reward is changing? Thanks
Thank you for your response! We are glad you found the new experiments to be convincing and interesting. We were also excited to see that it's possible to use the F-CIP reward in a more complicated scenario and that it incentivises primitive “touching” behavior.
Based on your suggestion we ran two mutual-information skill-learning methods on the same task: SEE (Ji et al.) and MISL (Zheng et al.). SEE has finished training. MISL takes considerably longer to train, so the table shows its latest checkpoint (72M of 148M steps); we will update it once training completes.
| avg gripper-cube distance (m) | contacts / s | time to first contact (s) | |
|---|---|---|---|
| Random control | 0.459 ± 0.008 | 0.007 ± 0.003 | 5.927 ± 0.031 |
| F-CIP | 0.074 ± 0.000 | 0.232 ± 0.022 | 0.312 ± 0.004 |
| Oracle | 0.025 ± 0.001 | 0.455 ± 0.021 | 0.354 ± 0.007 |
| SEE (potential, 148M steps) | 0.910 ± 0.011 | 0.000 ± 0.000 | 6.000 ± 0.000 |
| SEE (best skill, 148M steps) | 0.822 ± 0.007 | 0.000 ± 0.000 | 6.000 ± 0.000 |
| MISL (random skill, 72M steps) | 0.874 ± 0.015 | 0.000 ± 0.000 | 6.000 ± 0.000 |
| MISL (best skill, 72M steps) | 0.542 ± 0.007 | 0.000 ± 0.000 | 6.000 ± 0.000 |
| MISL (random skill, 148M steps) | 0.823 ± 0.013 | 0.008 ± 0.008 | 5.986 ± 0.014 |
| MISL (best skill, 148M steps) | 0.505 ± 0.005 | 0.023 ± 0.011 | 5.922 ± 0.030 |
| (mean ± standard error over 256 episodes of 6.0s; time to first contact is 6.0 when never touched) |
For each baseline we report its default behavior (SEE’s potential policy, MISL with randomly sampled skills) and its single best skill, selected by contact rate. Neither baseline learns to interact with the cube. Both methods are rewarded for learning skills that are distinguishable from each other so arm motion alone can satisfy this. On average their skills stay farther from the cube than random control.
You are right that the F-CIP reward is zero when the arm is away from the cube (see videos in supplementary material). We used a similar reset curriculum to the one used in our gibbon environment, where 50% of the time the arm is reset near the cube so it has a higher chance of grazing against it and observing the reward. Without resetting near the cube the task is a hard exploration problem, since pure random control has a low likelihood of encountering the cube by chance alone. F-CIP, SEE, and MISL are trained with the reset curriculum, and all methods are evaluated with the arm started away from the cube.
We have uploaded some videos of the F-CIP driven cube touching behavior and the corresponding reward over time during the episode to the supplementary materials. The reward is zero when the arm is away from the cube and becomes large during interaction. To view the videos please download the zip file from openreview and unzip it. Inside there are 5 videos, each showing a different evaluation episode.
Did you have any additional questions / concerns regarding the paper? If not, would you be open to increasing your score? Thank you!
Thank you, reviewer U4us, for the helpful feedback! We are glad that the paper was overall easy to read aside from section 3. We will update that section as described below. Your other comments prompted us to add two new experiments: (1) calculating the F-CIP reward and memory from a learned world model and (2) applying the method in the more complex setting of a Franka Emika Panda robotic arm. The results show that our method is not inherently limited by a differentiable simulator and that it is capable of scaling to more complex scenarios.
Critical Weaknesses:
Explicit Jacobian Access: We ran new experiments that demonstrate our method is capable of running without explicit access to the simulator’s Jacobians. We train a forward dynamics model to predict the next state from the current state and action and show that our method is still capable of self-righting on the benchmark tasks from Table 1 when Jacobians are extracted from the learned model. We report the results under the same evaluation procedure as in Table 1 of the main paper: average normalized height in the last 10% of the episode averaged over 10 seeds.
More Complex Scenario: We ran a new experiment and trained our method on an object manipulation task. The agent controls a robotic arm on a table which also has a cube on it. In this setting, we restricted the reward (Equation 6) to the state of the cube and observed that it incentivises the agent to learn to move the arm to the location of the cube and to maintain contact via touching and “kicking” the cube around with the gripper. We will include images of this behavior in the camera-ready version of the paper. Quantitatively, we observe that the contacts per second are high when the agent is trained with our F-CIP object-restricted reward. For comparison, we also evaluated the behavior of a random agent and an agent trained under an oracle reward for the pick-and-place task for Franka Emika Panda in MuJoCo playground (Zakka et al., 2025).
Section 3: We agree that section 3 is not written as clearly as the rest of the paper. We will include the following edits to improve readability:
Retitle the section to ``Background: Controllable Information Production’’ so that it is clear that this section is background information.
We will formally define the Lyapunov exponents as λi=limT→∞TΔt1lnσi(Φ0:T) and to provide intuition that they measure information production/dissipation rates along dimensions of a dynamical system. We will define Φ0:T=∏t=0T−1fxt as the product of Jacobians along the nominal trajectory.
We will explicitly define Λ+=∑λi>0λi as the sum of positive Lyapunov exponents.
We will explain the meaning of the stop gradient symbol, where we define the open-loop dynamics fol. The stop gradient exists in this definition to isolate only the open-loop Jacobian fxt; otherwise, the Jacobian would be of the closed-loop system, which includes the feedback of the policy: fxt+futπxt. Since CIP is related to open-loop information production, we need to make this distinction clear.
Zakka, Kevin, et al. "MuJoCo Playground." arXiv preprint arXiv:2502.08844 (2025).