acceptodds
Under review as a conference paper at ICLR 2027

BELLMAN MEETS LYAPUNOV: UNSUPERVISED REINFORCEMENT LEARNING THROUGH MASTERING CHAOS

Abstract

Reinforcement learning (RL) is a powerful paradigm for training agents, yet its success rests on domain expertise of human engineers who design informative reward signals for every new task. Unsupervised RL aims to replace this engineering with intrinsic motivation (IM): reward signals that emerge from the agent environment interaction itself. Existing IM objectives, however, involve the selection of information variables, which re-introduces domain expertise the field has sought to eliminate. We introduce (F-CIP), an RL-native formulation of the Controllable Information Production (CIP) objective, which is defined by the system's dynamics alone and requires no such selection. We prove that F-CIP is compatible with RL and demonstrate its effectiveness with existing algorithms. Training agents with F-CIP results in unsupervised discovery of primitive behaviors such as balancing and maintaining controllability, which are essential for more complex robot behaviors. Paired with a minimal forward-velocity reward, our method produces coordinated gaits such as hopping and running which otherwise require reward engineering to learn.

Primary area:
reinforcement learning
Keywords:
Intrinsic Motivation, Unsupervised Reinforcement Learning, Robot Learning, Lyapunov Exponents, Edge of Chaos

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U4usOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper designs a method for automatically obtaining reward functions for useful behaviours such as standing or walking in Mujoco simulated robots. The main contribution of the paper is to obtain a Markovian reward such that maximizing that reward corresponds to maximizing the positive LE of open-loop dynamics; this intuitively encourages the agent to go to states where the dynamics will take the agent to further states in the future for small changes in the current state, basically areas of the environment where dynamic and active behaviours are possible. They find a forward form of the backward LE formula and show its equivalence and show how to obtain a Markovian reward that maximizes CIP. In their experiments, they first verify that the value function indeed estimates the CIP properly. They show that their method can perform well in downstream tasks and show emergent behaviours with their intrinsic reward + forward velocity reward.

Critical strengths

1- The theoretical analysis of the paper seems sound and useful. I am not a control expert, but to me it seems their contribution in converting a backward formulation into a forward formulation for CIP that allows for direct RL reward maximization is significant.

2- The problem the paper tries to address is a very critical problem in unsupervised RL: the design of rewards that lead to emergent behaviour, and their view is pretty unique in that, in most prior approaches, intrinsic reward based on novelty is used; But their work uses a very new reward which is based on the interestingness of the dynamics.

3- The paper is well-written and easy to understand for the most part (except for sec 3)

Critical weaknesses

1- The paper requires explicit access to the dynamics to compute the Jacobian matrix. This is a very big drawback, as in most real experimental setups we only get to learn the environment through interactions and not through explicit dynamic formulas. Can you show if a trained forward model of the environment would perform well?

2- The experimental setup is somewhat limited. Can you try with a humanoid or with manipulation tasks?

I will be happy to accept the paper if the above concerns are solved.

Questions for authors

Can you rewrite sec 3 of the paper to give more context to a reader who is familiar with unsupervised RL but not with control? The rest of the paper is very well-written, but I had a serious problem understanding Sec 3. Also, notations are not well defined in that section.

5 comments

Thank you, reviewer U4us, for the helpful feedback! We are glad that the paper was overall easy to read aside from section 3. We will update that section as described below. Your other comments prompted us to add two new experiments: (1) calculating the F-CIP reward and memory from a learned world model and (2) applying the method in the more complex setting of a Franka Emika Panda robotic arm. The results show that our method is not inherently limited by a differentiable simulator and that it is capable of scaling to more complex scenarios.

Critical Weaknesses:

Explicit Jacobian Access: We ran new experiments that demonstrate our method is capable of running without explicit access to the simulator’s Jacobians. We train a forward dynamics model to predict the next state from the current state and action and show that our method is still capable of self-righting on the benchmark tasks from Table 1 when Jacobians are extracted from the learned model. We report the results under the same evaluation procedure as in Table 1 of the main paper: average normalized height in the last 10% of the episode averaged over 10 seeds.

SystemF-CIP (Learned Model)
Cart pole0.9890
Double pendulum0.9990
Triple pendulum0.9913
Gibbon0.9913

More Complex Scenario: We ran a new experiment and trained our method on an object manipulation task. The agent controls a robotic arm on a table which also has a cube on it. In this setting, we restricted the reward (Equation 6) to the state of the cube and observed that it incentivises the agent to learn to move the arm to the location of the cube and to maintain contact via touching and “kicking” the cube around with the gripper. We will include images of this behavior in the camera-ready version of the paper. Quantitatively, we observe that the contacts per second are high when the agent is trained with our F-CIP object-restricted reward. For comparison, we also evaluated the behavior of a random agent and an agent trained under an oracle reward for the pick-and-place task for Franka Emika Panda in MuJoCo playground (Zakka et al., 2025).

avg gripper-cube distance (m)contacts / stime to first contact (s)
Random control0.459 ± 0.0080.007 ± 0.0035.927 ± 0.031
F-CIP0.074 ± 0.0000.232 ± 0.0220.312 ± 0.004
Oracle0.025 ± 0.0010.455 ± 0.0210.354 ± 0.007

Section 3: We agree that section 3 is not written as clearly as the rest of the paper. We will include the following edits to improve readability:

  • Retitle the section to ``Background: Controllable Information Production’’ so that it is clear that this section is background information.

  • We will formally define the Lyapunov exponents as λi=lim⁡T→∞1TΔtln⁡σi(Φ0:T)\lambda_i = \lim_{T\to\infty} \frac{1}{T \Delta t} \ln \sigma_i(\Phi_{0:T}) and to provide intuition that they measure information production/dissipation rates along dimensions of a dynamical system. We will define Φ0:T=∏t=0T−1fxt\Phi_{0:T} = \prod_{t=0}^{T-1} f_{x_t} as the product of Jacobians along the nominal trajectory.

  • We will explicitly define Λ+=∑λi>0λi\Lambda^+ = \sum_{\lambda_i > 0} \lambda_i as the sum of positive Lyapunov exponents.

  • We will explain the meaning of the stop gradient symbol, where we define the open-loop dynamics folf^\mathbf{ol}. The stop gradient exists in this definition to isolate only the open-loop Jacobian fxtf_{x_t}; otherwise, the Jacobian would be of the closed-loop system, which includes the feedback of the policy: fxt+futπxtf_{x_t} + f_{u_t} \pi_{x_t}. Since CIP is related to open-loop information production, we need to make this distinction clear.

Zakka, Kevin, et al. "MuJoCo Playground." arXiv preprint arXiv:2502.08844 (2025).

Thanks for the rebuttal and for the new experiments. The new experiments are convincing, and the fact that the method works with learned dynamics is great. The fact that the agent learns to interact with the block in an unsupervised manner is very interesting. Could you please qualitatively compare your method in the Franka robotic arm setting with methods based on mutual information skill learning such as [Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning, Zheng et al.] or [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction, Park et al.]? I understand you have compared your method with DIAYN, which is an MI maximization method in the paper, but the referenced papers are usually significantly better than DIAYN in exploration. I think comparing your method with those baselines rather than the random policy would provide a much better way to evaluate the method. One clarifying question: if you only use the object position as the state for the FRANKA arm, then how does it learn to find the object in the first place? Because it won't receive any reward before seeing the object, right? Could you share a video showing the agent's behavior alongside how the F-CIP reward is changing? Thanks

Thank you for your response! We are glad you found the new experiments to be convincing and interesting. We were also excited to see that it's possible to use the F-CIP reward in a more complicated scenario and that it incentivises primitive “touching” behavior.

Based on your suggestion we ran two mutual-information skill-learning methods on the same task: SEE (Ji et al.) and MISL (Zheng et al.). SEE has finished training. MISL takes considerably longer to train, so the table shows its latest checkpoint (72M of 148M steps); we will update it once training completes.

avg gripper-cube distance (m)contacts / stime to first contact (s)
Random control0.459 ± 0.0080.007 ± 0.0035.927 ± 0.031
F-CIP0.074 ± 0.0000.232 ± 0.0220.312 ± 0.004
Oracle0.025 ± 0.0010.455 ± 0.0210.354 ± 0.007
SEE (potential, 148M steps)0.910 ± 0.0110.000 ± 0.0006.000 ± 0.000
SEE (best skill, 148M steps)0.822 ± 0.0070.000 ± 0.0006.000 ± 0.000
MISL (random skill, 72M steps)0.874 ± 0.0150.000 ± 0.0006.000 ± 0.000
MISL (best skill, 72M steps)0.542 ± 0.0070.000 ± 0.0006.000 ± 0.000
MISL (random skill, 148M steps)0.823 ± 0.0130.008 ± 0.0085.986 ± 0.014
MISL (best skill, 148M steps)0.505 ± 0.0050.023 ± 0.0115.922 ± 0.030
(mean ± standard error over 256 episodes of 6.0s; time to first contact is 6.0 when never touched)

For each baseline we report its default behavior (SEE’s potential policy, MISL with randomly sampled skills) and its single best skill, selected by contact rate. Neither baseline learns to interact with the cube. Both methods are rewarded for learning skills that are distinguishable from each other so arm motion alone can satisfy this. On average their skills stay farther from the cube than random control.

You are right that the F-CIP reward is zero when the arm is away from the cube (see videos in supplementary material). We used a similar reset curriculum to the one used in our gibbon environment, where 50% of the time the arm is reset near the cube so it has a higher chance of grazing against it and observing the reward. Without resetting near the cube the task is a hard exploration problem, since pure random control has a low likelihood of encountering the cube by chance alone. F-CIP, SEE, and MISL are trained with the reset curriculum, and all methods are evaluated with the arm started away from the cube.

We have uploaded some videos of the F-CIP driven cube touching behavior and the corresponding reward over time during the episode to the supplementary materials. The reward is zero when the arm is away from the cube and becomes large during interaction. To view the videos please download the zip file from openreview and unzip it. Inside there are 5 videos, each showing a different evaluation episode.

Did you have any additional questions / concerns regarding the paper? If not, would you be open to increasing your score? Thank you!

Reviewer_AWDpOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces Forward CIP (F-CIP), a formulation of the Controllable Information Production (CIP) objective and shows it is compatible with RL optimization. The authors use F-CIP with existing algorithms to train agents leading to unsupervised discovery of primitive behaviors such as balancing and maintaining controllability. The paper shows among others, how walking or hopping behaviors can emerge when training agents with F-CIP and a forward velocity reward.

Critical strengths

The idea to replace the control-side backward recursion with its estimation-side dual forward-in-time calculation of CIP to enable training with RL is very interesting.

Critical weaknesses

Missing baselines. The paper discusses the most relevant baselines, such as MPC-CIP, SuPLE and Empowerment. However comparison against those baselines are missing.

Missing ablations. While F-CIP seems promising as presented, no ablations of its components are included.

Contradicting statements. Section 5 (Q2) states that all agents start from the hanging pose. Appendix A.5 states that gibbon resets are split evenly between hanging and the vicinity of the upright stand.

Questions for authors

Why did you choose the baselines you did? How does you approach compare to relevant approaches such as MPC-CIP, SuPL and Empowerment?

What is the contribution of each component of your approach?

Is it correct that for the most complex environment of Q2 (the gibbon, also shown in figure 4), half of the episodes start near the target? If yes, how does this affect your claims?

4 comments

Thank you, Reviewer AWDp for your comments on our paper! Your review prompted us to add an additional baseline (Empowerment) and to ablate an important property of our method (the agent’s memory).

Critical Weaknesses and Questions:

Missing Baselines: We have added an additional comparison to a recent scalable implementation of empowerment from the paper “Scaling Empowered Agents” (Ji et al., 2026) called Skills Estimate Empowerment (SEE). We find that this baseline is still unable to perform unsupervised-self righting on our benchmark task suite. We evaluate SEE with its “potential” form and where we choose the skill with the highest average torso height, and in neither case is the algorithm able to perform self-righting.

TaskF-CIP (PPO)SEE (potential)SEE (best skill)
Cart pole0.99840.00000.0710
Double pendulum0.99890.00000.0389
Triple pendulum0.99250.17930.1653
Gibbon0.98840.40610.2196

Regarding the MPC-CIP and SuPLE baselines. We chose not to include MPC-CIP in the comparison since it is a purely sampling-based approach and not an RL learning-based approach like ours. However, MPC-CIP was shown to work on the same self-righting benchmark tasks in the original paper (Shah and Tiomkin, 2026). We also did not include SuPLE since it is not as scalable as the other approaches, as it involves pre-calculating passive (zero-control), finite-horizon Lyapunov Exponents over a discretization of the state space.

Missing Ablations:

We agree that the paper did not include sufficient ablations. In response, we ablated the core component of our method: the agent's memory. In both columns, the reward is still computed from the full matrix; only the agent's input changes. The "diag" column passes only the diagonal of the matrix to the agent, and "none" passes no memory. The diagonal memory observation alone matches the full matrix (Table 1) on CartPole, double pendulum, and gibbon, and degrades on the triple pendulum. Removing memory entirely fails on three of the four systems, which supports Proposition 3.

SystemMemory: diagMemory: none
Cart pole1.0000.000
Double pendulum0.9990.007
Triple pendulum0.6340.031
Gibbon0.9880.987

Contradicting Statements:

This was an oversight on our part. During training, the gibbon resets 50% of the time near the upright pose and the rest of the resets occur near the bottom. We will update the main text to correct this.

Ji, C., Myers, V., Lin, J., Wingate, A., Levine, S., & Eysenbach, B. (2026). Scaling Empowered Agents. arXiv preprint arXiv:2610.13002.

Tristan Shah and Stas Tiomkin. Emergence of physical intelligence via controllable information production. In Advances in Neural Information Processing Systems, volume 39, Main Conference, 2026.

Did our response adequately address your concerns? Since our last message we have also added a Franka arm manipulation task with further baseline comparisons (SEE and MISL), which you can see in our discussion with Reviewer U4us. If your concerns are resolved, would you please consider raising your score? We are happy to answer any remaining questions before the discussion period ends. Thank you!

Thank you for the additional effort that addressed many of my concerns. I have also read your responses to the other reviewers and I find that the new evidence you presented during the rebuttal have made your paper significantly stronger. I really appreciate the experiments with the Franka Emika Panda arm. As a result, I raise my grade to 3.

Thank you for increasing your score and supporting our work! And thank you once again for your helpful suggestions in this rebuttal period. The additional baselines and ablations you asked for make the paper much stronger. We are also glad you found the Franka arm experiments valuable.

Reviewer_T8bvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Forward CIP, a forward-in-time formulation of Controllable Information Production for reinforcement learning. It derives incremental rewards using dynamics Jacobians and an augmented information-matrix state.

Critical strengths

The forward decomposition of CIP provides a useful connection between dynamical systems theory and reinforcement learning. The experiments demonstrate interesting emergent behaviors without explicitly rewarding upright posture.

Critical weaknesses
  1. The main contribution is a reformulation of an existing CIP objective rather than a new intrinsic motivation principle. The novelty lies primarily in making CIP compatible with standard RL optimization.
  2. The theoretical equivalence is established for infinite-horizon average rewards. The experiments optimize finite-horizon discounted returns with PPO, and the effect of this approximation is not sufficiently analyzed.
  3. The method requires analytical dynamics Jacobians at every environment step. This substantially restricts its applicability to environments with accessible differentiable dynamics.
  4. The information matrix requires quadratic memory in the state dimension.
  5. The proposed reward admits undesirable solutions. The authors report that agents can maximize CIP through uncontrolled spinning or thrashing rather than learning useful behaviors.
  6. Additional joint damping is required to remove these undesirable solutions. This environment modification weakens the claim that the method avoids domain-specific engineering.
  7. The method is not evaluated without the additional damping in the final benchmark comparisons.
  8. The gibbon reset protocol requires clarification. Section 5 states that agents start from the hanging configuration, whereas Appendix A.5 specifies that half of the gibbon resets occur near the upright pose. This affects the interpretation of unsupervised self-righting.
Questions for authors
  1. Can the authors report results without additional joint damping and quantify the frequency of degenerate behaviors?
  2. Why do the locomotion baselines discard their intrinsic objectives during fine-tuning while F-CIP retains its intrinsic reward?
  3. Does the gibbon experiment always start from the hanging configuration, or are some episodes initialized near upright states?
  4. How does F-CIP compare with the original CIP-based MPC approach in sample efficiency and wall-clock cost?
  5. How sensitive is performance to Jacobian estimation errors and the information-matrix dimension?
  6. Can the authors provide experiments on more complex locomotion tasks or with learned dynamics?
3 comments

Thank you, Reviewer T8bv, for your helpful comments on our paper. As a result of your comments, we have added several new experiments.

Critical Weaknesses:

Novelty: We acknowledge that our primary contributions are (1) a reformulation of the original CIP quantity and (2) empirical demonstration that it can be optimized with RL for unsupervised pretraining and as an auxiliary term during fine-tuning. Our work constitutes a significant knowledge advancement since the previous formulation of CIP was structurally incompatible with RL, and our work resolves this issue.

Finite horizon: We agree that the equivalence between CIP and the sum of positive Lyapunov exponents of the dynamical system is established for infinite horizons. However, even at a finite horizon, the sum of rewards still has an interpretation as the log determinant of the state covariance matrix in Equation 3. So at a finite horizon the agent optimizes a notion of “open-loop volume growth” (Theorem 4). Empirically, we optimize a lightly discounted objective using γ=0.999\gamma = 0.999, which corresponds to a long horizon estimate.

Analytical Jacobians: We ran new experiments that demonstrate that our method works for unsupervised self-righting with a learned dynamics model rather than true Jacobians from the simulator. These experiments show that our method does not inherently require a differentiable simulator.

Please see our response to reviewer U4us titled Explicit Jacobian Access.

Quadratic Memory Requirement: We ran a new ablation on what the agent observes of the information matrix. In both columns, the reward is still computed from the full matrix; only the agent's input changes. "diag" passes only the diagonal of the matrix to the agent, and "none" passes no memory. The full-matrix results are in Table 1 of the paper.

Please see our response to reviewer AWDp titled Missing Ablations.

The diagonal alone matches the full matrix on cart pole, double pendulum, and gibbon, and degrades slightly on the triple pendulum. Removing memory entirely fails on three of the four systems (gibbon is the exception), which supports Proposition 3: in general, the agent needs the memory for the reward to be Markovian. With the diagonal variant, each environment still propagates one full matrix to compute the reward, but the agent's input and each stored transition are O(n) rather than O(n^2).

Undesirable Solutions: We agree, and state this as a limitation in Section 6: CIP has maximizers such as free spinning that are not useful behaviors. They are avoided via joint damping, which is a physical parameter that is present in real robotic joints. All baselines are trained and evaluated in the same damped environments. A principled objective-level fix, for example penalizing closed-loop information production, is future work.

Evaluation Without Damping: Training without added damping yields the degenerate maximizers described in Appendix A.4, so we do not report those as benchmark results. In a new experiment, we evaluate hopper and walker policies trained with damping at lower damping values and count the seeds that remain upright.

Leg-joint dampingWalkerHopper
029/3226/32
0.128/3228/32
0.230/3232/32
0.331/3232/32
0.4 to 1.032/3232/32

Gibbon Reset Protocol:

This was an oversight in writing. We will correct the text to clearly indicate that the Gibbon environment was trained with a curriculum where 50% of the time it is reset near the upright pose. During evaluation, the Gibbon resets only from the bottom, so it must perform a complete swing up and stabilization near the top. Finding the upright pose is a difficult exploration problem, and the reset curriculum assists in that exploration. In the simpler environments, the agents are all trained with resetting to the downright pose only.

Questions:

1.) Please see Evaluation Without Damping above

2.) The locomotion baselines discard their intrinsic objective during fine-tuning because they broadly incentivise exploration. The CIP objective targets dynamically unstable configurations such as the upright position, which enables it to be trained in conjunction with a forward velocity reward

3.) Please see Gibbon Reset Protocol above

4.) At evaluation time, MPC-CIP must plan, so its cost is approximately O(HS), where H is the planning horizon and S is the number of “shots”. An RL policy produces an action in a single forward pass, so its cost is approximately O(1)

5.) The learned dynamics model is an imperfect neural network approximation, yet it still drives self-righting in all four systems, so our method is not critically sensitive to Jacobian estimation errors

6.) We added an experiment on the Franka Emika Panda arm with a cube. Restricting the CIP reward (Equation 6) to the state of the cube incentivises the arm to reach out and touch it. Please see More Complex Scenario in our response to reviewer U4us

Did our response adequately address your concerns? We have also added new baseline comparisons on the Franka task in our discussion with Reviewer U4us. If your concerns are resolved, would you please consider raising your score? We are happy to answer any remaining questions before the discussion period ends. Thank you!

Since the rebuttal period is closing soon we wanted to follow up to make sure we don't miss the opportunity to hear your feedback on our new results. Since our last correspondence we have shown that:

  • Our method can work without access to analytical Jacobians from a simulator
  • Our method can be applied on a simulated Franka robotic arm where it produces “object touching” behavior (see videos attached in the supplementary material)
  • We added additional ablations to highlight the importance of the agent’s recurrent memory
  • We added new baselines: Empowerment (Ji et al.) on the self-righting tasks and Franka, and MISL (Zheng et al.) on Franka.

If you have any additional suggestions on improving our work we would greatly appreciate your thoughts. Thank you!

Comments on the paper

1 comment

We thank all of the reviewers for their participation in this discussion so far. We have received very helpful feedback to improve the manuscript and would like to summarize our responses to the main concerns.

Analytical Jacobians: We have included a new experiment to show that our method can still succeed in unsupervised self-righting using Jacobians from a learned dynamics model. This addresses the concern that our method is only limited to differentiable simulators. Please see our response to reviewers T8bv and U4us.

More complex tasks: We have included an experiment to show that our system is capable of training a Franka Emika Panda arm to reach out and touch/manipulate a cube. Please see our response to reviewers T8bv and U4us.

Additional ablations: We ablated an important component of our agent: its memory. Passing only the diagonal of the memory matrix to the agent matches the full matrix on three of the four self-righting systems, while removing memory entirely fails on three of four. Please see our responses to reviewers T8bv and AWDp.

Additional baselines: We added a recent scalable empowerment baseline (Ji et al., 2026) and evaluated it on the self-righting benchmark tasks. It was unable to perform unsupervised self-righting on any of the four systems. Please see our response to reviewer AWDp.

The comments by the reviewers have been very helpful in improving our work. We look forward to continued discussion through the remainder of the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.