acceptodds
Withdrawn from ICLR 2027

Different Intrinsic Drives, Similar Skills

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Different intrinsic drives can lead agents to learn similar skills without rewards specified for each skill. We develop HoDA (Homeostatically Driven Agent), a world-model-based agent, and compare the capabilities that agents develop with different intrinsic rewards in Crafter and two Atari games. We use achievement evaluation and behavioral statistics to examine which skills emerge across methods and how often they occur. In Crafter, intrinsic drives based on homeostasis, novelty, and predictability all support similar skills in physiological maintenance, tool use, and terrain modification. HoDA is competitive in survival and tool use and also explores broadly without novelty rewards. This study provides systematic behavioral evidence that different intrinsic drives can produce similar skills, offering a basis for understanding the relationships among intrinsic drives and further developing these shared capabilities.

Primary area:
reinforcement learning
Keywords:
Intrinsic Drives, Homeostatic Reinforcement Learning, World Models, Behavioral Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BrDnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose an intrinsic drive (HODA) based on tracking progress of a user defined set of homeostatic drives and compare this with 6 other intrinsic drives in a common dreamerv3 framework. The paper evaluates their agents primarily in Crafter, studying basic survival skills like eating, sleeping, and shelter construction. They show that all intrinsic drives basically exhibit the same behavior, with minor differences here and there.

Honestly I am not sure what the contributions are, since this paper contains very little novelty. The concept of homeostatic reinforcement learning is not new, and even the authors specific proposal of HODA is not new--it is basically identical to the drive the authors cite as previous work in Yoshida & Kuniyoshi (2025). The authors experiments are quite shallow and don't reveal any meaningful insight into how intrinsic drives differ. If the claim is that they develop skills, analysis should be done that rigorously investigates this convergence and design experiments that tease apart different intrinsic motivations. If the paper is meant to focus on HODA, none of the experiments really demonstrate a valuable use of HODA, especially since the authors do not describe the principles of homeostatic motivation and how it can be formulated in general control settings.

Critical strengths

The general idea of the role of homeostasis in intrinsically motivated behavior is an interesting question to explore.

Critical weaknesses

Overall notes

  • Intro
    • Novelty here seems to be very minimal compared to prior work, which already studies both normal intrinsic drives and homeostatic ones in crafter. “These findings motivate us to ask whether similar skills arise across different intrinsic drives and extend to tool use and terrain modification that require several consecutive steps.” This question is not substantive enough for a publication, particularly if you only study it in a single environment where these behaviors are relatively simple.
    • Results described in intro seem trivial:
      • “Similar physiological rhythms also appear under different intrinsic drives, with food levels showing similar cycles across methods. HoDA is competitive in survival and tool use and also explores broadly without an explicit novelty reward. Together, skill performance, physiological rhythms, and spatial exploration reveal behavioral connections among different intrinsic drives.”
    • These results are very weak, already in the introduction. Many papers in the past have shown that different intrinsic rewards can do similar things, particularly when they are within the same model family (knowledge-based, competency-based, or entropy-based). All of the baseline algorithms the authors chose were within the knowledge/curiosity/prediction-error based family, so this is not surprising that the algorithms behave similarly. Further, the physiological rhythms can be simply attributed to the periodic availability of the relevant achievements depending on the time of day in the environment. While more interesting explanations could be at play, the authors did not investigate them.
  • Related work
    • Homeostatic angle is kind of odd, because it is clearly a biologically motivated approach, yet not a single reference or motivation in the related work regarding homeostatic behaviors in animals. For an RL agent, basic homeostasis regulation of food/water are artificial, since they are not formulated as properties of the embodiment but of the extrinsic task. Thus, this isn’t really an interesting application of homeostasis in RL, since the authors simply use existing crafter achievements (which are extrinsic rewards in the game) as a proxy for homeostatic drives/.
  • HODA
    • HODA is formulated such that the user must choose both the homoestatic set point and the weights of different intrinsic drives. However, in all experiments, they were set to one. So, it is not clear why this formulation is necessary since there are no demonstrated use of nontrivial weights or setpoints. Also, the authors do not demonstrate the extent to which these parameters would need to be varied depending on the task, even though the relative value of different basic needs and the optimal setpoint is an obvious experiment to pursue.
    • The authors do not clearly explain how HODA differs (if at all) from Yoshida & Kuniyoshi (2025). The authors use a very similar logic of presentation, including the order and variable names, as Yoshida & Kuniyoshi (2025). Then, the authors write a value function for HODA in 4, which contributes nothing since obviously the RL agent maximizes a value function of some form, yet it as written as if this is an important part of the HODA agent. Further, its not clear how accurate this even is since HODA is introduced as a model-based dreamerV3 agent, which uses more nuanced learning objectives in addition to the actor-critic component.
  • Results
    • Overall performance
      • HODA achieving the highest crafter score is a bit misleading, since its “intrinsic” reward happens to already be crafter achievements that are directly rewarded. e.g. eating, drinking, sleeping, etc are all crafter achievements, so you are basically training on a subset of achievements long with “not dying”.
      • The Atari tasks are strange to randomly throw in. Seaquest and Ms Pac man are not explained or interpreted clearly. Further, they were only used to demonstrate the point that all agents learn similar skills, which is not a very interesting point to make especially when the environments themselves are relatively simple.
      • The score units in the table are not explained. Crafter has 22 achievements. What does 27.77 score mean?
    • Achievement profiles and tool use
      • This whole paragraph is just an extremely basic description of the figure. It offers no insight or interpretation. “This method builds furnaces. This method often defeats skeletons. This method often defeats zombies.” What is the reader supposed to gain from this?
    • Physiological maintenance and shelter construction
      • All the agents exhibit some basic periodicity in their homeostatic behaviors, but because the availability of these rewards varies periodically in the environment, this is not an interesting result. The authors are basically showing that the agents learn basic contingencies in the environment when certain things are available or not.
      • Again, all agents build enclosed sleeping shelters, including the task reward. Why is this interesting?
    • The spatial exploration
      • Experiment is not described. Is this from rollouts after training, or during training? How many rollouts?
      • Again, the authors offer no rigorous analyses as to why or how the HODA agent should explore more. They speculate extremely vaguely, but could have interpreted why according to their specific proposed reward, spatial exploration should be stronger than the compared methods.
    • Discussion and limitations
      • This paper says the commonalities and differences between methods should be used to inform future research and make use of existing drives. But this paper offered no insight as to any systematic way to understand which drive is better at what. In fact, they simply show all drives do pretty much the same thing, and do not push the analyses to understand what experiment should we design/implement to separate drives.
      • Much existing work in unsupervised skill learning or options discovery concerns learning new skills autonomously, contrary to the skills in this paper which are just ad hoc defined based on specific task properties. Yet the authors claim this is for future work, as if there paper is the first in this area.
      • The limitations the authors describe are exactly the reasons why this paper is not ready for publication. They only use crafter, hardly discuss even the two Atari games in the appendix, and provide no scientific train of experiments that demonstrate the source of the similarities and how to exploit the differences between extrinsic drives.
      • The authors provide no reason for why HODA specifically was proposed. The paper was mainly about the rather boring observation that, in crafter, many intrinsic drives do the same thing. So what? HODA is proposed here for no reason, simply to compare with the existing intrinsic drives. Fine. But the authors did not discuss why HODA should be a good drive in general, beyond crafter, and how it would even be implemented. In fact, this is likely because HODA is a trivial homeostatic reward, based on the specific “homeostatic” properties of crafter. So, to take this opportunity to define existing task variables as homeostatic, without offering an algorithmic principles for general control tasks (e.g. what are common shared homeostatic variables that would be useful to endow an agent with), is quite lazy.
Questions for authors

I have many questions for the authors, but none would materially affect my assessment. The paper does not provide a valuable contribution to the community.

Reviewer_yqCeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents a new intrinsic motivation method where rewards maximize homeostasis of physiological variables, and demonstrates the method in Crafter. Furthermore, the authors present an analysis of various intrinsic motivation methods (e.g. RND, E3B, etc.) in Crafter using a Dreamer V3 backbone, arguing that all of these methods produce similar skills.

Critical strengths
  • The presented intrinsic exploration results using Dreamer V3 are very strong (Table 1). While there is minimal experimental detail nor explanation of the choice + use of Dreamer V3, these are impressive results that show that intrinsic motivation, in a complex environment like Crafter, leads to significant task rewards.
Critical weaknesses
  • The main claim of the method Different Intrinsic Drives, Similar Skills'' rely really heavily on Crafter and Atari environments. My suspicion is that much of the overlapping behavior'' come from the restrictions in the environment structures themselves -- that survival of any form (e.g. a reward that is 1 at every time step, or -1 upon death) would give a lot of the same results. I think it's critical to show ``Different Intrinsic Drives, Similar Skills'' arising from environments where there is NOT a persistent survival pressure in, for example, open-ended manipulation tasks, or similar -- otherwise, the evidence is insufficient to support the claim.

  • The presented HoDa is not generalizable to settings where physiological states are ill or not defined. Intrinsic exploration methods should really be simple methods that can generalize across environments, else one should simply use extrinsic rewards. As such, I do not believe that HoDa is a convincing intrinsic motivation method.

Questions for authors
  • Can you show that different intrinsic motivations lead to learning similar skills in environments beyond Crafter and Atari? E.g., open-ended manipulation environments, or other open-ended settings without health bars?
  • Can HoDa be well-defined and lead to interesting behavior even in settings without explicit physiological degrees of freedom? For example, can these ``physiological'' d.o.f. be learned latents?
  • Why is Dreamer V3 a natural choice for probing intrinsically-motivated exploration?
Reviewer_y94DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether agents trained purely with different intrinsic motivation signals develop similar multi-step skills without task-specific rewards.

To compare physiological regulation alongside curiosity and predictability objectives, the paper presents HoDA (Homeostatically Driven Agent), which combines squared physiological deficit reduction with a DreamerV3 world model.

Using a shared DreamerV3 backbone, the authors compare HoDA against E3B, ICM, RND, P2E, SMiRL, and a task-reward reference in Crafter, along with supplementary experiments in Atari Seaquest and Ms. Pac-Man.

Through achievement metrics and behavioral analyses of sword combat, shelter construction, water-path building, physiological cycles, and spatial coverage, the experiments show that distinct intrinsic drives converge on many of the same survival, tool-use, and terrain-modification behaviors.

Critical strengths

The paper is very well written and extensively studies emergent multi-step behaviors beyond aggregate scores, including topological shelter enclosure, water-path construction, matched-budget coverage, and matched-state reward versus action-probability comparisons (Figure 5), showing that shared shelter building is driven by downstream return rather than immediate construction reward.

On a shared DreamerV3 backbone in Crafter, HoDA demonstrates strong survival time, tool use, shelter construction, and spatial coverage without requiring an explicit novelty bonus or per-dimension reward tuning.

Critical weaknesses

Because episode termination in Crafter cuts off future positive intrinsic rewards, eating, drinking, weapon combat, and safe shelter sleeping are all instrumental to staying alive. The paper does not include a constant survival-bonus baseline (+1 per alive step) to disentangle whether these shared skills come from the specific intrinsic drives or simply from surviving longer to collect positive rewards (nor does it isolate DreamerV3 planning versus model-free homeostatic RL).

In Atari Seaquest (Table E1), HoDA completes repeated oxygen-replenishment cycles in only 1.2% of episodes (tied with the Random baseline at 1.2%), which is glossed over in the main text and highlights a failure case of drive reduction when resource depletion can be avoided by staying passive at the surface.

Questions for authors

It would be very useful to include a DreamerV3 baseline trained in Crafter with a constant survival reward (+1 per alive step, along with a Random baseline). Does a simple +1 survival reward already produce similar eating, sword-combat, and shelter-sleeping behaviors, or do the six intrinsic drives yield meaningfully richer skills?

In Table E1 (Seaquest), why does HoDA only achieve 1.2% on repeated oxygen-diving cycles (identical to Random, whereas RND, P2E, E3B, and Empowerment exceed 94%)? Does HoDA simply stay near the surface once oxygen reaches the 0.5 setpoint so that future drive stays at zero? It would be good to discuss this failure case directly in the main text.

How does HoDA compare against a model-free homeostatic baseline (Yoshida & Kuniyoshi, 2025) or shorter DreamerV3 imagination horizons in Crafter, so we can better understand the role of world-model planning in multi-step shelter and water-path construction?

How many training seeds and evaluation episodes went into generating the Atari results in Tables E1 and E2, and do the trends remain consistent with standard deviations included? Also, is there a reason why Empowerment was included in Atari but omitted from the Crafter comparison?

I am happy with the behavioral analysis in Crafter, and adding the +1 survival baseline along with clarifying the Seaquest failure case and Atari seed statistics would further strengthen the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.