- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The authors propose an intrinsic drive (HODA) based on tracking progress of a user defined set of homeostatic drives and compare this with 6 other intrinsic drives in a common dreamerv3 framework. The paper evaluates their agents primarily in Crafter, studying basic survival skills like eating, sleeping, and shelter construction. They show that all intrinsic drives basically exhibit the same behavior, with minor differences here and there.
Honestly I am not sure what the contributions are, since this paper contains very little novelty. The concept of homeostatic reinforcement learning is not new, and even the authors specific proposal of HODA is not new--it is basically identical to the drive the authors cite as previous work in Yoshida & Kuniyoshi (2025). The authors experiments are quite shallow and don't reveal any meaningful insight into how intrinsic drives differ. If the claim is that they develop skills, analysis should be done that rigorously investigates this convergence and design experiments that tease apart different intrinsic motivations. If the paper is meant to focus on HODA, none of the experiments really demonstrate a valuable use of HODA, especially since the authors do not describe the principles of homeostatic motivation and how it can be formulated in general control settings.
Critical strengths
The general idea of the role of homeostasis in intrinsically motivated behavior is an interesting question to explore.
Critical weaknesses
Overall notes
- Intro
- Novelty here seems to be very minimal compared to prior work, which already studies both normal intrinsic drives and homeostatic ones in crafter. “These findings motivate us to ask whether similar skills arise across different intrinsic drives and extend to tool use and terrain modification that require several consecutive steps.” This question is not substantive enough for a publication, particularly if you only study it in a single environment where these behaviors are relatively simple.
- Results described in intro seem trivial:
- “Similar physiological rhythms also appear under different intrinsic drives, with food levels showing similar cycles across methods. HoDA is competitive in survival and tool use and also explores broadly without an explicit novelty reward. Together, skill performance, physiological rhythms, and spatial exploration reveal behavioral connections among different intrinsic drives.”
- These results are very weak, already in the introduction. Many papers in the past have shown that different intrinsic rewards can do similar things, particularly when they are within the same model family (knowledge-based, competency-based, or entropy-based). All of the baseline algorithms the authors chose were within the knowledge/curiosity/prediction-error based family, so this is not surprising that the algorithms behave similarly. Further, the physiological rhythms can be simply attributed to the periodic availability of the relevant achievements depending on the time of day in the environment. While more interesting explanations could be at play, the authors did not investigate them.
- Related work
- Homeostatic angle is kind of odd, because it is clearly a biologically motivated approach, yet not a single reference or motivation in the related work regarding homeostatic behaviors in animals. For an RL agent, basic homeostasis regulation of food/water are artificial, since they are not formulated as properties of the embodiment but of the extrinsic task. Thus, this isn’t really an interesting application of homeostasis in RL, since the authors simply use existing crafter achievements (which are extrinsic rewards in the game) as a proxy for homeostatic drives/.
- HODA
- HODA is formulated such that the user must choose both the homoestatic set point and the weights of different intrinsic drives. However, in all experiments, they were set to one. So, it is not clear why this formulation is necessary since there are no demonstrated use of nontrivial weights or setpoints. Also, the authors do not demonstrate the extent to which these parameters would need to be varied depending on the task, even though the relative value of different basic needs and the optimal setpoint is an obvious experiment to pursue.
- The authors do not clearly explain how HODA differs (if at all) from Yoshida & Kuniyoshi (2025). The authors use a very similar logic of presentation, including the order and variable names, as Yoshida & Kuniyoshi (2025). Then, the authors write a value function for HODA in 4, which contributes nothing since obviously the RL agent maximizes a value function of some form, yet it as written as if this is an important part of the HODA agent. Further, its not clear how accurate this even is since HODA is introduced as a model-based dreamerV3 agent, which uses more nuanced learning objectives in addition to the actor-critic component.
- Results
- Overall performance
- HODA achieving the highest crafter score is a bit misleading, since its “intrinsic” reward happens to already be crafter achievements that are directly rewarded. e.g. eating, drinking, sleeping, etc are all crafter achievements, so you are basically training on a subset of achievements long with “not dying”.
- The Atari tasks are strange to randomly throw in. Seaquest and Ms Pac man are not explained or interpreted clearly. Further, they were only used to demonstrate the point that all agents learn similar skills, which is not a very interesting point to make especially when the environments themselves are relatively simple.
- The score units in the table are not explained. Crafter has 22 achievements. What does 27.77 score mean?
- Achievement profiles and tool use
- This whole paragraph is just an extremely basic description of the figure. It offers no insight or interpretation. “This method builds furnaces. This method often defeats skeletons. This method often defeats zombies.” What is the reader supposed to gain from this?
- Physiological maintenance and shelter construction
- All the agents exhibit some basic periodicity in their homeostatic behaviors, but because the availability of these rewards varies periodically in the environment, this is not an interesting result. The authors are basically showing that the agents learn basic contingencies in the environment when certain things are available or not.
- Again, all agents build enclosed sleeping shelters, including the task reward. Why is this interesting?
- The spatial exploration
- Experiment is not described. Is this from rollouts after training, or during training? How many rollouts?
- Again, the authors offer no rigorous analyses as to why or how the HODA agent should explore more. They speculate extremely vaguely, but could have interpreted why according to their specific proposed reward, spatial exploration should be stronger than the compared methods.
- Discussion and limitations
- This paper says the commonalities and differences between methods should be used to inform future research and make use of existing drives. But this paper offered no insight as to any systematic way to understand which drive is better at what. In fact, they simply show all drives do pretty much the same thing, and do not push the analyses to understand what experiment should we design/implement to separate drives.
- Much existing work in unsupervised skill learning or options discovery concerns learning new skills autonomously, contrary to the skills in this paper which are just ad hoc defined based on specific task properties. Yet the authors claim this is for future work, as if there paper is the first in this area.
- The limitations the authors describe are exactly the reasons why this paper is not ready for publication. They only use crafter, hardly discuss even the two Atari games in the appendix, and provide no scientific train of experiments that demonstrate the source of the similarities and how to exploit the differences between extrinsic drives.
- The authors provide no reason for why HODA specifically was proposed. The paper was mainly about the rather boring observation that, in crafter, many intrinsic drives do the same thing. So what? HODA is proposed here for no reason, simply to compare with the existing intrinsic drives. Fine. But the authors did not discuss why HODA should be a good drive in general, beyond crafter, and how it would even be implemented. In fact, this is likely because HODA is a trivial homeostatic reward, based on the specific “homeostatic” properties of crafter. So, to take this opportunity to define existing task variables as homeostatic, without offering an algorithmic principles for general control tasks (e.g. what are common shared homeostatic variables that would be useful to endow an agent with), is quite lazy.
- Overall performance
Questions for authors
I have many questions for the authors, but none would materially affect my assessment. The paper does not provide a valuable contribution to the community.