acceptodds
Withdrawn from ICLR 2027

Emergent foraging strategies and functional neural motifs in homeostatic Deep RL agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Adaptive behavior in natural environments depends on a dynamic coupling between an organism’s internal physiological state and its ongoing motor decisions. However, the computational mechanisms underlying the integration of interoceptive signals with motor control in embodied agents remain poorly understood. We train embodied artificial agents on a continuous two-resource foraging task using a homeostatic reinforcement-learning (HRL) objective, in which the agent minimizes deviation of its internal state from a set point rather than maximizing extrinsic reward. The resulting agents develop state-dependent search strategies that parallel biological foraging principles. Both exploration extent and movement velocity scale with homeostatic state, a dynamic analogous to hunger-modulated behavior. Clustering population activity around foraging events, such as food picking, reveals recurring temporal motifs: pre-event, peri-event, post-event, ramping, and sustained. Systematic cluster-level ablation demonstrates that these motifs contribute distinctly to behavior; for instance, ablating ramping clusters yields hyperactive, excessive-exploration policies. To assess the importance of learning under physiological constraints, we also train agents with an identical neural architecture, body, and environment, but using conventional reinforcement learning (RL). On the behavioral side, the RL agents lack the general state-dependent foraging strategies; and on the neural side, most of the functional neural motifs found in HRL agents become less prevalent or disappear. These results provide a neuroethologically grounded computational framework that links behavior to neural circuit dynamics and embodied constraints.

Primary area:
applications to neuroscience & cognitive science
Keywords:
homeostatic reinforcement learning, embodied agents, NeuroAI, foraging behavior, neural population dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_gaHGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper shows how homeostatic objectives shape behavioral patterns and corresponding neural organization in reinforcement learning agents by comparing homeostatic RL agents with conventional agents. The authors use PPO for training and conduct experiments in a MuJoCo environment, where the agents' locomotor activity can be measured. The authors design an internal state regulation task based on food consumption to test whether homeostatic RL agents change their behavioral patterns according to their internal state dynamics. They measure how the agents' locomotor activity changes and how actively they search for food under different levels of food deprivation, ranging from mild to full deprivation. The authors further investigate neural motifs around these behavioral events to examine whether changes in homeostatic states affect neural activity, and conduct ablation experiments to test the distinct functional roles of these neural patterns.

Critical strengths

The paper's key insight is to show how homeostatic objectives can influence the agent's behavioral strategies. The authors provide a clear comparison between conventional RL and homeostatic RL agents in terms of how their behavior differs across levels of food deprivation. This perspective extends conventional RL formulations by examining how behavior changes with the agent's internal needs and physiological conditions.

Critical weaknesses

The study uses a single algorithm (PPO) and a single environment to test the agents' homeostatic behavior. This limits the generalizability of the main claims, although the comparison between homeostatic and conventional RL agents helps isolate the effect of the optimization objective. It remains questionable whether this HRL behavior is limited to a specific environment and training setup or whether it can be observed with other RL algorithms, such as SAC, or in more complex 3D embodied environments. Also, in terms of generalizability, the behavioral analysis focuses only on food intake, which provides a limited interpretation on how homeostatic objectives shape the general adaptive behavior. Evaluating additional conditions, such as water deprivation and competing physiological needs, would provide more robust evidence that the learned policies exhibit flexible homeostatic regulation.

Questions for authors

Q1. The authors analyzed hidden-unit activity around behavioral events and identified several neural motifs to examine how homeostatic objectives influence neural activity. However, it is unclear what these motifs represent and how they relate to the agents' physiological states. For example, neural activity during food-seeking behavior may also reflect sensory feedback or motor-related activity. Without distinguishing these possibilities, it is difficult to determine whether the identified motifs reflect functional organization related to homeostatic regulation. What evidence supports this interpretation, and what additional insights do the neural findings provide beyond the behavioral results?

Q2. In the study, only the PPO algorithm is used to test the agents, which makes it unclear whether the behavioral results only occur under a specific algorithm or would be generalizable across different algorithms, such as SAC or Dreamer (a model-based approach). In a similar context, it is also questionable why the authors only used food intake as a factor for homeostatic regulation to analyze the agents' behavioral patterns. Homeostatic regulation involves multiple physiological needs, such as hydration and energy balance, while survival in dynamic environments may require responding to external threats, such as the appearance of enemies. It would be more reliable if the authors examined whether similar behavioral patterns emerge under other forms of physiological deprivation or in response to dynamic environmental challenges.

Reviewer_xUS8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study investigated the effect of internal states of behaving agents using homeostatic reinforcement learning and reported state-dependent policy and latent dynamics. The study also showed the absence of these findings when the agent is trained without the internal reward as control.

Critical strengths

I think the interest in the interoceptive aspect of agent is important and I find the framework of homeostatic RL interesting for foraging.

Critical weaknesses

But I do not fully understand the specific design and analysis the authors used, and I am uncertain about the validity of the conclusions.

I suspect the internal state follows exponential decay dynamics (see question below), which will explain why movement is discouraged at the deprived state since the agent does not need to do anything to be rewarded. However, it makes the observation less surprising and the conclusion less relevant to a real agent in my opinion. I would suggest the authors explain more about the internal state dynamics and why it is a good model to study, for readers who are not familiar with homeostatic RL.

The second design that puzzles me is the two-resource environment. It seems to me all the analysis this study showed can be achieved with just one resource. I did not recall any important conclusion about the interaction between two resources such as trade-off or pareto optimality. In addition, many figures look at the state from “fully deprived” to “fully sated”, but which resource does it refer to, or the two resources are always set at the same level? I hope the authors can add some discussions to justify the use of two-resource setup.

The control of conventional RL seems valid, but I have doubts about its significance. Since the internal states are purely irrelevant in training these agents (Eq. 1), it is no surprise that there is no internal state dependency. I am not sure if this leads to any non-trivial insights, and even if so, it might not be worth much space used in the manuscript.

Lastly, I feel it is a little overreach to claim the existence of clusters in Fig 3a, they look more like a continual spectrum to me. The ablation results could use more explanation. I find Fig 4a hard to understand.

Overall, I like the question and the general framework, but I don’t understand the method and feel skeptical about the conclusions of this study.

Questions for authors

My main question is about the internal state dynamics. On line 137-138, “both internal variables decay at a constant metabolic rate” is puzzling to me. Does it mean h values decay to 0 exponentially (e.g., h(t+1) = (1-alpha)*h(t)) or they are consumed at fixed rate (e.g., h(t+1) = h(t)-alpha)? If it is the former case, does it mean there is an intrinsic drive reward when h<0, which is counterintuitive to me since it implies the internal reservoir gets replenished automatically? If it is the latter case, what happens when h values go extremely low? Will the agents be terminated?

Reviewer_vmMhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors train embodied agents to forage for 2 resources while regulating their internal state. Agents trained with a homeostatic reward show state-dependent movement and feeding, incl reduced activity under severe deprivation. The paper groups neural activity around feeding events and ablates those groups to investigate their behavioral roles. Its main contribution is connecting foraging behavior to recurring neural activity patterns and testing how these differ under a conventional food-intake reward. Evaluation includes 10 trained networks/reward condition, repeated rollouts, random-neuron ablations, interoceptive-input perturbations, and supplementary analyses across environmental layouts and network layers.

Critical strengths

An interesting contribution is the connection between state-dependent foraging and neural interventions. Perturbing the interoceptive inputs and ablating groups of hidden units provides a concrete way to investigate how the agent uses its internal state to guide behavior.

The main experiments include 10 independently trained networks, supplemented by tests across environmental layouts and in the second hidden layer. These are useful because they give the behavioral findings a stronger empirical foundation.

Critical weaknesses
  1. The ablation plots do not establish the claimed directions of behavioral change. Section 3.3 describes increases and decreases relative to the intact agent, but Appendix J states that zero in Figure 4d represents the average grouped ablation effect. An effect above that average could still mean reduced food intake relative to the intact policy. Raw paired changes are needed to support claims such as increased consumption after ramping-cluster ablation. This affects the paper’s central interpretation of the motifs’ behavioral roles.
  2. The interpretation of severe-deprivation inactivity as adaptive torpor is not supported. In Appendix I, agents fail to approach available food and reach the survival boundary; placing food within reach prolongs survival. This establishes a limitation in food approach, but does not demonstrate that inactivity conserves resources or improves survival. Moreover, evaluation continues beyond the boundary that terminates training (Appendix C). The low-activity behavior is interesting, but its adaptive value is not unclear.
Questions for authors
  1. Can you report the raw ablation-minus-intact changes underlying Figure 4d, with uncertainty across independently trained networks? Do these support the increases and decreases described in §3.3?
  2. What in the current results supports interpreting inactivity as adaptive resource conservation? Does that interpretation hold when analysis is restricted to behavior before the survival boundary is crossed?
  3. Which meta-clustering rule produced the reported 15 HRL groups versus zero conventional-RL groups? Appendix E describes both 2/3 and 3/3 agreement, while Appendix F specifies 3/3. Clarify the implemented rule and whether this choice changes the conclusion.
Reviewer_hY9EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the behavior of RL agents when presented with internal homeostatic signals. The RL agent can explore a navigation environment with food in it, and it observes its internal state, which represents its current levels of two resources associated with food types 1 and 2. As time goes on, these internal resources decrease, and the agent's goal is to maintain them close to a certain level. It receives a reward for reducing the deviation from this target level.

The main contribution is that they show that agents trained this way develop behaviors that resemble those of biological agents. For instance, if their resource levels are moderately low, they actively search for food, but if they are extremely low, they move less and explore only locally, which the authors interpret as energy conservation. They also observe that agents trained this way develop specialized groups of neurons in the policy that show different activation patterns. For instance, some are ramping, meaning their activity increases leading up to food collection, while others are pre-event, meaning they show elevated activity before food collection. They also perform causal interventions by zeroing out these groups of neurons and show that doing so has different effects on the agent's behavior.

Critical strengths

The paper studies the question of how agents use internal signals to explore and interact with the environment. This is a complicated and important problem, as many different factors in practice will change this behaviour.

Their figures are nice and easy to read.

Critical weaknesses

While the paper's results are nice, I don't think its contribution is significant enough to be accepted as an ICLR conference paper. The agent's behaviour similarity to biological agents isn't particularly surprising because the reward function is set such that doing the intended behaviour is optimal for maximizing reward. The clustering of neurons and causal intervention experiments aren't particularly surprising either; I can imagine that in any RL environment, different neurons are responsible for different actions and fire at different times, and those fires, of course, have different patterns around reward collection states. I am not convinced why the result of this paper is significant enough for a conference paper; I believe that by setting up the right environment, similar results could be obtained in different environments. Moreover, the model of internal state is too simplistic

Questions for authors

Could you show that your results are biologically significant by showing some emperical connection with biological data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.