acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Abstract

Animals have a remarkable ability to act autonomously in the absence of external sensory cues or extrinsic objectives, unlike most frontier artificial agents today. The computational mechanisms behind this ability largely remain a mystery. We leverage recent whole-brain recordings in the fruit fly (Drosophila melanogaster), in which neural activity predicts spontaneous turns more than seconds in advance, to test computational substrate of future volitional action. Specifically, we study whether intrinsically motivated artificial agents embodied in a biomechanically realistic fly-body can develop behavioral and internal representations quantitatively similar to those observed in biological flies during exploration. Our policies operate over a discrete set of high-level motor primitives while low-level central pattern generators actuate joint dynamics, closely matching current ethological theories of behavioral syllables. We train agents with no external reward using intrinsic motivation algorithms spanning curiosity, entropy, skill discovery, and empowerment methods. We then evaluate frozen policies in a virtual tethered-ball assay modeled after the fruit-fly experiment, comparing the extent to which spontaneous locomotor statistics and internal policy representations predict future turn direction on the same timescales observed in the fly brain. Using these criteria, we find operating hierarchically over motor primitives enables DIAYN—a skill-discovery algorithm that learns a discrete set of policies to maximize both the entropy of behavior and the separability in sensory observations those behaviors induce—to achieve the strongest behavioral and neural alignment with the fruit-fly. Notably, the set of policies DIAYN discovers closely matches the wide individual variability observed in fly exploration strategies, thus suggesting this behavioral diversity optimizes a concrete computational goal. Together, these results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.

Primary area:
applications to neuroscience & cognitive science
Keywords:
intrinsic motivation, unsupervised reinforcement learning, autonomous behavior, exploration, computational neuroscience, embodied agents, motor planning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7sTdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper is an RL/computational neuroscience paper that uses recently-developed biomechanically-realistic simulators to investigate normative models of animal behavior via intrinsic motivation objectives. The method compares several intrinsic motivation methods, including the standard RND alongside newer methods like C-TeC, mutual information skill learning methods, and empowerment-maximization methods, and compares the learned representations and behavior against actual representations and behavior in fly recording data. The paper finds signficant alignment between the learned representations and behaviors of skill learning method DIAYN and real behavior + representation data with a variety of probes (e.g. turning bias, PCA of internal representations) for these signatures.

Critical strengths

I think there are critical strengths to the paper, that make it a very valuable contribution to the field of biologically-plausible skill learning.

  • I think it is fantastic to utilize these biomechanically realistic simulators with intrinsic reward to provide, quote, a normative model of animal behavior via skill learning. This is a new research push using these simulators, that pure behavioral cloning-like approaches (or even using purely extrinsic rewards) cannot capture. The results themselves are very interesting and compelling, particularly Fig. 5 which is very interesting.

  • Furthermore, the comparison to actual neural + behavioral signatures in fly data is very welcome, necessary for any clear contribution to the computational neuroscience community (beyond simply the community in RL that studies intrinsic motivations). This is a big strength that sets this paper apart from a lot of ML conferencepapers that try to make biological conclusions from purely synthetic data.

Critical weaknesses

That being said, there are some critical (though very addressable in the review period) weaknesses that I am concerned about that, if resolved, would move me to Accept.

  • The definitions of the empowerment methods are vague in the paper (with no clear prior reference/the authors develop and augment prior methods). While having public access to the code should help with this, I think it is important to concisely conceptually and mathematically describe every implemented intrinsic somewhere in the main-text. As it is, the current writeup of these methods treatment of empowerment seems ad hoc.

I think being careful about introducing and discussing empowerment is extremely important, especially given that the predecessor of VIC by overlapping set of authors (see VIM by Mohamed and Rezende) optimize empowerment, VIC mentions empowerment but is categorized under skill learning, and VIC itself is a precursor of DIAYN. In other words, there exists a close connection between skill learning objectives and empowerment objectives, and being very clear which is which (without expecting readers to automatically understand the difference between optimizing channel capacities vs. directly optimizing mutual informations) is critical.

  • Furthermore, there should be more pause in explaining/describing the actual plotted metrics (Figs. 3-5) and the underlying methods. ``Internal representations'' (I think I understand what is used here, but need clarification) can mean multiple things -- for instance, these representations can literally refer to internal representations within a neural network. And the way these representations are used is different between e.g. prediction error methods like RND, skill learning methods like DIAYN, etc., so an apples-to-apples comparison is a little strange/not immediately obvious. Note that the math ϕt\phi_t is not written in the main text at all, even though parameterizations of intrinsic reward methods in terms of ϕt\phi_t is referred to in the figures.

  • Again along these lines, there needs to be more main text description of what it means to run DIAYN on low level primitives. I do not really know what this means without digging into the appendix, and I think these types of modifications to underlying methods has to be made crystal clear.

  • Minor nitpick, but I think it is important to reference prior works in neuroscience literature that look at the representations of motor skill learning (there is not much but still some). E.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC12439889/ which I think has some neat connections with the author's work.

Questions for authors
  • What are your modifications to DIAYN + empowerment methods used in the paper?

  • Would you be able to provide an anonymous GitHub link to the codebase repository as stated in your reproducibility statement? This would significantly help me parse the choices for the various skill learning and intrinsic motivation methods.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7sTd

We thank the reviewer for their time in evaluating our paper and providing thorough, constructive feedback. We have incorporated their suggestions into our revisions, and address each and every of the reviewer's concerns and questions below.

Response to Weaknesses

W1: We have clarified the definitions and implementations of DIAYN, Empowerment and the other intrinsic objectives in the main text (Section 2.2 and 3.2) and Appendix A.4.

Prior work has discussed at length the connections between empowerment and DIAYN such as [1] and our motivation in this work is not to re-iterate these ideas but to evaluate these objectives in our context for representing biological and neural. We believe it is outside the scope of this work to exhaustively explain the evolution and connection between empowerment and skill-learning methods, especially since our results rely only on understanding DIAYN, which can be intuitively understood as a skill-discovery algorithm independently from channel-capacity empowerment.

[1] C. Ji, V. Myers, S. Levine, and B. Eysenbach, “The Geometry of Empowerment,” arXiv preprint arXiv:2610.07796, 2026. [Online]. Available: https://arxiv.org/abs/2610.07796

W2: All compared objectives use the same recurrent policy architecture. Consequently, the internal representations analyzed in our neural comparison refer consistently to network activations in the policy, not to the auxiliary networks specific to each intrinsic objective. This provides a consistent basis for comparing the effects of different intrinsic training objectives (Section 4; Appendix A.3 and A.10). To make the metrics more clear, we replaced Neural and Behavioral Alignment with Neural and Behavioral prospective decoding.

We also clarify the plotted metrics:

  • Figure 3: Neural and behavioral prospective decoding measure balanced accuracy in predicting future turn direction from actor/fly neural activity and preceding locomotor velocity, respectively.
  • Figure 4: Turning bias is measured as log(N_R/N_L), where N_R and N_L are the numbers of right and left turns.
  • Figure 5: Functional similarity measures the relationships of model activations and fly IPS activity to angular and forward velocity, including the squared Pearson correlation-based heatmap comparison.

These definitions and the meaning of the encoded representation φ_t are now stated explicitly in the corresponding sections and appendices (Sections 3.2–4; Appendices A.3, A.9–A.11).

W3: Our implementation uses DIAYN as a high-level, skill-conditioned policy over seven pre-defined low-level motor primitives, rather than learning continuous joint-level actions directly.

Specifically, a latent skill identifies a recurrent policy conditioned on that skill. At each 100 ms control step, the policy selects a motor primitive based on proprioceptive and other available observations together with its recurrent state. The skill code remains fixed over the evaluation episode, but the motor-primitive selection remains dynamic. DIAYN therefore learns distinguishable, state-dependent strategies for combining existing motor primitives. The skill is an identifier of the strategy assigned through learning, not a prescribed motor sequence.

We have expanded Sections 3.1–3.2 and Appendix A.4 to make this hierarchical implementation more explicit.

W4: We thank the reviewer for their suggestion here, and have incorporated literature by Amematsro et al. (2025), along with a related work, Braun et al. (2024), into the discussion of motor subskills and neural representations (Section 2.1).

Response to Quesitons

Q1: What are your modifications to DIAYN + empowerment methods used in the paper?

A1: We do not make any modifications to the DIAYN algorithm, or any other intrinsic motivation, in this paper beyond their novel application in a hierarchical neuromechanical model of drosophila. For DIAYN, we optimize with PPO instead of the original SAC, but this is just an optimizer choice and not a change to the intrinsic reward itself. We realize these ideas were not clear in the original submission because an explanation of DIAYN and other algorithms was missing. However, we have amended this in sections 2.2 and 3.2 and believe it should now be clear how DIAYN is used in our paper.

Q2: Would you be able to provide an anonymous GitHub link to the codebase repository as stated in your reproducibility statement? This would significantly help me parse the choices for the various skill learning and intrinsic motivation methods.

A2: The anonymized source code was included in the supplementary materials as a .zip archive, and we apologize if this was not clear. For easier access, we additionally provide the repository at: https://anonymous.4open.science/r/vigilant-memory-583D/

Reviewer_7sTdOpenReview ↗

Thank you for the clarification

Thank you for the clarification on all these points, which have sufficiently answered my questions.

I have now raised my score to 3 and am advocating for a weak accept.

If you could please put your paper revisions in a different color and upload, I think it would be easier for all of us to see what revisions you've made.

We thank the reviewer for reconsidering their score and for their support.

As requested, we have uploaded a version of the manuscript with revisions highlighted in a different color as part of the supplementary materials to make the changes easier to identify.

Reviewer_T89qOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether intrinsic motivation can explain spontaneous fly behavior and internal activity predictive of future turns. It trains recurrent policies with 20 objectives in a biomechanical fly simulation, using 7 predefined motor primitives. Frozen agents are evaluated in a dark, odor-free tethered-ball assay and compared with recordings from nine flies. The evaluation measures turning-bias variability, prediction of turn direction from activity and behavior 8-15s beforehand, and relationships between low-dimensional activity and locomotor velocity, with random-policy and extrinsic-reward controls. The authors identify DIAYN as the strongest joint match and propose skill discovery as a computational account of both behavioral diversity and prospective internal activity.

Critical strengths

The paper compares 20 intrinsic objectives using a common body, action repertoire, and policy architecture, providing a useful setting for evaluating candidate objectives against fly behavior. The comparisons with CSF and Random-DIAYN under the same skill-holding schedule also help establish that DIAYN’s broad turning-bias distribution is not an automatic consequence of holding a skill constant. These controls support the observed learned behavior, although its biological interpretation requires further validation.

Critical weaknesses
  1. The prospective-decoding result does not distinguish information about an upcoming turn from persistent turning preferences. DIAYN receives one skill fixed throughout each 30-minute rollout, while cross-validation splits individual turns from pooled rollouts (App A.6, A.10). A representation of skill identity can therefore predict direction through that skill’s turning bias, with the same skills represented in training and test. Thus, matching the turning-bias distribution and obtaining early-window decoding may partly measure the same underlying statistic. The sustained-strategy hypothesis in 5.1 is plausible, but the analysis does not establish predictive information beyond a static skill/handedness baseline or its time dependence before a turn.

  2. The comparison changes skill persistence at precisely the timescale being evaluated. DIAYN and CSF retain a skill for 30 minutes, whereas VIC/VALOR resample every 5s and DADS/rDADS every 10s (Appendix A.6). For those competitors, the skill governing a future turn can change after the 15 to 8s window; DIAYN’s remains fixed. This changes future predictability independently of the intrinsic objective. The same-duration CSF and Random-DIAYN controls are useful, but do not resolve comparisons with the shorter-duration methods. Each objective is also trained only once, so the reported fold-level uncertainty does not establish that the method ranking is reproducible across training runs.

Questions for authors

After accounting for each skill’s or fly’s persistent turning bias, how much decoding remains in the -15 to -8 s window under temporally blocked validation? Does this additional predictability increase as the turn approaches, compared with much earlier windows?

Does DIAYN’s advantage persist when the skill-discovery methods are evaluated with the same skill-holding duration? VIC/VALOR and DADS/rDADS can resample their skill between the observation window and the future turn, whereas DIAYN retains its skill throughout the rollout. The same-duration CSF control is useful, but does not resolve this difference for the other methods.

Table 8 reports 79.1% decoding accuracy from DIAYN’s proprioceptive observations, compared with 82.1% from its actor LSTM. Does the internal state improve prediction beyond those observation features when evaluated on the same held-out data? This would clarify how much the reported signal adds beyond available sensory information. Also, why is NGU’s 67.2% observation-decoding result included as “neural alignment” in the main comparison?

2 comments
AuthorsOpenReview ↗

Response to Reviewer T89q

We thank the reviewer for their time in evaluating our paper and providing excellent feedback. We have worked to carefully address each of your concerns in the revisions to improve the quality of our paper. Below we address these identified weaknesses and questions.

W1: While a known turning bias allows predicting future turns, this is a bit tautological. The turning bias is an average statistic, which by definition allows you to predict a future turn (on average), assuming it is within distribution. However, this does not predict the moment-to-moment turns, precisely because its an average, or explain where the turning bias comes from. Our study is concerned with discovering what objective explains how an individual turning-bias that characterizes fly exploration can emerges from dynamic, moment-to-moment decisions. The skill itself just identifies a policy. As such, it is the policy itself that drives the behavior, not the skill. This behavior is dynamic through its dependence on proprioceptive feedback and the recurrent state at every time-step.

W2: We would first like to clarify that the 30 minute skill horizon was a experimental choice, matching the duration of the biological recordings and the biological correspondence of skills with the individual flies reaffirms that flies themselves encode persistent behavior strategies rather than switch behavioral strategies during a trial. We have added to our appendix to directly address this weakness. We re-evaluated all of the skill learning objectives with the same matching skill horizons: 5 seconds, 10 seconds and 30 minutes. VIC, DADS, and rDADS did not generate sufficient turns in both directions for defined decoding scores. DIAYN retains the highest defined prospective decoding score at every matched horizon. Please refer to Appendix B.1 and Table 9 for these results. Therefore the original decoding difference cannot be attributed DIAYN's longer skill horizon. Please see our response to Q2 for more discussion on the skill horizon.

Regarding seeds, while each objective was trained with one initialization, each of the 2048 training environments were independently seeded with procedurally generated obstacle and odor configurations (Section 3.2; Appendix A.5). Thus each policy was trained across a diverse data stream, which we believe is more relevant than network seeding. In addition, our evaluations were performed by sampling the policy rather than greedy/deterministic evaluation (Appendix A.6).

Q1: After accounting for each skill’s or fly’s persistent turning bias, how much decoding remains in the -15 to -8 s window under temporally blocked validation? Does this additional predictability increase as the turn approaches, compared with much earlier windows?

A1: As discussed in W1 (as well as our response to Reviewer 9f72 W2 and Q2 if the reviewer would like more discussion), a skill does not prescribe a turning bias or future action. Skills are sampled from a uniform prior, and their associated behavioral strategies emerge through optimization of the intrinsic reward. Turning bias is measured after learning as an aggregate property of each policy (skill) rather than a specified a priori.

Importantly, predicting turn direction from skill identity simply exploits the learned policy’s average turning bias, and reflects precisely how a skill is defined as an aggregate cluster of similar behaviors/turning statistics. As the reviewer suggests, this may account for some prospective decoding accuracy; however, the skill conditioned policy is recurrent and state-dependent so it’s internal activity provides additional information about actions beyond this aggregate bias. To quantify this distinction, we performed temporally blocked decoding using three features: (1) one-hot skill/fly identity, (2) actor-LSTM/IPS activity, and (3) their combination. We evaluated accuracy across multiple temporal windows preceding a turn (Appendix B.4, table 11). Again, we do this analysis for completeness for the reviewer; however, the result is unsurprising as it is tautological. While average prediction is carried by the skill identity itself (which is expected since skills are learned to identify average behavior), prediction immediately before and during the turn rely on the information carried by the network activity.

...(response to questions continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer Questions (continued)

Q2: Does DIAYN’s advantage persist when the skill-discovery methods are evaluated with the same skill-holding duration? VIC/VALOR and DADS/rDADS can resample their skill between the observation window and the future turn, whereas DIAYN retains its skill throughout the rollout. The same-duration CSF control is useful, but does not resolve this difference for the other methods.

A2: We have responded to a similar question for reviewer 9f72, we reiterate the same response here for your convenience. The reviewer may also be interested in discussion for reviewer 9f72 Weakness 2.

DIAYN samples a skill from a uniform prior and conditions its policy on that skill for a particular choice of time horizon. We choose a time horizon of 30-minutes to match the biological recording duration, and find empirically this provides the best correspondence with the data (Section 3.3; Appendix A.6). This is not a limitation of our setup, this is deliberate hypothesis testing. The result that persisting a skill for the 30-minute trial best explains the data confirms the empirical fact that flies do not switch behavioral strategies within a trial, but rather maintain a single strategy.

However, we agree that a comparison for all skill discovery algorithms over the same horizon is needed. Thus, we re-evaluate the skill learning IDMs with the 3 evaluated skill horizons: 5 seconds, 10 seconds, 30 minutes (Appendix B.1, Table 9). The results of the ablation show that DIAYN, CSF and VALOR prospective decoding improves as the skill horizon lengthens and VIC, DADS and rDADS behavior stays the same (no turns or only turning in one direction).

These results are consistent with fly’s maintaining a persistent strategy over the course of the trial. Thus, the remaining performance differences at 30-minutes result from DIAYNs intrinsic reward formulation itself, not the the skill horizon.

Q3: Table 8 reports 79.1% decoding accuracy from DIAYN’s proprioceptive observations, compared with 82.1% from its actor LSTM. Does the internal state improve prediction beyond those observation features when evaluated on the same held-out data? This would clarify how much the reported signal adds beyond available sensory information. Also, why is NGU’s 67.2% observation-decoding result included as “neural alignment” in the main comparison?

A3: We performed the requested matched train-test split experiment as requested. DIAYN’s Actor LSTM achieved 81.7% balanced accuracy, compared with 79.4% from the raw observations. Thus, the actor LSTM provides a 2.3 percentage-point improvement over the raw observations. However, we note that this property is also true of biological flies (Figure 3a; 3c). Thus, we seek an algorithm that captures the relationship between these properties, shown in the ratio test of Figure 3d.

We also thank the reviewer for identifying the NGU reporting error. The submitted version incorrectly included NGU's 67.2% raw-observation decoding in the neural comparison. The revision now reports 65.4% decoding from NGU's internal representation, while the 67.2% raw-observation result is presented separately in the appendix table.

Reviewer_9f72OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether intrinsic motivation can provide a computational explanation for spontaneous and future-oriented behavior in Drosophila. The authors train biomechanically embodied fly agents driven by recurrent neural networks with 20 intrinsic-reward algorithms. The trained agents were evaluated in a simulated tethered-ball task and comapred against published fly whole-brain recordings. Among the methods, DIAYN skill discovery was found to offer the closest match to several aspects of fly data, including the encoding of future-turns, cross-individual diversity of turning bias, and low-dimensional activity related to forward and angular velocity. The authors conclude that hierarchical skill discovery provides a plausible explanation of fly autonomous behavior and neural states.

Critical strengths
  1. This study leverages the NeuroMechFlyv2 model together with recorded motor primitives to address an interesting biological question, namely whether exploration driven by intrinsic reward can lead to the behavioral diversity and neural dynamics seen in real flies. This sets the study apart from many that just trains a biomechanically-constrained model to perform motor tasks.
  2. The study implemented a wide range of intrinsic reward algorithms and compared them at both the behavioral and neural representation levels.
  3. Random policy and task-trained navigational agents serve as useful controls that the biomechanics and the recurrent policy alone cannot produce the reported behavioral diversity and future-encoding neural dynamics.
Critical weaknesses
  1. Lack of biological support how the discrete behavioral modes are learned and sampled at test time. DIAYIN assumes a discrete set of latent codes already exists, and an intrinsic motive drives the model to learn diversified mapping between the latent codes and the behavioral variables. It is unclear what the biological basis for this design comes from. At the test time, a single code is chosen at random and held fixed without regard to the changing sensory environment. This may not reflect the biological reality of the fly sensorimotor circuit.
  2. The prospective encoding of future turns found in the DIAYIN model may be an artifact of pre-selected latent code. Since each latent code is mapped to a policy of skills, and a single code is held fixed throughout the task, it is likely that the model encodes a persistant turning bias, rather than a dynamically varying turning probability or internal "plan" for turning.
  3. Limited relevance of DIAYIN as an exploration algorithm. The study compares DIAYIN against many well-known exploration algorithms, yet does not assess how DIAYIN motivates exploration, promotes representation learning or facilitates adaptation to the task. It is thus unclear why a fly would prioritize policy distinguishability over other intrinsic motives such as curiosity and information gain.
Questions for authors
  1. What is the biological basis for assuming a discrete set of latent skill codes?
  2. Why is one code sampled at random and held fixed for an entire 30 min test rollout? Would the key behavioral and neural observations still hold if the the code is selected in a context or state-dependent manner, or allowed to switch probabilistically?
  3. Can the authors train the models with more than one random seeds to confirm whether DIAYIN has a robust performance gain over other methods?
  4. What justifies DIAYIN as an exploration algorithm or a computational account of biological intrinsic motivation?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 9f72

We thank the reviewer for their time in evaluating our paper and providing thorough, constructive feedback. We have incorporated their suggestions into our revisions, and address each and every of the reviewer's concerns and questions below.

Response to Weaknesses

W1: Our model does not assume biological flies possess discrete latent skill codes. Rather the discrete prior is inherited from the original formulation of DIAYN, and simply concerns whether policy trajectories are low-dimensional enough to admit a discrete cluster representation rather than a continuous one. The biological hypothesis does not concern any assumptions about the skill distribution, but rather the behavioral strategies that emerge from DIAYN’s intrinsic reward, which also is independent of any assumptions about the skill distribution.

Importantly, the model/policies do not implement a fixed sequence of actions or a biologically specific sensorimotor circuit. Each skill selects a recurrent policy that samples from seven low-level motor primitives based on ongoing sensory observations and the recurrent state (Section 3.1). Thus a fixed skill does not imply fixed behavior as proprioceptive feedback and the recurrent state continuously influences the action selection. A fixed skill simply means selecting a particular behavioral strategy, but that behavioral strategy is dynamic in time through its dependence on its sensory stream and latent state.

Empirically, we find that each behavioral strategy identified by a particular skill is analogous to the individual exploratory strategy of each fly within the population. (Figure 4; Section 5). We do not claim the skills themselves correspond to biological variables.

W2: We would like to clarify that a fixed skill is not to be conflated with a fixed action policy. A skill is simply an identifier for a policy, which is a stochastic distribution dynamic in time through its dependence on ongoing proprioceptive feedback and the recurrent state. For DIAYN, the policy is specifically optimized to maximize entropy, conditioned on the current proprioceptive observation and RNN hidden state. Thus, it is not a static action-rule that simply picks a turn with a fixed probability. Further, turns are not discrete events. They are identified by continuous movements with a particular angular velocity which requires sustained effort, matching the experimental protocol in biological flies.

Furthermore, turning biases are not specified before training by a skill. The skills are sampled uniformly from a categorical distribution. It has no interpretation or inductive bias a-priori, rather it only gains interpretation after learning. DIAYN learns to correspond specific behavioral strategies to these arbitrary sampled numbers, where the correspondence is determined by how distinguishable different strategies are. A characteristic turning bias over a 30 minute evaluation window is thus fully emergent from a learned policy. The turning bias is an aggregate behavioral statistic that averages moment-to-moment turns over an entire trial, it is not a constant-probability rule that is imposed on or emerges from the policy/agent.

For example, if we calculate the turning bias over 5 minutes windows instead of the full 30 minute trial, we can the distribution of turns dynamically changes over time for both DIAYN agents and the biological flies themselves (Appendix B.4, Figure 6).

W3: We agree that the original version did not have a clear explanation of DIAYN, and added both a general description of intrinsic motivation families in section 2.2, as well as specific explanation of DIAYN in Section 3.2. DIAYN is a well-established unsupervised exploration and skill-discovery objective [1]. It combines two complementary incentives: maximizing action entropy within skill-conditioned policies and encouraging distinguishable state visitation across skills. In our setting, these correspond to exploratory variability within an individual modeled strategy and behavioral diversity across the population of learned strategies.

Our contribution is to evaluate whether this objective reproduces biological signatures of exploration under a common biomechanical model. Our results allow for the interpretation that the individual fly maximizes entropy and we find this empirically over curiosity and information gain. Whereas from the data, we see that maximizing distinguishability at a population level matches the fly data.

We would like to clarify that we do not propose DIAYN as a biological mechanism that flies are known to implement, nor is the objective of this work to study DIAYN as a representation learning algorithm or its adaptation to downstream tasks. Rather we study whether reinforcement learning from the intrinsic reward set by DIAYN (and other intrinsic motivations) produces the neurological and behavioral markers seen in the data of real flies.

..(response to question continued in next comment)

AuthorsOpenReview ↗

Response to Reviewer Questions

Q1: What is the biological basis for assuming a discrete set of latent skill codes?

A1: Please to our response to W1 for more discussion. We have made revisions to the manuscript to add more explanation/discussion on our interpretation.

We do not pre-suppose that flies implement skill codes a-priori. We utilize discrete skills as it is the formulation of DIAYN, but other skill-discovery algorithms use continuous skills instead. Our interpretation of our empirical finding that discrete skill algorithm, DIAYN, is the best fit is simply that the diversity in fly exploration strategies does not require a continuous, fine-grained representation, since our behavioral outputs are low-dimensional (angular and forward velocity). If we were instead using kinematic trajectories over all the animal’s joints, a continuous mapping for skills could be preferable. However, we point out that the assumptions of the skill prior, whether discrete or continuous and what distribution skills are sampled from, is independent from the intrinsic reward itself. That is the skill prior and the intrinsic reward serve distinct roles. The prior determines how the skills are sampled; while the intrinsic reward determine what behaviors the skill-conditioned policies learn. All of our biological interpretation concerns the intrinsic reward, not the skill prior.

Q2: Why is one code sampled at random and held fixed for an entire 30 min test rollout? Would the key behavioral and neural observations still hold if the the code is selected in a context or state-dependent manner, or allowed to switch probabilistically?

A2: DIAYN samples a skill from a uniform prior and conditions its policy on that skill for a particular choice of time horizon. We choose a time horizon of 30-minutes to match the biological recording duration, and find empirically this provides the best correspondence with the data (Section 3.3; Appendix A.6). This is not a limitation of our setup, this is deliberate hypothesis testing. The result that persisting a skill for the 30-minute trial best explains the data confirms the empirical fact that flies do not switch behavioral strategies within a trial, but rather maintain a single strategy.

However, we agree that a comparison for all skill discovery algorithms over the same horizon is needed. Thus, we re-evaluate the skill learning IDMs with the 3 evaluated skill horizons: 5 seconds, 10 seconds, 30 minutes (Appendix B.1, Table 9). The results of the ablation show that DIAYN, CSF and VALOR prospective decoding improves as the skill horizon lengthens and VIC, DADS and rDADS behavior stays the same (no turns or only turning in one direction).

These results are consistent with fly’s maintaining a persistent strategy over the course of the trial. Thus, the remaining performance differences at 30-minutes result from DIAYNs intrinsic reward formulation itself, not the the skill horizon.

Q3: Can the authors train the models with more than one random seeds to confirm whether DIAYIN has a robust performance gain over other methods?

A3: While each objective was trained with one initialization, each of the 2048 training environments were independently seeded with procedurally generated obstacle and odor configurations (Section 3.2; Appendix A.5).

Thus each policy was trained across a diverse data stream, which we believe is more relevant than network seeding. In addition, our evaluations were performed by sampling the policy rather than greedy/deterministic evaluation (Appendix A.6).

Q4: What justifies DIAYIN as an exploration algorithm or a computational account of biological intrinsic motivation?

A4: We refer the reviewer to discussion in W1, W3 and Q1/A1. Simply put, DIAYN maximizes the entropy of behavior assigned to a particular skill, and keeps behaviors produced by different skills separate. Maximum entropy exploration has a long history in biological exploration at various spatial scales and species, from foraging in mice to chemotaxis in C. elegans to motor babble in babies. The second part of DIAYN, that keeps skills separate, is just a clustering technique to separate different types of maximum entropy exploration into different clusters. We empirically find that the diversity of these clusters matches the diversity in the fly population, suggesting each cluster/skill is analogous to an individual fly. The individual fly itself is simply maximizing its own local action entropy. Thus, we offer a novel interpretation that the persistence we observe in flies, like motor handedness or a turning bias, is a genetically encoded prior for each individual that maximizes diversity across the population. But each individual fly pursues its own maximum entropy exploration, conditioned on its own genetic prior, without needing to know the population-level objective.

I appreciate the authors’ detailed rebuttal and added analyses. My main concern remains about the paper’s overall claim. The results show that an existing AI objective can fit certain aspects of fly behavior and neural representation better than the tested alternatives. This is interesting, but it does not yet provide a biologically grounded mechanism for how flies generate behavioral diversity or select movement strategies. It also does not advance exploration algorithms or show better exploration or downstream task performance. Therefore, I still find the broader contribution to neuroscience or AI limited, and will hence maintain my score.

Thank you for following up. We respectfully disagree with characterizing our contribution as “fitting selected fly statistics,” as this misses the central scientific contribution. To our knowledge, we provide the first explicit computational account of how a single intrinsic objective can jointly generate fly-like diversity in individual exploratory strategies and internal representations predictive of future turns, that have been recently found via advances in whole-brain imaging (e.g. by Brezovec et al. 2023,2024a). These behavioral and neural signatures emerge in our hierarchical DIAYN agent without training the high-level policy to reproduce them. The scientific advance is identifying and testing this common explanation, rather than simply ranking algorithms by their correspondence to fly data.

As a result of our model-brain comparisons in this study, the computational mechanism is explicit: action entropy within strategies and distinguishable sensory consequences across strategies generate diverse exploratory policies, which dynamically select motor primitives through sensory feedback and recurrent state (§3.1–3.2). Individual turning biases emerge through learning; they are not prescribed.

This common objective jointly captures, at test time, population-level behavioral diversity and exploratory trajectories that it was not explicitly trained to reproduce (Fig. 4), turn-direction predictability 8-15 seconds before turns (Fig. 3), and the IPS-like sum-difference organization of kinematics (which is the mechanism identified in the fly by Brezovec et al.'s study) (Fig. 5). CSF and Random-DIAYN use the same skill-persistence schedule but fail to reproduce the broad behavioral diversity (Fig. 4a). Our added controls further identify persistent strategies as the principal source of early predictability, with recurrent activity contributing additional information nearer execution (Appendix B.3).

Our results therefore provide an explicit mechanistic computational account: sensory inputs and recurrent dynamics govern motor-primitive selection, and the learned internal activity reproduces the IPS-like sum–difference organization of kinematics (Fig. 5). Using AI objectives to construct and test explanations of behavior and neural organization is well-established & central to the goal-driven modeling approach in NeuroAI (Yamins & DiCarlo, Nature Neuroscience 2016); the scientific contribution lies in the biological explanation and its empirical support, which we are showing in this agent-based whole-brain setting, and has been less common in past work in NeuroAI which has often been restricted to single-brain regions. Demonstrating downstream-task gains are outside of the scope of this study, and is not a prerequisite for a mechanistic computational account. We ask that our contribution be assessed against the joint evidence and controlled comparisons across 20 intrinsic objectives.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Overall Changes

We thank the reviewers for their time and constructive feedback. We have made significant improvements to the revisions to incorporate each aspect of this feedback. These revisions clarify the scope of our biological claims and the relationship between the intrinsic objectives, learned policies, and experimental observations. We highlight the high-level changes below.

  • First, we clarify DIAYN's role as an intrinsic exploration objective. DIAYN encourages high-entropy action selection within each skill-conditioned policy while promoting distinguishable state distributions between policies with other skills. An individual fly simply maximizes its own action entropy, but the population as a whole maximizes diversity/distingiushablity across individuals. Importantly, an individual fly is not maximizing population-level distinguishability, only its local max-entropy strategy. Our results show that each of these local max-entropy strategies, when viewed across the population, also maximize diversity across individuals. This is a novel scientific interpretation of how behavioral strategies that are persistent at the individual-level but diverse at the population-level may emerge from a shared computational objective. Our experiments show this hypothesis uniquely reproduces the key details of the fly behavioral and neural data. We have revised the main narrative of the paper to highlight these points so they are clear to the reader. The revisions include explanations in the abstract, introduction, results, and discussion.

  • A related source source of confusion was whether discrete vs. continuous skills themselves have a biological correspondence and why they are discrete. Skills are simply a way of clustering similar behaviors into an identifier, they are not a neural representation of behavior. While DIAYN uses discrete skills/clusters, many other algorithms use continuous skills. This choice doesn’t have a relevant bearing on the neural interpretation of the algorithm, it simply reflects that the behavioral data is low-dimensional and doesn’t require a more complex (continous) clustering representation.

  • A third source of confusion was skills a-priori “induce” specific behavioral policies and thus specific turn-bias distributions. This is not the case. Skills are learned from data in order to separate policies into distinct clusters. Each skill is an arbitrary number assigned to a policy. The policy is then trained with an intrinsic reward to maximize the entropy of its actions while staying different enough from the actions of policies with another skill assignment, where “different enough” is quantified by a decoder’s ability to tell different policies with different skills apart. Intuitively, each skill indexes a unique recurrent policy over low-level motor primitives. Although the skill remains fixed during evaluation, the policy itself is state-dependent at every moment in time; its action distribution changes with proprioceptive feedback and a recurrent latent state. Because each skill, which clusters policies that produce similar behaviors, captures a behavioral motif of an individual fly, we interpret a skill as an exploration strategy of an individual fly. We are not asserting a correspondence between skills and biological circuits. Skills are simply an unsupervised clustering of behavioral trajectories that maximize the distinguishability between clusters. The behavioral trajectories within a cluster are produced by a dynamic, state-dependent neural policy, which we then map to the neural architecture of the biological fly.

  • Fourth, we made several improvements to the overall clarity and writing, such as a general explanation of each intrinsic motivation family and how it relates to animal exploration (section 2.2). We also included a new figure (Fig. 4b) that visualizes the diversity in exploration strategies in x-y space of the biological flies compared with DIAYN.

  • Finally, although we provided the code as a .zip file in the supplementary files of the original submission, we realize that this was not clear and we apologize for any confusion. For ease of access, we provide an anonymized git repository hosted here: https://anonymous.4open.science/r/vigilant-memory-583D/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.