- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper is an RL/computational neuroscience paper that uses recently-developed biomechanically-realistic simulators to investigate normative models of animal behavior via intrinsic motivation objectives. The method compares several intrinsic motivation methods, including the standard RND alongside newer methods like C-TeC, mutual information skill learning methods, and empowerment-maximization methods, and compares the learned representations and behavior against actual representations and behavior in fly recording data. The paper finds signficant alignment between the learned representations and behaviors of skill learning method DIAYN and real behavior + representation data with a variety of probes (e.g. turning bias, PCA of internal representations) for these signatures.
Critical strengths
I think there are critical strengths to the paper, that make it a very valuable contribution to the field of biologically-plausible skill learning.
-
I think it is fantastic to utilize these biomechanically realistic simulators with intrinsic reward to provide, quote, a normative model of animal behavior via skill learning. This is a new research push using these simulators, that pure behavioral cloning-like approaches (or even using purely extrinsic rewards) cannot capture. The results themselves are very interesting and compelling, particularly Fig. 5 which is very interesting.
-
Furthermore, the comparison to actual neural + behavioral signatures in fly data is very welcome, necessary for any clear contribution to the computational neuroscience community (beyond simply the community in RL that studies intrinsic motivations). This is a big strength that sets this paper apart from a lot of ML conferencepapers that try to make biological conclusions from purely synthetic data.
Critical weaknesses
That being said, there are some critical (though very addressable in the review period) weaknesses that I am concerned about that, if resolved, would move me to Accept.
- The definitions of the empowerment methods are vague in the paper (with no clear prior reference/the authors develop and augment prior methods). While having public access to the code should help with this, I think it is important to concisely conceptually and mathematically describe every implemented intrinsic somewhere in the main-text. As it is, the current writeup of these methods treatment of empowerment seems ad hoc.
I think being careful about introducing and discussing empowerment is extremely important, especially given that the predecessor of VIC by overlapping set of authors (see VIM by Mohamed and Rezende) optimize empowerment, VIC mentions empowerment but is categorized under skill learning, and VIC itself is a precursor of DIAYN. In other words, there exists a close connection between skill learning objectives and empowerment objectives, and being very clear which is which (without expecting readers to automatically understand the difference between optimizing channel capacities vs. directly optimizing mutual informations) is critical.
-
Furthermore, there should be more pause in explaining/describing the actual plotted metrics (Figs. 3-5) and the underlying methods. ``Internal representations'' (I think I understand what is used here, but need clarification) can mean multiple things -- for instance, these representations can literally refer to internal representations within a neural network. And the way these representations are used is different between e.g. prediction error methods like RND, skill learning methods like DIAYN, etc., so an apples-to-apples comparison is a little strange/not immediately obvious. Note that the math is not written in the main text at all, even though parameterizations of intrinsic reward methods in terms of is referred to in the figures.
-
Again along these lines, there needs to be more main text description of what it means to run DIAYN on low level primitives. I do not really know what this means without digging into the appendix, and I think these types of modifications to underlying methods has to be made crystal clear.
-
Minor nitpick, but I think it is important to reference prior works in neuroscience literature that look at the representations of motor skill learning (there is not much but still some). E.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC12439889/ which I think has some neat connections with the author's work.
Questions for authors
-
What are your modifications to DIAYN + empowerment methods used in the paper?
-
Would you be able to provide an anonymous GitHub link to the codebase repository as stated in your reproducibility statement? This would significantly help me parse the choices for the various skill learning and intrinsic motivation methods.
3 comments
Thank you for the clarification
Thank you for the clarification on all these points, which have sufficiently answered my questions.
I have now raised my score to 3 and am advocating for a weak accept.
If you could please put your paper revisions in a different color and upload, I think it would be easier for all of us to see what revisions you've made.
We thank the reviewer for reconsidering their score and for their support.
As requested, we have uploaded a version of the manuscript with revisions highlighted in a different color as part of the supplementary materials to make the changes easier to identify.
Response to Reviewer 7sTd
We thank the reviewer for their time in evaluating our paper and providing thorough, constructive feedback. We have incorporated their suggestions into our revisions, and address each and every of the reviewer's concerns and questions below.
Response to Weaknesses
W1: We have clarified the definitions and implementations of DIAYN, Empowerment and the other intrinsic objectives in the main text (Section 2.2 and 3.2) and Appendix A.4.
Prior work has discussed at length the connections between empowerment and DIAYN such as [1] and our motivation in this work is not to re-iterate these ideas but to evaluate these objectives in our context for representing biological and neural. We believe it is outside the scope of this work to exhaustively explain the evolution and connection between empowerment and skill-learning methods, especially since our results rely only on understanding DIAYN, which can be intuitively understood as a skill-discovery algorithm independently from channel-capacity empowerment.
[1] C. Ji, V. Myers, S. Levine, and B. Eysenbach, “The Geometry of Empowerment,” arXiv preprint arXiv:2610.07796, 2026. [Online]. Available: https://arxiv.org/abs/2610.07796
W2: All compared objectives use the same recurrent policy architecture. Consequently, the internal representations analyzed in our neural comparison refer consistently to network activations in the policy, not to the auxiliary networks specific to each intrinsic objective. This provides a consistent basis for comparing the effects of different intrinsic training objectives (Section 4; Appendix A.3 and A.10). To make the metrics more clear, we replaced Neural and Behavioral Alignment with Neural and Behavioral prospective decoding.
We also clarify the plotted metrics:
These definitions and the meaning of the encoded representation φ_t are now stated explicitly in the corresponding sections and appendices (Sections 3.2–4; Appendices A.3, A.9–A.11).
W3: Our implementation uses DIAYN as a high-level, skill-conditioned policy over seven pre-defined low-level motor primitives, rather than learning continuous joint-level actions directly.
Specifically, a latent skill identifies a recurrent policy conditioned on that skill. At each 100 ms control step, the policy selects a motor primitive based on proprioceptive and other available observations together with its recurrent state. The skill code remains fixed over the evaluation episode, but the motor-primitive selection remains dynamic. DIAYN therefore learns distinguishable, state-dependent strategies for combining existing motor primitives. The skill is an identifier of the strategy assigned through learning, not a prescribed motor sequence.
We have expanded Sections 3.1–3.2 and Appendix A.4 to make this hierarchical implementation more explicit.
W4: We thank the reviewer for their suggestion here, and have incorporated literature by Amematsro et al. (2025), along with a related work, Braun et al. (2024), into the discussion of motor subskills and neural representations (Section 2.1).
Response to Quesitons
Q1: What are your modifications to DIAYN + empowerment methods used in the paper?
A1: We do not make any modifications to the DIAYN algorithm, or any other intrinsic motivation, in this paper beyond their novel application in a hierarchical neuromechanical model of drosophila. For DIAYN, we optimize with PPO instead of the original SAC, but this is just an optimizer choice and not a change to the intrinsic reward itself. We realize these ideas were not clear in the original submission because an explanation of DIAYN and other algorithms was missing. However, we have amended this in sections 2.2 and 3.2 and believe it should now be clear how DIAYN is used in our paper.
Q2: Would you be able to provide an anonymous GitHub link to the codebase repository as stated in your reproducibility statement? This would significantly help me parse the choices for the various skill learning and intrinsic motivation methods.
A2: The anonymized source code was included in the supplementary materials as a .zip archive, and we apologize if this was not clear. For easier access, we additionally provide the repository at: https://anonymous.4open.science/r/vigilant-memory-583D/