Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila

Animals have a remarkable ability to act autonomously in the absence of external sensory cues or extrinsic objectives, unlike most frontier artificial agents today. The computational mechanisms behind this ability largely remain a mystery. We leverage recent whole-brain recordings in the fruit fly (Drosophila melanogaster), in which neural activity predicts spontaneous turns more than $10$ seconds in advance, to test computational substrate of future volitional action. Specifically, we study whether intrinsically motivated artificial agents embodied in a biomechanically realistic fly-body can develop behavioral and internal representations quantitatively similar to those observed in biological flies during exploration. Our policies operate over a discrete set of high-level motor primitives while low-level central pattern generators actuate joint dynamics, closely matching current ethological theories of behavioral syllables. We train agents with no external reward using $20$ intrinsic motivation algorithms spanning curiosity, entropy, skill discovery, and empowerment methods. We then evaluate frozen policies in a virtual tethered-ball assay modeled after the fruit-fly experiment, comparing the extent to which spontaneous locomotor statistics and internal policy representations predict future turn direction on the same timescales observed in the fly brain. Using these criteria, we find operating hierarchically over motor primitives enables DIAYN—a skill-discovery algorithm that learns a discrete set of policies to maximize both the entropy of behavior and the separability in sensory observations those behaviors induce—to achieve the strongest behavioral and neural alignment with the fruit-fly. Notably, the set of policies DIAYN discovers closely matches the wide individual variability observed in fly exploration strategies, thus suggesting this behavioral diversity optimizes a concrete computational goal. Together, these results establish hierarchical, intrinsically motivated agents as a powerful framework for reverse-engineering animal autonomy.