Spherical Recognition-Parameterized State Space Models for Learning Shared Neural and Behavioral Dynamics

Simultaneous recordings of neural activity and behavior offer a window into how brain dynamics relate to an animal’s actions. However, recovering shared dynamical structure from multimodal timeseries is challenging when each modality contains substantial private variation. Generative approaches must reconstruct this variation, while standard contrastive methods learn shared representations without explicitly modeling their evolution. We introduce sRP-SSM, a spherical recognition-parameterized state space model that jointly learns cross-modal representations and stochastic latent dynamics without reconstructing observations. The spherical geometry allows recognition factors to remain log-linear in the latent state even with observation-dependent confidence, so evidence from each modality and from past timesteps combines by simple addition. We derive a scalable contrastive training objective that approximates the model’s evidence lower bound, connecting contrastive representation learning with probabilistic modeling of shared dynamics. In synthetic experiments, sRP-SSM combines strong shared-state recovery and cross-modal retrieval with more accurate dynamics recovery than competing generative and contrastive methods. In simultaneous neural and 3D pose recordings from a freely moving mouse, sRP-SSM improves cross-modal retrieval, future prediction, and keypoint decoding. During rearing, it detects behavior from neural activity and recovers bout-averaged neural spectral patterns from pose. These results show how using shared dynamics to guide representation learning can improve the recovery of neural–behavioral relationships, establishing a foundation for dynamics-informed learning across complementary modalities.