Structured Learning of Behaviorally Relevant Neural Dynamics in Multimodal Neural Time-series

Multimodal neural data can enable a more complete understanding of brain dynamics underlying behavior. Modalities such as neuronal spiking activity, local field potentials (LFPs), and behavioral signals capture diverse spatiotemporal aspects of brain processes. By leveraging these complementary strengths, multimodal neural fusion can provide a unified, rich representation of brain-behavior processes and address the limitations of single-modality analyses, such as incomplete or noisy data. While recent works have jointly modeled behavior and neural data to separate sources of variability, they largely rely on latent variable models that use a single modality of neural data. Here we develop BREM-NET, a multimodal dynamical model that integrates behavioral signals and multiple neural modalities—such as LFPs and spike counts—with distinct statistical characteristics and temporal resolutions. BREM-NET performs multimodal neural fusion during inference while also separating behaviorally relevant and residual neural dynamics in multimodal neural time-series via a structured learning framework. In two independent public multimodal neural datasets spanning multiple behaviors and neural modalities, our method learns a low-dimensional behaviorally relevant latent subspace that is sufficient for behavior decoding while separately modeling residual neural dynamics that improve neural prediction. Furthermore, our method achieves these capabilities even when different neural time-series modalities are asynchronous and have distinct temporal resolutions, which is a major challenge in real-world neural recordings. This framework provides a new tool for studying behaviorally relevant neural computations across different spatiotemporal scales of brain activity.