acceptodds
Under review as a conference paper at ICLR 2027

Structured Learning of Behaviorally Relevant Neural Dynamics in Multimodal Neural Time-series

Abstract

Multimodal neural data can enable a more complete understanding of brain dynamics underlying behavior. Modalities such as neuronal spiking activity, local field potentials (LFPs), and behavioral signals capture diverse spatiotemporal aspects of brain processes. By leveraging these complementary strengths, multimodal neural fusion can provide a unified, rich representation of brain-behavior processes and address the limitations of single-modality analyses, such as incomplete or noisy data. While recent works have jointly modeled behavior and neural data to separate sources of variability, they largely rely on latent variable models that use a single modality of neural data. Here we develop BREM-NET, a multimodal dynamical model that integrates behavioral signals and multiple neural modalities—such as LFPs and spike counts—with distinct statistical characteristics and temporal resolutions. BREM-NET performs multimodal neural fusion during inference while also separating behaviorally relevant and residual neural dynamics in multimodal neural time-series via a structured learning framework. In two independent public multimodal neural datasets spanning multiple behaviors and neural modalities, our method learns a low-dimensional behaviorally relevant latent subspace that is sufficient for behavior decoding while separately modeling residual neural dynamics that improve neural prediction. Furthermore, our method achieves these capabilities even when different neural time-series modalities are asynchronous and have distinct temporal resolutions, which is a major challenge in real-world neural recordings. This framework provides a new tool for studying behaviorally relevant neural computations across different spatiotemporal scales of brain activity.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Multimodal Deep Learning, Nonlinear Dynamical Modeling, Neural-Behavioral Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_3b5JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Different modalities used to record neural activity have unique trade-offs. Electrophysiological recordings of spike trains capture faster timescale dynamics at single-neuron resolution. Local field potentials (LFPs) better capture longer-timescale dynamics and are more stable over days, but provide a coarser picture of population activity. While latent models of neural dynamics are often unimodal, there is growing interest in combining the advantages of multiple modalities to learn a more complete model. In parallel, there is a growing body of literature focused on isolating behaviorally-relevant activity, which can sometimes be lower variance and masked by other neural dynamics. This paper proposes BREM-NET, which combines multimodal modeling with separate latent variables for capturing behaviorally-relevant dynamics and the residual dynamics that capture the remaining variance. The authors applied BREM-NET on two multimodal NHP datasets, showing that two modalities improves neural activity prediction over using any one modality in isolation. The largest gains tended to be over LFP in isolation, but both spiking and LFP recordings benefitted from multimodal modeling. The authors compared their model to a number of existing multimodal models, showing that the addition of behavior-specific latents was helpful. An ablation analysis showed that their three stages of training, which first focus on learning behaviorally-relevant dynamics, was critical to their improved performance.

Critical strengths

The paper clearly demonstrates the advantages of incorporating multiple modalities and explicitly considering behavior when modeling neural dynamics. The analysis was sufficiently rigorous, comparing to many relevant baselines and performing ablations to understand which components of the pipeline were most critical.

Critical weaknesses

The approach is a relatively straightforward combination of two existing methods (minor).

Questions for authors

N/A

Reviewer_ApDnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a nonlinear recurrent model for learning behaviorally relevant neural dynamics from multimodal neural recordings, so called BREM-NET. The model combines spikes and LFPs through separate encoders and uses a three-stage training procedure to learn behaviorally relevant and residual neural latent states. The authors evaluate the method on two reaching datasets, where the results show improved behavior decoding and neural prediction compared to several baseline methods.

Critical strengths
  1. The idea of combining different neural modalities to better understand neural activity and behavior is well motivated and has been studied in computational neuroscience.
  2. The proposed framework is clear and straightforward.
  3. The authors compare the method with several existing approaches and provide ablation studies.
Critical weaknesses
  1. In the results, the paper moves directly to real neural datasets after introducing the model, and the synthetic experiments are placed in the appendix. I think the synthetic experiments deserve more attention, since they can help validate whether the proposed method actually learns the underlying dynamics. In the current experiments, the authors mainly evaluate prediction accuracy, although the ground-truth behaviorally relevant and independent latent components are known. A direct comparison between the learned and true latent components would be much more convincing. I would suggest including such an experiment in the main paper to better validate the proposed structured learning approach.
  2. The three stage learning procedure is structured or modular. However, I did not get the real usage of stage 2 results. Stage 2 learns separate neural decoders from the behaviorally relevant latent states, while Stage 3 models the remaining neural activity. This structure seems useful for interpreting how neural activity is separated into behaviorally relevant and residual components. But the current experiments mainly evaluate prediction accuracy. Could the authors further the neural activity explained by each component, for example, by comparing their reconstructed neural signals or temporal properties?
Questions for authors

A question throughout the paper is that the experiments mostly compare the prediction accuracy. I wonder what additional interpretation of neural dynamics can the proposed method provide. For example, can the learned latent states reveal meaningful dynamical properties or neural activity patterns related to behavior?

Reviewer_bk57OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work, the authors proposed a framework named the behaviorally relevant modeling of multimodal neural time-series (BREM-NET), which incorporates both the LFPs and spike counts modality neural signals in the whole algorithm. The framework is composed of three stages: 1. learning behaviorally-relevant neural latents, 2. learning of the neural decoders. 3. learn residual neural latents. On two non-human primate real-world neural datasets arcoss behaviors and neural time-series modalities, the method could infer a behaviorally-relevant latent while disentangle the residual neural dynamics for the neural prediction purposes.

Critical strengths
  1. The central theme of this proposed BREM-NET framework: inferring behaviorally-relevant neural dynamics, plays an important role in neuroscience research.
  2. I roughly agree with the claim in Line 31 of the paper, BREM-NET would be potentially a new tool for understanding behaviorally-relevant neural computations across different spatio-temporal scales of neural activities and signals.
  3. The methodology of the proposed BREM-NET is grounded in first principles and could adapt to neural data with heterogeneous distributions as well as various temporal resolutions.
Critical weaknesses
  1. The main framework illustration in Figure 1 is a bit small, making it difficult for readers to follow.
  2. as BREM-NET is proposed as a multimodal method, the paper would benefit from additional results across different species and neural modalities beyond spike counts and LFPs. The current evaluation on two datasets seems somewhat limited.
  3. I find the NLL term in the formulations of Equation (5), (6), (7) a bit unclear and weird. If a standard gaussian is assumed for the behavior variables, minimizing the NLLs would be equivalent to optimizing the mean-squared-error loss. It would be helpful to clarify the assumptions you made here and clarify the loss BREM-NET is used in practice.
Questions for authors

I have no more or further questions at this point, please refer to my weaknesses section for my other questions. thanks.

Reviewer_2jieOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes BREM-NET, a neural model architecture and training method which trains on paired neural and behavioral time-series data and learns two distinct latent spaces: one trained to predict behavioral features given neural input, and another to reconstruct the neural timeseries themselves. The method learns behavioral and residual neural features in a modular way while leveraging multi-modal input. These modalities can operate at distinct timescales, and the method is robust to noise and missing data common in the real world. The approach is evaluated on held-out data and compared against various baselines and ablations, with consistently positive results.

Critical strengths

The aim and contribution of the work is relatively clear. The method is explained precisely, and results thoroughly deliver on the work’s stated ambitions. The ablations are through and strongly contributed to the credibility of the claims.

Critical weaknesses

There are a few potential weaknesses of the paper, non of which I would consider critical.

  • The BREM-NET method seems incremental in the sense that it's largely an additive synthesis of existing methods (DPAD’s prioritized learning scheme, MSID's synthesis of multi-scale input data). The missingness (dropout) problem is addressed via zeroing out the encoder output which also falls out naturally for additive encoders.
  • The idea of functional separation as opposed to statistical separation / orthogonality is intriguing, and I would be curious to see more justification for this. The functional separation is done via staged training with stop gradient operators that prevent training signals to flow to components learned in earlier stages. The paper contains comparison against BREM-NET-Joint, which uses a single combined loss term and also removes training stages. I’d be curious to see more fine-grained ablations that keep the latent separation but removes training stages and/or stop gradient operators, and how it influences the functional separation and results.
  • The claim that BREM-NET performs multimodal neural fusion with distinct temporal resolutions is not supported in full generality. The method performs inference on the timescale of the faster modality, and treats the data with the slower modality as missing. In the experiments the two modalities have the same native sample rates of 10ms, and asynchrony is introduced only by artificially downsampling or dropping samples in one modality. The paper's claims would be made stronger with experiments on data with natively different sampling rates and/or analyses of distinct timescales captured by the latents.
Questions for authors
  • What's the scientific significance of the functionally separated latents, both in its own right and as opposed to statistically separated ones?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.