- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Different modalities used to record neural activity have unique trade-offs. Electrophysiological recordings of spike trains capture faster timescale dynamics at single-neuron resolution. Local field potentials (LFPs) better capture longer-timescale dynamics and are more stable over days, but provide a coarser picture of population activity. While latent models of neural dynamics are often unimodal, there is growing interest in combining the advantages of multiple modalities to learn a more complete model. In parallel, there is a growing body of literature focused on isolating behaviorally-relevant activity, which can sometimes be lower variance and masked by other neural dynamics. This paper proposes BREM-NET, which combines multimodal modeling with separate latent variables for capturing behaviorally-relevant dynamics and the residual dynamics that capture the remaining variance. The authors applied BREM-NET on two multimodal NHP datasets, showing that two modalities improves neural activity prediction over using any one modality in isolation. The largest gains tended to be over LFP in isolation, but both spiking and LFP recordings benefitted from multimodal modeling. The authors compared their model to a number of existing multimodal models, showing that the addition of behavior-specific latents was helpful. An ablation analysis showed that their three stages of training, which first focus on learning behaviorally-relevant dynamics, was critical to their improved performance.
Critical strengths
The paper clearly demonstrates the advantages of incorporating multiple modalities and explicitly considering behavior when modeling neural dynamics. The analysis was sufficiently rigorous, comparing to many relevant baselines and performing ablations to understand which components of the pipeline were most critical.
Critical weaknesses
The approach is a relatively straightforward combination of two existing methods (minor).
Questions for authors
N/A