- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a causal transformer architecture with a SPINT-like neural tokenization scheme and a LeJEPA-based pretraining strategy. The authors show that the proposed architecture outperforms several baseline methods across decoding tasks and provides better stability across recording days.
Critical strengths
- The paper is clearly written; it is easy to follow the motivation, methodology, and evaluation setup.
- I appreciate that the authors included a decent number of baseline methods for performance comparisons and evaluated their model across multiple datasets.
Critical weaknesses
- Even though the authors considered several baseline methods for comparison, I think models specifically developed for across-day stability could be considered beyond SPINT, for which no results could be shown for human data. For example, it would be interesting to see how ADAN or NoMAD perform in the authors' evaluation setup.
- It is not clear which component in the formulation results in stable ranks across recording days and improved performance. The ablation studies show similar performance across variants and no direct comparison with baselines.
- The results include no statistical tests, which is important since the performance margins seem pretty close.
- Stability results across multi-unit vs. single-unit spikes can show significant performance differences. For example, NoMAD uses multi-unit spikes, which provides a more stable baseline for the aligner network compared to sorted spikes. I am curious how the proposed method would compare to NoMAD for the motor datasets used in the NoMAD paper across single-unit and multi-unit spikes. Even though the authors have two datasets in their manuscript, I think including NHP motor datasets and comparing with other methods such as NoMAD would significantly improve the manuscript.
Questions for authors
What is fundamentally different between day-specific FiLM (used for human recordings) and projecting FR, ACG, and PC into token space? Are the authors using the same FR, ACG, and PC statistics for FiLM conditioning? If so, is the main difference the inductive bias of creating a pathway for the model to leverage these statistics through modulation if needed, rather than directly encoding them?
4 comments
W2. Clarification of day-specific FiLM
We thank the reviewer for the opportunity to clarify the motivation for day-specific FiLM. Fundamentally, both statistical neuron-ID embeddings and day-specific FiLM support adaptation to unseen neural populations, but address different sources of variability: statistical neuron IDs accommodate changes in the recorded neurons through matching their functional similarity, whereas FiLM infers and compensates for day-specific drift.
Mechanistically, statistical neuron ID embeddings leverage the firing properties of neurons (FR, ACG, and PC) to capture functional similarity that can transfer across sessions without knowing a neuron’s fixed identity. Once we embed these neurons’ firing signatures into the same latent space, similarly behaving neurons are mapped to similar locations in the token space.
We do not use this approach for channel-based Utah array recordings because (1) each channel records from multiple units, mixing their firing properties and making their signatures less reliable than those of single units; (2) it performs poorly empirically, as shown in the new ablation study summarized below (also in Appendix J and Table 7 of the updated paper); and (3) prior work on Utah array-based human BCI recordings commonly models per-day drift using day-specific affine transformations [2, 3].
Therefore, we propose per-day FiLM, a nonlinear extension of a per-day affine transformation, for multi-unit data. The MLP in per-day FiLM contains multiple linear layers, each with a hypernetwork trained to predict the parameters of each linear layer. When the model encounters drift on a new day, it can infer the parameters needed to compensate for the drift based on patterns learned during training.
Empirically, we compare statistical neuron IDs, per-day FiLM, and per-day linear transformations on human BCI data and summarize the ablation results below.
Table 2
| Day adaptation method | PER (%) ↓ | Top-5 accuracy (%) ↑ |
|---|---|---|
| Per-day FiLM (calibration statistics) | 38.80 | 29.80 |
| Per-day linear FiLM | 49.20 | 22.10 |
| Statistical neuron ID | 65.00 | 7.90 |
We have added this design rationale and the ablation results to the updated paper.
[2] Willett, Francis R., et al. "High-performance brain-to-text communication via handwriting." Nature 593.7858 (2021): 249-254.
[3] Willett, Francis R., et al. "A high-performance speech neuroprosthesis." Nature 620.7976 (2023): 1031-1036.
Other Questions:
- The results include no statistical tests, which is important since the performance margins seem pretty close.
For the quantitative results in the main paper, we provided boxplots summarizing each baseline’s performance distribution. To further address your concerns, we evaluated NeuroLens on the FALCON benchmark’s M2 track. We provide an anonymized codebase to reproduce these benchmark results here:
https://anonymous.4open.science/r/LATTE-4FA2/README.md
- Are the authors training the decoders for each held-out recording day, or are they using the same decoder fitted on the source day across GF, GB and full-finetune strategies? If not, I would like to see how the results look like when decoders are fixed, especially for the GB scenario which truly evaluates the zero-shot setting.
Thank you for the question. Because the IBL chronic mouse Neuropixels dataset is not yet publicly available for independent replication, we evaluated your proposed zero-shot setting on the public FALCON benchmark’s M2 track [1] and provided an anonymized codebase to reproduce our results. In this setting, we kept both the encoder and decoder fixed without calibration. As shown in Table 1, NeuroLens achieves strong zero-shot performance (R2=0.16), substantially exceeding the strongest ZS baseline (R2=0.06).
- It is not clear which component in the formulation results in stable ranks across recording days and improved performance. The ablation studies show similar performance across variants and no direct comparison with baselines.
We thank the reviewer for helping us clarify the contributions of individual model components. We expanded our ablations to compare statistical and learned neuron IDs (Appendix J), compare adaptation methods for channel-based neural recordings (Appendix K), and evaluate pretraining and calibration data size requirements (Appendix L).
I thank the authors for their rebuttal, I will adjust my score accordingly. That said, I would suggest avoiding words like "strong" and "remarkable" for 0.16 R2 metric for behavior decoding, even though I acknowledge it indicates a better performance compared to baseline methods, and the low-value is potentially dataset specific rather than model specific.
Thank you again for your constructive feedback and for raising your scores! We will revise the paper to improve the precision of our wording when describing the performance gains.
We thank the reviewer for recognizing our thorough baseline comparisons and our approach’s generalizability across multiple datasets. We also appreciate your suggestions to compare additional stable decoders, include chronic NHP motor datasets, and clarify day-specific FiLM, which we address below. Revisions are highlighted in blue in the updated paper.
Summary: Before answering your questions, we would like to highlight our novelty and contributions through our newly added analyses:
[1] Benchmark website: https://snel-repo.github.io/falcon/
M2 track: https://eval.ai/web/challenges/challenge-page/2319/leaderboard/5751
W1. Comparison with NoMAD, SPINT, and other baselines on the FALCON benchmark’s NHP motor dataset across zero-shot, few-shot unsupervised, and few-shot supervised adaptation settings
We thank the reviewer for this question. To compare NeuroLens with NoMAD, SPINT, and other well-known stable decoding methods on this task, we evaluated NeuroLens on the FALCON benchmark [1] M2 track (updated Section 5.2 and Table 1). We chose the M2 track because the task there is to predict 2-D finger velocity from Utah array multi-unit spiking activity recorded from a non-human private (NHP) monkey from motor brain areas.
At the time of submission, NeuroLens ranks 1st among non-ensemble methods across all few-shot adaptation settings: (1) zero-shot (ZS), which uses no held-out calibration data; (2) few-shot unsupervised (FSU), which uses only calibration neural activity; and (3) few-shot supervised (FSS), which uses both calibration neural activity and behavioral labels. NeuroLens flexibly supports all 3 adaptation settings using a unified model. Here, we summarize the results for NeuroLens, NoMAD, CycleGAN, SPINT and NDT2; see Sec 5.2 for more details.
The results are:
These results highlight NeuroLens's strong zero-shot generalization and effective adaptation with limited calibration data.
Table 1