- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper aims at zero-shot neural decoding of behaviors from across-subject, spike-sorted electrophysiology recordings: a setting where new recordings come (1) from previously unseen neurons (new subjects or measurement locations), and (2) without paired target behavioral variables, limiting fine-tuning of the input-output decoding map. To explain the difficulty of this setting, the authors define the neuron-identity problem (not knowing the neurons one is decoding from a priori), and point out that current state-of-the-art multi-recording decoders typically have recording-specific parameters that need to be learned from neural activity paired with behavior. The authors tackle this problem through their framework called Zen (Zero-shot on new neurons). It consists of (1) a shared Neuron-Level Encoder (NLE), which computes a representation for each neuron from activity, and (2) a decoder that is pre-trained over this inferred representation space. The authors evaluate this framework with two well-established NLEs (plus fixed inter-spike-interval features) and two decoders from existing literature, along with a newly introduced decoder called Time-Patch, which preserves separate neuron-level tokens at the input to a Transformer backbone. On two electrophysiology datasets, the authors' best-performing model configuration Zenith (NuCLR NLE + Time-Patch decoder), outperforms other zero-shot baselines in decoding whisker motion energy, wheel and running speed, and reward. Furthermore, under the standard regime where decoders can be fine-tuned on the target recordings with the behavior supervision, Zenith shows state-of-the-art performance in a majority of decoding tasks of IBL with only 25% of the fine-tuning trials.
Overall, this paper could be a significant contribution to an important problem. If the authors address the two main concerns listed in the Weaknesses below (the fairness of the comparison to baselines and the missing analysis of failure cases), I would be willing to increase my score.
Critical strengths
- Motivation and novelty. This work is well-motivated, targeting an important problem. Additionally, connecting self-supervised neuron-representation learning with neural decoders is novel: while the cited SPINT (Le et al., 2025) also infers neuron identities from activity, it learns them jointly with the decoder rather than using a separately pre-trained NLE as the input space. The paper is also clearly written and well structured.
- Strong empirical results. In the zero-shot regime, Zenith outperforms all baselines on all five tasks across both datasets, often by large margins (e.g., reward balanced accuracy on IBL improves from 0.708 to 0.828 over SPINT, and running-speed on Allen VBN from 0.311 to 0.611). In the fine-tuning regime, Zenith with only 25% of the target trials already surpasses the best fully fine-tuned baselines on IBL wheel speed and reward decoding, and with full fine-tuning it matches or exceeds all baselines on the three IBL tasks.
Critical weaknesses
- Unequal access to neural data during pre-training. The paper does not establish that the NLEs and the baselines were pre-trained on the same neural activity. The baselines appear to be trained only on short trial windows (1.0 s in IBL, 0.8 s in Allen VBN), while NuCLR is pre-trained on 20 s context windows that include activity outside these trials, and NEMO is pre-trained on waveform templates and ACGs whose time range for training recordings is not specified. It is also not stated whether the test subjects' recordings were withheld from NLE pre-training, or whether the lower-quality sessions "included in pretraining" were used for the baselines as well. I ask the authors to specify the exact data (subjects, sessions, and time ranges) used for NLE pre-training, and ideally to add a control where the NLE is pre-trained only on the same trial windows and training subjects as the baselines.
- Little analysis of failure cases. The paper provides almost no analysis of when zero-shot transfer fails. For example, neurons whose NLE embeddings are out of distribution, or are poorly represented by the NLE, may hurt zero-shot performance. Evaluating with subsets of neurons (e.g., grouped by brain region or by distance to the training embeddings) could show whether such neurons exist and how important they are. Additionally, there is no per-subject or per-session analysis, which could quantify the reliability of Zen.
Questions for authors
Please consider the questions below as ways to strengthen the paper, not necessarily as prerequsities for publication.
- Table 2 suggests the NLE drives most of the performance: with NuCLR, the gap between POYO and Time-Patch is small. Is the large transformer backbone actually needed? How does a more lightweight decoder on top of NuCLR embeddings (e.g., a linear version of the NDT-Stitch integration with a linear readout) perform?
- The NLE is frozen during both decoder pre-training and fine-tuning. Have the authors tried updating the NLE jointly with the decoder either during pre-training or during fine-tuning? This would show whether the frozen, self-supervised embeddings are a strength (better generalization) or a limitation (not optimized for decoding).
- Could the authors discuss (either in the paper or here) the feasibility of across-subject zero-shot neural decoding of more complex variables, including stimuli rather than behaviors (e.g., visual stimuli)?
7 comments
Rebuttal (2/5)
Weakness 1: Unequal access to neural data during pre-training. (Continued)
I ask the authors to [...] ideally to add a control where the NLE is pre-trained only on the same trial windows [...] as the baselines.
Thank you for this suggestion. We agree the NLEs and baselines should be compared with access to the same neural activity. To control for this, we pretrain representative baselines with the same activity available to the NLEs during pretraining, including inter-trial periods and longer context windows. We chose this over restricting the NLEs to 1 s/0.8 s trial windows, which would depart from the original NLE pretraining procedures, where each neuron is identified from long spans of activity.
Zero-Shot Comparisons: It is true that the NLEs are pretrained on inter-trial periods, which many of the baselines cannot make use of. However, we would like to note that SPINT's ID encoder also draws its 1s/0.8s windows from the same time range as the NLEs, and not just from the trial periods. So SPINT, which is the strongest zero-shot baseline, already has access to this activity. We performed an additional control experiment, and gave SPINT’s ID encoder 20s windows (matching NuCLR’s window size), and found that its performance does not increase.
Table R3-3: Control experiment 2. SPINT with ID encoder window size matched to NuCLR’s (20s) on IBL. Performance reported as mean ± SEM over 3 seeds.
| Method | Whisker ME R² | Wheel Speed R² | Reward bal. acc. |
|---|---|---|---|
| SPINT (1s windows) | 0.079 ± 0.013 | 0.222 ± 0.001 | 0.744 ± 0.008 |
| SPINT (20s windows) | 0.068 ± 0.009 | 0.211 ± 0.004 | 0.752 ± 0.005 |
| ZENITH | 0.163 ± 0.004 | 0.262 ± 0.002 | 0.828 ± 0.006 |
Finetuning Comparisons: In the case of the pretrained-finetuned analysis of Section 4.5, the baselines pretrained with self-supervised objectives (NDT-Stitch and MTM) are the only ones that can readily pretrain on activity outside the trial windows, since their masked-modeling objective does not require behavioral labels. They are pretrained on the same 391 sessions as the NLEs, but only on the trial periods. We removed this restriction for NDT-Stitch, and pretrained it on windows sampled from the whole recording. With this, NDT-Stitch is pretrained on exactly the same neural activity as NuCLR. We then finetuned it on the test sessions as before. We find that its performance is unchanged on all three tasks (Table R3-4).
Table R3-4: Control experiment 3. Finetuned performance of pretrained baseline NDT-Stitch, when pretrained on trial periods only vs when pretrained on the whole recording (IBL).
| Method | Whisker | Wheel Speed | Reward |
|---|---|---|---|
| NDT-Stitch (pretrained on trial periods only) | 0.299 ± 0.003 | 0.299 ± 0.002 | 0.852 ± 0.001 |
| NDT-Stitch (pretrained on whole recording) | 0.302 ± 0.011 | 0.298 ± 0.006 | 0.847 ± 0.004 |
| ZENITH | 0.350 ± 0.009 | 0.372 ± 0.003 | 0.899 ± 0.010 |
We hope that the three control experiments above address the reviewer's concern about unequal access to neural data.
Manuscript updates:
- New Appendix A.3: data used to pretrain each model.
- New Appendix A.4: three control experiments for differences in pretraining data access.
- Revised Appendix A.1 to clarify which models use the lower-quality sessions.
- Revised Section 4.1 to state that test subjects are withheld from all pretraining stages.
Rebuttal (3/5)
Weakness 2: Missing analysis of failure cases.
The paper provides almost no analysis of when zero-shot transfer fails.
Thank you for this suggestion. We have performed a comprehensive analysis of failure cases of ZENITH and have added it as Appendix I in the updated manuscript.
Out of the 117 session-task pairs across IBL and Allen datasets, we found 10 cases where the zero-shot performance of ZENITH was lower than chance (R² < 0, or bal. acc. < 0.5). We examined all 10 failures, and found that they fall under three failure modes. We summarize them here, and refer you to Appendix I for a detailed description along with the figures.
- Neural activity containing little information about the behavior (8/10 failures). We observe a strong correlation between zero-shot and finetuned performance across the test sessions, and we find that 8/10 failures are sessions that also have low performance after finetuning on that session. This suggests that the relevant behavioral variables may be difficult to decode from these recordings, even with session-specific supervision. However, we cannot rule out limitations of the decoder or fine-tuning procedure as alternative explanations. In Appendix I.1, we additionally analyze the particular instance of the IBL whisker failures, where we find that these sessions have very few neurons that are tuned to whisker motion.
- Misleading neuron representations (1/10 failures). This is a case where neurons are poorly represented by the NLE. In one IBL session, zero-shot reward decoding is below chance (0.35), although finetuning on this session reaches 0.98. This suggests that the neural activity contains sufficient information for reward decoding, but the pretrained decoder fails to interpret it. To investigate, we train a linear probe on NuCLR embeddings to distinguish reward-preferring neurons (which fire more after reward) from error-preferring neurons (which fire more after error). When we look at the accuracy of this probe for each test session, we find that this session is the only one where the probe is below chance. That is, in this session many error-preferring neurons actually look like reward-preferring ones in the embedding space. To test this hypothesis, we attempt to correct the neuron embeddings by shifting them along the reward-preference direction identified by the linear probe (see Appendix I.2 for details). With this correction, zero-shot reward balanced accuracy improves from 0.35 to 0.62. This suggests that the decoder reads whether a neuron is reward- or error-preferring from its embedding, and that in this session the faulty embeddings mislead the decoder. We do not yet know why NuCLR represents this session poorly.
- Out-of-distribution behavior (1/10 failures). The remaining failure is the running speed decoding on one Allen VBN session. We observe that the running speed of this mouse is much higher than what is seen during pretraining: it runs at 80 cm/s on average, while the fastest pretraining animal averages 54 cm/s. Upon looking at what the decoder predicts for this session, we find that the decoded speed follows the changes in the actual speed from trial to trial, but it is different in terms of raw magnitude. So the failure likely happens because the decoder needs to predict speeds that it has rarely seen during pretraining, and it does not predict their absolute value correctly.
For example, neurons whose NLE embeddings are out of distribution, or are poorly represented by the NLE, may hurt zero-shot performance. Evaluating subsets of neurons [...] how important they are.
In addition to the analyses presented above, we also tried this suggestion. However, we find that NuCLR embeddings of test neurons are not really out-of-distribution from pretraining neurons in any trivially measurable way (cosine or euclidean distances). We also attempted the idea of ablating one brain region at a time (Cosmos atlas), but found that this did not help recover any failing session. Regardless, we agree these were good hypotheses for failure modes, and we thank the reviewer for suggesting them to us.
Additionally, there is no per-subject or per-session analysis, which could quantify the reliability of Zen.
The failure case analyses partly answer this question: ZENITH obtains below chance performance in only 10 out of 117 session-task pairs. We also perform another comparative per-subject and per-session analysis, comparing ZENITH with NuCLR + POYO, NuCLR + NDT-Stitch, and SPINT, to see if the performance gains of ZENITH are reliable across subjects and sessions. This analysis has been added as Appendix H in the updated manuscript.
Manuscript updates:
- New Appendix I: failure analysis of all 10 failing session-task pairs, with Figures A3-A6.
- New Appendix H: paired per-animal and per-session comparative analysis between methods.
Rebuttal (1/5)
We thank the reviewer for their careful and constructive review of our manuscript. We are glad that they found our work well-motivated and targeting an important problem, and that they found the connection between self-supervised neuron representation learning and neural decoding novel.
The reviewer’s concerns led us to rigorously test the fairness of our comparisons, and to a closer look at when and why zero-shot transfer fails, which we found especially insightful. In summary, we have:
We address each concern and question in detail below.
Weakness 1: Unequal access to neural data during pre-training.
Thank you for raising this. We agree that the submission did not state clearly which data each model was pretrained on. We have now specified this in a new Appendix A.3 of the updated manuscript, which lists the subjects, sessions, and time ranges used for NLE pretraining, and which of these sessions are used by each model. We also ran three control experiments to test whether the differences in data access can explain ZENITH’s advantage over the baselines. We have added these as a new Appendix A.4.
To summarize our response: the test animals are not used in any pretraining stage, and the NLEs and the baselines are pretrained on the same animals and sessions, except for 63 IBL sessions which do not pass behavior QC and thus can only be used by models with self-supervised pretraining (NLEs, and pretrained NDT-Stitch and MTM). Through a control experiment, we find that these extra sessions do not explain ZENITH's advantage over the baselines, since its performance stays similar when they are removed from NuCLR's pretraining. We ran two more control experiments in which the baselines are given this activity. SPINT's ID encoder is given 20s windows, and NDT-Stitch is pretrained on whole recordings including inter-trial periods. We find that neither of them improves with these changes. This suggests that ZENITH's advantage comes from what NuCLR learns from this activity, and not simply from having access to inter-trial periods.
Table R3-1: Data used for NLE pretraining (NuCLR and NEMO). Also added as Table A2 in Appendix A.3.
The test animals’ recordings were withheld from all pretraining stages, including NLE pretraining, and are only used at evaluation. We have added this to Section 4.1.
We apologize for the confusion caused by this sentence. We have updated the IBL dataset appendix (Appendix A.1) with clearer language. As summarized above, only the 63 IBL sessions that do not pass behavior QC differ, and they are only used by models with self-supervised pretraining. (NuCLR, NEMO, and the pretrained NDT-Stitch and MTM of Section 4.5). We additionally perform a control experiment below by removing these sessions from ZENITH’s pretraining.
Control: We pretrained NuCLR only on the same 328 sessions (106 animals) used by the supervised baselines, and then retrained the Time-Patch decoder on top of it. We find that the performance stays similar (Table R3-2). Whisker motion energy increases from 0.163 to 0.184 R², wheel speed decreases from 0.262 to 0.249 R², and reward remains unchanged. With the sessions and animals matched, ZENITH still outperforms SPINT on all three tasks.
Table R3-2: Control experiment 1. ZENITH with NuCLR pretrained only on the baselines' pretraining sessions (IBL). Performance reported as mean ± SEM over 3 seeds.
*Note: We have improved the performance of SPINT baseline. For details, please see our answer to Q2 of reviewer c69y.