acceptodds
Under review as a conference paper at ICLR 2027

Pretrained Neuron-Level Representations Enable Across-Subject Zero-Shot Transfer of Neural Decoders

25 views

Abstract

Pretraining neural decoders across electrophysiology recordings offers a path toward models that generalize to unseen subjects, yet existing approaches typically require adaptation to each new neural population. A key obstacle is that spike-sorted recordings lack a shared input space. Each recording contains a new, unordered population of neurons, which existing decoders typically represent using recording-specific parameters. We introduce Zen, a framework that replaces these parameters with transferable neuron representations inferred by a pretrained neuron-level encoder. A decoder trained over this space then operates directly on the new population, without target-specific gradient updates, alignment, or behavioral labels. We evaluate three neuron-level encoders and three decoder architectures, including our proposed Time-Patch decoder. Time-Patch preserves neuron-specific tokens throughout the Transformer backbone, whereas prior architectures aggregate activity across neurons before the Transformer backbone. Our strongest configuration, Zenith, combines NuCLR representations with Time-Patch and achieves state-of-the-art cross-subject zero-shot decoding on the IBL Brain-Wide Map and Allen Visual Behavior Neuropixels datasets, reaching approximately 83% and 85% balanced accuracy on reward decoding, respectively. Under the finetuning setting, Zenith surpasses the best fully finetuned baselines on IBL wheel speed and reward decoding with only 25% of the available target finetuning trials. These results establish transferable neuron representations as a practical basis for both zero-shot neural decoding and data-efficient supervised adaptation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
neural decoders, spike-sorted, electrophysiology, neural population, computational neuroscience, systems neuroscience, transformer, tokenization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_sYzLOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper aims at zero-shot neural decoding of behaviors from across-subject, spike-sorted electrophysiology recordings: a setting where new recordings come (1) from previously unseen neurons (new subjects or measurement locations), and (2) without paired target behavioral variables, limiting fine-tuning of the input-output decoding map. To explain the difficulty of this setting, the authors define the neuron-identity problem (not knowing the neurons one is decoding from a priori), and point out that current state-of-the-art multi-recording decoders typically have recording-specific parameters that need to be learned from neural activity paired with behavior. The authors tackle this problem through their framework called Zen (Zero-shot on new neurons). It consists of (1) a shared Neuron-Level Encoder (NLE), which computes a representation for each neuron from activity, and (2) a decoder that is pre-trained over this inferred representation space. The authors evaluate this framework with two well-established NLEs (plus fixed inter-spike-interval features) and two decoders from existing literature, along with a newly introduced decoder called Time-Patch, which preserves separate neuron-level tokens at the input to a Transformer backbone. On two electrophysiology datasets, the authors' best-performing model configuration Zenith (NuCLR NLE + Time-Patch decoder), outperforms other zero-shot baselines in decoding whisker motion energy, wheel and running speed, and reward. Furthermore, under the standard regime where decoders can be fine-tuned on the target recordings with the behavior supervision, Zenith shows state-of-the-art performance in a majority of decoding tasks of IBL with only 25% of the fine-tuning trials.

Overall, this paper could be a significant contribution to an important problem. If the authors address the two main concerns listed in the Weaknesses below (the fairness of the comparison to baselines and the missing analysis of failure cases), I would be willing to increase my score.

Critical strengths
  1. Motivation and novelty. This work is well-motivated, targeting an important problem. Additionally, connecting self-supervised neuron-representation learning with neural decoders is novel: while the cited SPINT (Le et al., 2025) also infers neuron identities from activity, it learns them jointly with the decoder rather than using a separately pre-trained NLE as the input space. The paper is also clearly written and well structured.
  2. Strong empirical results. In the zero-shot regime, Zenith outperforms all baselines on all five tasks across both datasets, often by large margins (e.g., reward balanced accuracy on IBL improves from 0.708 to 0.828 over SPINT, and running-speed R2R^2 on Allen VBN from 0.311 to 0.611). In the fine-tuning regime, Zenith with only 25% of the target trials already surpasses the best fully fine-tuned baselines on IBL wheel speed and reward decoding, and with full fine-tuning it matches or exceeds all baselines on the three IBL tasks.
Critical weaknesses
  1. Unequal access to neural data during pre-training. The paper does not establish that the NLEs and the baselines were pre-trained on the same neural activity. The baselines appear to be trained only on short trial windows (1.0 s in IBL, 0.8 s in Allen VBN), while NuCLR is pre-trained on 20 s context windows that include activity outside these trials, and NEMO is pre-trained on waveform templates and ACGs whose time range for training recordings is not specified. It is also not stated whether the test subjects' recordings were withheld from NLE pre-training, or whether the lower-quality sessions "included in pretraining" were used for the baselines as well. I ask the authors to specify the exact data (subjects, sessions, and time ranges) used for NLE pre-training, and ideally to add a control where the NLE is pre-trained only on the same trial windows and training subjects as the baselines.
  2. Little analysis of failure cases. The paper provides almost no analysis of when zero-shot transfer fails. For example, neurons whose NLE embeddings are out of distribution, or are poorly represented by the NLE, may hurt zero-shot performance. Evaluating with subsets of neurons (e.g., grouped by brain region or by distance to the training embeddings) could show whether such neurons exist and how important they are. Additionally, there is no per-subject or per-session analysis, which could quantify the reliability of Zen.
Questions for authors

Please consider the questions below as ways to strengthen the paper, not necessarily as prerequsities for publication.

  1. Table 2 suggests the NLE drives most of the performance: with NuCLR, the gap between POYO and Time-Patch is small. Is the large transformer backbone actually needed? How does a more lightweight decoder on top of NuCLR embeddings (e.g., a linear version of the NDT-Stitch integration with a linear readout) perform?
  2. The NLE is frozen during both decoder pre-training and fine-tuning. Have the authors tried updating the NLE jointly with the decoder either during pre-training or during fine-tuning? This would show whether the frozen, self-supervised embeddings are a strength (better generalization) or a limitation (not optimized for decoding).
  3. Could the authors discuss (either in the paper or here) the feasibility of across-subject zero-shot neural decoding of more complex variables, including stimuli rather than behaviors (e.g., visual stimuli)?
7 comments
AuthorsOpenReview ↗

Rebuttal (1/5)

We thank the reviewer for their careful and constructive review of our manuscript. We are glad that they found our work well-motivated and targeting an important problem, and that they found the connection between self-supervised neuron representation learning and neural decoding novel.

The reviewer’s concerns led us to rigorously test the fairness of our comparisons, and to a closer look at when and why zero-shot transfer fails, which we found especially insightful. In summary, we have:

  • Conducted 3 control experiments that equalize access to neural data between ZENITH and the baselines (Weakness 1).
  • Clarified the data used to pretrain each model (Weakness 1).
  • Categorized and analyzed all 10 of the 117 session-task pairs where ZENITH performs below chance (Weakness 2).
  • Added a paired per-animal and per-session comparative analysis between methods (Weakness 2).
  • Evaluated lightweight decoders on top of NuCLR representations (Q1).
  • Presented our preliminary findings on the feasibility of across-subject zero-shot decoding of visual stimuli (Q3).

We address each concern and question in detail below.


Weakness 1: Unequal access to neural data during pre-training.

Thank you for raising this. We agree that the submission did not state clearly which data each model was pretrained on. We have now specified this in a new Appendix A.3 of the updated manuscript, which lists the subjects, sessions, and time ranges used for NLE pretraining, and which of these sessions are used by each model. We also ran three control experiments to test whether the differences in data access can explain ZENITH’s advantage over the baselines. We have added these as a new Appendix A.4.

To summarize our response: the test animals are not used in any pretraining stage, and the NLEs and the baselines are pretrained on the same animals and sessions, except for 63 IBL sessions which do not pass behavior QC and thus can only be used by models with self-supervised pretraining (NLEs, and pretrained NDT-Stitch and MTM). Through a control experiment, we find that these extra sessions do not explain ZENITH's advantage over the baselines, since its performance stays similar when they are removed from NuCLR's pretraining. We ran two more control experiments in which the baselines are given this activity. SPINT's ID encoder is given 20s windows, and NDT-Stitch is pretrained on whole recordings including inter-trial periods. We find that neither of them improves with these changes. This suggests that ZENITH's advantage comes from what NuCLR learns from this activity, and not simply from having access to inter-trial periods.

Table R3-1: Data used for NLE pretraining (NuCLR and NEMO). Also added as Table A2 in Appendix A.3.

IBL BWMAllen VBN
Pretraining animals12246
Pretraining sessions391 (includes 63 sessions that do not pass behavioral QC, discussed below)88
Time range within a sessionWhole recording (~1 hr/session)Whole active task block (~1 hr/session)

It is also not stated whether the test subjects' recordings were withheld from NLE pre-training [...]

The test animals’ recordings were withheld from all pretraining stages, including NLE pretraining, and are only used at evaluation. We have added this to Section 4.1.

It is also not stated [...] whether the lower-quality sessions "included in pretraining" were used for the baselines as well.

We apologize for the confusion caused by this sentence. We have updated the IBL dataset appendix (Appendix A.1) with clearer language. As summarized above, only the 63 IBL sessions that do not pass behavior QC differ, and they are only used by models with self-supervised pretraining. (NuCLR, NEMO, and the pretrained NDT-Stitch and MTM of Section 4.5). We additionally perform a control experiment below by removing these sessions from ZENITH’s pretraining.

Control: We pretrained NuCLR only on the same 328 sessions (106 animals) used by the supervised baselines, and then retrained the Time-Patch decoder on top of it. We find that the performance stays similar (Table R3-2). Whisker motion energy increases from 0.163 to 0.184 R², wheel speed decreases from 0.262 to 0.249 R², and reward remains unchanged. With the sessions and animals matched, ZENITH still outperforms SPINT on all three tasks.

Table R3-2: Control experiment 1. ZENITH with NuCLR pretrained only on the baselines' pretraining sessions (IBL). Performance reported as mean ± SEM over 3 seeds.

MethodNLE pretraining sessionsWhisker ME R²Wheel Speed R²Reward bal. acc.
SPINT*–0.079 ± 0.0130.222 ± 0.0010.744 ± 0.008
ZENITH, matched sessions3280.184 ± 0.0020.249 ± 0.0010.823 ± 0.002
ZENITH3910.163 ± 0.0040.262 ± 0.0020.828 ± 0.006

*Note: We have improved the performance of SPINT baseline. For details, please see our answer to Q2 of reviewer c69y.

AuthorsOpenReview ↗

Rebuttal (2/5)

Weakness 1: Unequal access to neural data during pre-training. (Continued)

I ask the authors to [...] ideally to add a control where the NLE is pre-trained only on the same trial windows [...] as the baselines.

Thank you for this suggestion. We agree the NLEs and baselines should be compared with access to the same neural activity. To control for this, we pretrain representative baselines with the same activity available to the NLEs during pretraining, including inter-trial periods and longer context windows. We chose this over restricting the NLEs to 1 s/0.8 s trial windows, which would depart from the original NLE pretraining procedures, where each neuron is identified from long spans of activity.

Zero-Shot Comparisons: It is true that the NLEs are pretrained on inter-trial periods, which many of the baselines cannot make use of. However, we would like to note that SPINT's ID encoder also draws its 1s/0.8s windows from the same time range as the NLEs, and not just from the trial periods. So SPINT, which is the strongest zero-shot baseline, already has access to this activity. We performed an additional control experiment, and gave SPINT’s ID encoder 20s windows (matching NuCLR’s window size), and found that its performance does not increase.

Table R3-3: Control experiment 2. SPINT with ID encoder window size matched to NuCLR’s (20s) on IBL. Performance reported as mean ± SEM over 3 seeds.

MethodWhisker ME R²Wheel Speed R²Reward bal. acc.
SPINT (1s windows)0.079 ± 0.0130.222 ± 0.0010.744 ± 0.008
SPINT (20s windows)0.068 ± 0.0090.211 ± 0.0040.752 ± 0.005
ZENITH0.163 ± 0.0040.262 ± 0.0020.828 ± 0.006

Finetuning Comparisons: In the case of the pretrained-finetuned analysis of Section 4.5, the baselines pretrained with self-supervised objectives (NDT-Stitch and MTM) are the only ones that can readily pretrain on activity outside the trial windows, since their masked-modeling objective does not require behavioral labels. They are pretrained on the same 391 sessions as the NLEs, but only on the trial periods. We removed this restriction for NDT-Stitch, and pretrained it on windows sampled from the whole recording. With this, NDT-Stitch is pretrained on exactly the same neural activity as NuCLR. We then finetuned it on the test sessions as before. We find that its performance is unchanged on all three tasks (Table R3-4).

Table R3-4: Control experiment 3. Finetuned performance of pretrained baseline NDT-Stitch, when pretrained on trial periods only vs when pretrained on the whole recording (IBL).

MethodWhiskerWheel SpeedReward
NDT-Stitch (pretrained on trial periods only)0.299 ± 0.0030.299 ± 0.0020.852 ± 0.001
NDT-Stitch (pretrained on whole recording)0.302 ± 0.0110.298 ± 0.0060.847 ± 0.004
ZENITH0.350 ± 0.0090.372 ± 0.0030.899 ± 0.010

We hope that the three control experiments above address the reviewer's concern about unequal access to neural data.

Manuscript updates:

  • New Appendix A.3: data used to pretrain each model.
  • New Appendix A.4: three control experiments for differences in pretraining data access.
  • Revised Appendix A.1 to clarify which models use the lower-quality sessions.
  • Revised Section 4.1 to state that test subjects are withheld from all pretraining stages.
AuthorsOpenReview ↗

Rebuttal (3/5)

Weakness 2: Missing analysis of failure cases.

The paper provides almost no analysis of when zero-shot transfer fails.

Thank you for this suggestion. We have performed a comprehensive analysis of failure cases of ZENITH and have added it as Appendix I in the updated manuscript.

Out of the 117 session-task pairs across IBL and Allen datasets, we found 10 cases where the zero-shot performance of ZENITH was lower than chance (R² < 0, or bal. acc. < 0.5). We examined all 10 failures, and found that they fall under three failure modes. We summarize them here, and refer you to Appendix I for a detailed description along with the figures.

  1. Neural activity containing little information about the behavior (8/10 failures). We observe a strong correlation between zero-shot and finetuned performance across the test sessions, and we find that 8/10 failures are sessions that also have low performance after finetuning on that session. This suggests that the relevant behavioral variables may be difficult to decode from these recordings, even with session-specific supervision. However, we cannot rule out limitations of the decoder or fine-tuning procedure as alternative explanations. In Appendix I.1, we additionally analyze the particular instance of the IBL whisker failures, where we find that these sessions have very few neurons that are tuned to whisker motion.
  2. Misleading neuron representations (1/10 failures). This is a case where neurons are poorly represented by the NLE. In one IBL session, zero-shot reward decoding is below chance (0.35), although finetuning on this session reaches 0.98. This suggests that the neural activity contains sufficient information for reward decoding, but the pretrained decoder fails to interpret it. To investigate, we train a linear probe on NuCLR embeddings to distinguish reward-preferring neurons (which fire more after reward) from error-preferring neurons (which fire more after error). When we look at the accuracy of this probe for each test session, we find that this session is the only one where the probe is below chance. That is, in this session many error-preferring neurons actually look like reward-preferring ones in the embedding space. To test this hypothesis, we attempt to correct the neuron embeddings by shifting them along the reward-preference direction identified by the linear probe (see Appendix I.2 for details). With this correction, zero-shot reward balanced accuracy improves from 0.35 to 0.62. This suggests that the decoder reads whether a neuron is reward- or error-preferring from its embedding, and that in this session the faulty embeddings mislead the decoder. We do not yet know why NuCLR represents this session poorly.
  3. Out-of-distribution behavior (1/10 failures). The remaining failure is the running speed decoding on one Allen VBN session. We observe that the running speed of this mouse is much higher than what is seen during pretraining: it runs at 80 cm/s on average, while the fastest pretraining animal averages 54 cm/s. Upon looking at what the decoder predicts for this session, we find that the decoded speed follows the changes in the actual speed from trial to trial, but it is different in terms of raw magnitude. So the failure likely happens because the decoder needs to predict speeds that it has rarely seen during pretraining, and it does not predict their absolute value correctly.

For example, neurons whose NLE embeddings are out of distribution, or are poorly represented by the NLE, may hurt zero-shot performance. Evaluating subsets of neurons [...] how important they are.

In addition to the analyses presented above, we also tried this suggestion. However, we find that NuCLR embeddings of test neurons are not really out-of-distribution from pretraining neurons in any trivially measurable way (cosine or euclidean distances). We also attempted the idea of ablating one brain region at a time (Cosmos atlas), but found that this did not help recover any failing session. Regardless, we agree these were good hypotheses for failure modes, and we thank the reviewer for suggesting them to us.

Additionally, there is no per-subject or per-session analysis, which could quantify the reliability of Zen.

The failure case analyses partly answer this question: ZENITH obtains below chance performance in only 10 out of 117 session-task pairs. We also perform another comparative per-subject and per-session analysis, comparing ZENITH with NuCLR + POYO, NuCLR + NDT-Stitch, and SPINT, to see if the performance gains of ZENITH are reliable across subjects and sessions. This analysis has been added as Appendix H in the updated manuscript.

Manuscript updates:

  • New Appendix I: failure analysis of all 10 failing session-task pairs, with Figures A3-A6.
  • New Appendix H: paired per-animal and per-session comparative analysis between methods.
Reviewer_RnsjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ZEN, a framework for cross-subject zero-shot decoding of spike-sorted neural activity. ZEN replaces recording-specific neuron parameters with representations inferred by a pretrained neuron-level encoder, allowing a shared decoder to operate on unseen neural populations without target behavioral labels, alignment, or parameter updates. The paper also introduces Time-Patch, a tokenization scheme that preserves separate neuron-level tokens throughout the Transformer backbone. The strongest configuration, ZENITH, combines NuCLR representations with Time-Patch. Experiments on the IBL Brain-Wide Map and Allen Visual Behavior Neuropixels datasets demonstrate improved zero-shot decoding across held-out subjects, alongside data-efficient supervised finetuning.

Critical strengths

1. A compelling approach to transferable neuron identity.

The paper connects neuron-level representation learning with decoder pretraining, replacing recording-specific parameters with representations inferred from neural observations. This directly addresses a central obstacle in transferring spike-sorted neural decoders: neurons in a new recording have no established correspondence with those seen during training. Enabling inference without target behavioral labels or parameter updates makes this a practically meaningful contribution toward reusable neural decoders.

2. Substantial improvements under a demanding evaluation setting.

The evaluation uses subject-disjoint splits on two large-scale datasets with acute probe insertions, where each session samples a new neural population and stable electrode or neuron correspondence cannot be assumed. ZENITH improves over the strongest reported zero-shot baseline across all five tasks; for example, IBL reward balanced accuracy increases from 0.708 to 0.828, and Allen running-speed decoding improves from an R² of 0.311 to 0.611. Its strong performance with limited supervised finetuning further supports the practical value of the learned representations.

Critical weaknesses

1. Limited evidence beyond the evaluated recording setting.

Both benchmarks involve mouse acute recordings and transfer within the same dataset. Although acknowledged by the authors, this limits the evidence for applicability to other species and chronic recording settings, including common primate motor-decoding applications.

2. The source and robustness of the representation gains remain insufficiently characterized.

The empirical gains are convincing, but their interpretation remains limited. It is unclear which properties of population context remain transferable when the recorded neurons change across subjects. Moreover, NEMO and NuCLR differ in their inputs, architectures, and training objectives, so their comparison does not isolate the contributions of waveform and single-neuron statistics versus population activity.

Questions for authors
  1. How should the conclusions be scoped to mouse acute recordings? Please discuss the expected challenges in primate or chronic recordings. Results in such settings, if already available, would strengthen the evidence for broader applicability.
  2. Could the authors clarify what shared structure in population activity allows NuCLR to contextualize unseen neurons across subjects, despite changes in the recorded population?
  3. Additional controlled experiments, or a more detailed discussion, would help distinguish the contributions of waveform and single-neuron statistics from those of population activity, given the architectural and training differences between NEMO and NuCLR.
6 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

We thank the reviewer for their feedback and insightful questions. We are glad that they found ZEN to be a compelling approach to transferable neuron identity and a practically meaningful step toward reusable neural decoders.

The reviewer’s questions helped us extend our evaluation and better characterize where our gains come from. In summary, we have:

  • Conducted zero-shot cross-session decoding experiments on chronic Utah array recordings from two nonhuman primates (Q1).
  • Added three controlled ablations that separate the contributions of waveform information, single-neuron spiking statistics, and population context to zero-shot decoding (Q3).

We address each question in detail below.

AuthorsOpenReview ↗

Rebuttal (2/4)

Q1. How should the conclusions be scoped to mouse acute recordings? Please discuss the expected challenges in primate or chronic recordings. Results in such settings, if already available, would strengthen the evidence for broader applicability.

Thank you for this question. While ZEN, as a framework, is general and applicable to chronic settings, the main motivation of our work was to address the neuron-identity problem which is most prevalent in acute recordings. Chronic recordings (such as non-human primate recordings) instead provide electrode indices that remain fixed for the lifetime of the implant. This partly resolves the neuron-identity problem, and decoders in this setting often use electrode-level identities instead. Our main experiments therefore focus on acute mouse recordings. However, the conclusions are not specific to mice or to acute probes. They apply whenever neuron correspondence across recordings is missing, which also includes spike-sorted chronic recordings, since sorted units are not matched across sessions even when the electrodes remain fixed. To test this, we evaluated ZEN in the chronic setting, and we find that it remains competitive.

Zero-shot cross-session decoding on primates: Following the reviewer’s suggestion, we are excited to share new results that take ZEN beyond acute mouse recordings. We evaluated zero-shot cross-session decoding of cursor velocity on spike-sorted chronic Utah array recordings from two nonhuman primates performing center-out reaching [1]. Without any target-session based parameter updates, ZENITH achieves strong decoding performance on two animals, and outperforms POYO trained with unsorted multi-units/threshold-crossings (i.e. it operates on electrode identities), which is a strong baseline in this setting.

Our results are summarized below in Table R2-1. In this experiment, we train the models using past sessions (23 for Chewie, and 14 for Mihili) and test them on 6 future held-out sessions each. We refer the reviewer to the new Appendix J in our updated manuscript for more details.

Table R2-1: Gradient-free zero-shot decoding on chronic primate recordings. Performance is reported as R² (mean ± SEM over 3 seeds).

MethodChewieMihili
Linear / pop. avg.0.193 ± 0.0010.060 ± 0.000
GRU / pop. avg.0.201 ± 0.0060.260 ± 0.009
SPINT0.569 ± 0.0280.249 ± 0.086
POYO / unsorted0.806 ± 0.0230.816 ± 0.014
ZENITH (Ours)0.895 ± 0.0070.857 ± 0.004

We note that the NLEs evaluated in our work do not yet incorporate the prior of stable electrode correspondence between sessions in chronic settings. We hypothesize inclusion of such priors in the model architectures and training methodologies could make ZEN-like approaches even more effective. We leave this exploration for future work.

Manuscript updates:

  • We have added a new Appendix section J: Details of the dataset, evaluation protocol, and comparison with baselines on chronic primate recordings.

[1] Perich MG, et. al. (2018). A Neural Population Mechanism for Rapid Learning. Neuron.

AuthorsOpenReview ↗

Rebuttal (3/4)

Q2. Could the authors clarify what shared structure in population activity allows NuCLR to contextualize unseen neurons across subjects, despite changes in the recorded population?

Thank you for this insightful question. While it is difficult to say definitively why population context enables generalization to unseen neurons, our hypothesis is that certain patterns of population activity are shared across animals, even when the recorded neurons differ. This is because the underlying neural circuits have similar organization and perform similar computations across subjects.

For example, neural populations exhibit shared fluctuations associated with movement and arousal [1, 2], and with action initiation and task engagement [3]. Because these signals are distributed across many neurons, they can be observed from different samples of simultaneously recorded neurons, without requiring correspondence between individual neurons.

We hypothesize that NuCLR uses these shared patterns as a common reference to characterize each neuron by its relationship to the surrounding population. For instance, neurons differ in how strongly, and in which direction, their firing follows the overall activity of the surrounding population. This property, known as population coupling, is a stable attribute of a neuron and has been shown to predict its local synaptic connectivity [4]. Importantly, because these relationships are defined relative to shared population activity rather than specific neuron identities, NuCLR can use them to contextualize previously unseen neurons even when the composition of the recorded population changes.


References:

[1] Stringer, C., et. al., (2019). Spontaneous behaviors drive multidimensional, brainwide activity. Science.
[2] Musall, S., et. al., (2019). Single-trial neural dynamics are dominated by richly varied movements. Nature neuroscience.
[3] Steinmetz, N. A., et. al., (2019). Distributed coding of choice, action and engagement across the mouse brain. Nature.
[4] Okun, M., et. al. (2015). Diverse coupling of neurons to populations in sensory cortex. Nature.

Reviewer_c69yOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the lack of correspondence between neurons recorded in different subjects. ZEN replaces recording-specific neuron parameters with representations inferred by a shared neuron-level encoder, allowing a pretrained behavioral decoder to operate on unseen neural populations without target behavioral labels or parameter updates. The authors also introduce Time-Patch, which retains separate neuron-by-time-patch tokens throughout the Transformer backbone. Their strongest configuration, ZENITH, combines the existing NuCLR encoder with Time-Patch. Experiments on the public IBL Brain-Wide Map and Allen Visual Behavior Neuropixels datasets evaluate transfer to held-out mice, together with supervised adaptation on IBL.

Critical strengths
  • A meaningful transfer problem and a reusable formulation. The paper clearly identifies why recording-specific parameters obstruct direct transfer to new neurons. Separating inferred neuron representations from behavioral decoding provides a practical interface for multiple encoders and decoders.

  • Substantial improvements on several zero-shot tasks. ZENITH exceeds the listed external baselines on all five tasks. Relative to SPINT, IBL reward balanced accuracy increases from 0.708 to 0.828, and Allen running-speed R² increases from 0.311 to 0.611. These are meaningful improvements within the evaluated setting.

  • Evidence that the representations matter. The factorial comparison covers three encoders and three decoders. Removing neuron embeddings reduces IBL reward balanced accuracy from 0.828 to 0.626, supporting their importance to the complete system.

  • Benefits extend to supervised adaptation. With 25% of the available finetuning trials, ZENITH reaches approximately 0.36 wheel-speed R² and 0.89 reward balanced accuracy. With full finetuning, it matches the best baseline on whisker motion and improves wheel speed and reward. The method therefore has value even when target labels are available.

Critical weaknesses
  1. The incremental methodological contribution and source of the gains need sharper validation. The strongest representation, NuCLR, is adopted from Arora et al. (2025), while the manuscript itself explains that SPINT already infers input identities from unlabeled activity. Thus, inferring identities instead of learning recording-specific embeddings is not, by itself, sufficient to distinguish the contribution. The relevant advances appear to be the use of separately pretrained representations for behavioral decoding and the proposed tokenization. Table 2 is useful, but comparing ISI, NEMO, and NuCLR changes the input information, architecture, and training objective simultaneously. It therefore does not isolate the contribution of population context. A controlled ablation of that factor: No direct evidence found in the manuscript. Time-Patch also contributes less consistently when paired with the strongest representation: NuCLR + POYO achieves 0.161 versus 0.163 on IBL whisker-motion R² and 0.616 versus 0.611 on Allen running-speed R². This does not invalidate Time-Patch, which performs well with other representations, but it makes a more precise account of its incremental value important. I recognize that combining existing components can produce a valuable contribution; my concern is whether the evidence sufficiently identifies the new insight beyond the improved complete system.

  2. The target-context protocol is insufficiently specified for the central zero-shot claim. The evaluation protocol permits target neural activity to be used for representation inference, and the encoder configuration specifies a 20-second NuCLR context. However, the manuscript does not clearly explain where these windows are drawn relative to each evaluated trial, whether they overlap evaluation windows, or whether embeddings are computed once per recording or refreshed during evaluation. These details determine what information is available at prediction time and how to reproduce the results. A complete target-context construction rule: No direct evidence found in the manuscript. The distinction matters for comparison with NEMO, whose autocorrelograms explicitly exclude spikes from the evaluated period. It is also unclear how NuCLR's context budget compares with the activity windows used by SPINT. Using unlabeled target activity is explicitly allowed by the paper's definition and is not inherently leakage. My concern is that the temporal access conditions are not sufficiently explicit to assess the comparisons and practical scope of the reported transfer results.

  3. The reported uncertainty does not establish how consistently the gains transfer across animals. Tables 1–2 report mean ± SEM over three seeds, whereas the central generalization claim concerns new animals. There are 13 IBL and 8 Allen test subjects, potentially with multiple sessions per subject. Seed variability does not directly quantify between-animal variability, and the presentation does not establish whether gains are broadly shared or concentrated in a subset of subjects. Animal-level paired confidence intervals: No direct evidence found in the manuscript. This limitation is particularly relevant to the smaller differences between NuCLR-based decoders and to the full-finetuning improvements, rather than being a reason to dismiss the larger aggregate gains outright.

Questions for authors

Q1. What is the specific contribution beyond existing activity-derived identities and NuCLR representations?

Please distinguish the contribution of separate representation pretraining from that of population context and Time-Patch, using the existing comparisons wherever possible. One focused control, such as a NuCLR variant without cross-neuron attention using the same data and decoder, would help test the population-context interpretation. Please also qualify the claim of consistently superior Time-Patch performance. I am not requesting a new benchmark suite or treating the absence of an uncited arXiv baseline as grounds for rejection; SPINT is already a central comparison in this submission.

Q2. Exactly how are target neuron representations constructed?

Please provide a timeline or pseudocode specifying the activity available to NuCLR, NEMO, and SPINT for each prediction, including context duration, overlap with evaluated windows, and embedding refresh frequency. If their temporal access differs, a focused comparison using a common, disjoint pre-evaluation context would clarify whether the relative advantage persists. If this is already the implemented protocol, documenting it would resolve much of this concern without a new experiment.

Q3. Are the improvements consistent across test animals?

Please report paired per-animal results for ZENITH versus SPINT and NuCLR + POYO, with confidence intervals obtained by resampling animals while keeping their sessions together. Please specify how session-level metrics are aggregated. This can largely use existing predictions and would establish the robustness of the central comparisons without requiring additional data collection.

7 comments
AuthorsOpenReview ↗

Rebuttal (1/4)

We thank the reviewer for their thorough and constructive review of our manuscript. We are glad that they found the transfer problem meaningful, the ZEN formulation reusable, and the zero-shot and finetuning gains substantial.

The reviewer’s questions helped us sharpen several parts of the paper. In summary, we have:

  • Added two new ablations isolating the contributions of NuCLR’s population context, pretrained neuron representations, and the Time-Patch decoder (Q1).
  • Documented exactly how target neuron representations are constructed, and improved the SPINT baseline so that all methods are evaluated with equal access to target-session activity (Q2).
  • Added a paired per-animal analysis with animal-level bootstrap CIs (Q3).

We address each question in detail below.

AuthorsOpenReview ↗

Rebuttal (2/4)

Q1. What is the specific contribution beyond existing activity-derived identities and NuCLR representations?

Please distinguish the contribution of separate representation pretraining from that of population context and Time-Patch [...] test the population-context interpretation.

Thank you for this insightful question. We conducted two focused ablation studies on the IBL dataset to address this.

First, to isolate the effect of population context, we pretrained a variant of NuCLR without its spatial attention layers, replacing them with temporal layers to match parameter count. We then trained all three decoders on its resulting representations. Our findings are as follows:

  • Removing population context significantly lowers performance on all three tasks for NDT-Stitch and POYO.
  • With Time-Patch we see smaller (but still meaningful) reductions for wheel speed and reward tasks, and it performs similarly on the whisker task (within 1 SEM). This is consistent with our main zero-shot results in Section 4.3, where we find that Time-Patch decoder is more robust to lower quality embeddings. Because Time-Patch does not compress the neuron tokens across the population, the neuron embeddings can attend to each other, which we believe offsets the lack of population context in the embeddings.

Table R1-1: Ablating NuCLR’s population context. Performance reported as mean ± SEM over 3 seeds.

DecoderNuCLR pop. contextWhisker MEWheel SpeedReward
NDT-Stitch✓0.124 ± 0.0050.240 ± 0.0030.812 ± 0.007
✗0.082 ± 0.0030.220 ± 0.0010.737 ± 0.000
POYO✓0.161 ± 0.0030.244 ± 0.0050.801 ± 0.006
✗0.108 ± 0.0070.218 ± 0.0070.752 ± 0.004
Time-Patch✓0.163 ± 0.0040.262 ± 0.0020.828 ± 0.006
✗0.166 ± 0.0030.255 ± 0.0010.810 ± 0.001

Our second new study is an ablation ladder that isolates the sources of the gains from SPINT to ZENITH.

  1. Pretrained representations (row 1 → 2): We start with SPINT (improved baseline, see Q2), and replace its ID encoder with pretrained embeddings from a NuCLR variant without population context. Importantly, the decoder architecture stays fixed, which helps attribute the performance improvement to just use of pretrained embeddings. Additionally, both settings don’t incorporate population context in the embeddings, so that is not a confound as well. We find that this transition mainly improves whisker (+0.048) and reward (+0.027), with a weaker improvement for wheel speed (+0.005).
  2. Population context (row 2 → 3): Then we bring in the pretrained full NuCLR representations which include population context. This mainly improves wheel speed (+0.023) and reward (+0.039), and leaves whisker unchanged.
  3. Decoder (row 3 → 4): Finally, we replace SPINT’s decoder with Time-Patch which is a transformer architecture. This accounts for the remaining gains on all three tasks.

Table R1-2: Ablation ladder going from SPINT to ZENITH. Performance is reported as mean ± SEM over 3 seeds.

#MethodWhisker ME R²Wheel speed R²Reward bal. acc.
1SPINT (improved)0.079 ± 0.0130.222 ± 0.0010.744 ± 0.008
2NuCLR w/o pop. context + SPINT’s Decoder0.127 ± 0.0030.227 ± 0.0010.771 ± 0.006
3NuCLR + SPINT’s Decoder0.127 ± 0.0090.250 ± 0.0050.810 ± 0.006
4NuCLR + Time-Patch (ZENITH)0.163 ± 0.0040.262 ± 0.0020.828 ± 0.006

In summary, each component of ZENITH contributes, but to different tasks. Importantly, the transition from row 1 to row 3 does indeed lead to significant improvements, and confirms our main thesis: usage of pretrained neuron representations is a strong contender for achieving good zero-shot decoding.

Manuscript updates:

  • New Appendix G.2: Population-context ablation study.
  • New Appendix G.3: SPINT → ZENITH ablation ladder.

Please also qualify the claim of consistently superior Time-Patch performance.

We agree that our wording “Time-Patch [...] consistently achieves higher performance” (Section 4.3) requires qualification. As you note, Time-Patch’s advantage is clear with weaker representations such as ISI and NEMO. With NuCLR, it is the best decoder on only 3 of 5 tasks and comparable to the best (within 1 SEM) on the other two. Thus, its benefit is better described as reliability, since it stays close to the best decoder across representations and tasks. The alternatives are less reliable with NuCLR: POYO falls behind on IBL reward (0.801 vs. 0.828 bacc.), and NDT-Stitch falls far behind on Allen running speed (R² = 0.440 vs. 0.616).

Manuscript updates:

  • Revised Section 4.3 with the qualification of Time-Patch’s consistency.
AuthorsOpenReview ↗

Rebuttal (3/4)

Q2. Exactly how are target neuron representations constructed?

Please provide a timeline or pseudocode specifying the activity available to NuCLR, NEMO, and SPINT for each prediction, including context duration, overlap with evaluated windows, and embedding refresh frequency.

Thank you for raising this. We agree that the manuscript did not describe this clearly enough. We have added the exact pseudo code in the updated manuscript in Appendix B.1.

All our NLE-based methods (NuCLR, NEMO, ISI) follow the same protocol:

  1. Test-time neural context: For each target session, the neuron representations are computed from all unlabeled neural activity from the start of the session up to 20s before the first test trial. On average, this is about 37 min of activity per session on IBL and about 38 min on Allen VBN.
  2. No overlap with test trials: The 20s gap makes sure that no neural activity from the test split, or from right before it, is used to build the neuron representations.
  3. No refresh: The representations are computed once per session, before evaluation, and stay fixed for all test trials of that session.

Within this context, each NLE builds its neuron representations as follows:

  • NuCLR: 20s sliding windows over the neural context with 10 s stride. Each neuron's embeddings are averaged over all windows.
  • NEMO: Each neuron's ACG is computed from its spikes in the context.
  • ISI: Each neuron's ISI histogram is computed from its spikes in the context.

We note that the "20s context" listed in the NuCLR hyperparameters (Table A6) refers to the length of a single input window to NuCLR, not the total amount of target activity it sees.

If their temporal access differs, a focused comparison using a common, disjoint pre-evaluation context would clarify whether the relative advantage persists.

This is a great suggestion. SPINT already used activity from the same disjoint pre-evaluation context as our NLEs, but it was given less temporal access in our original evaluation. We have fixed this in our updated manuscript, and describe the details below.

SPINT's ID encoder builds each neuron's identity by encoding many short windows of that neuron's activity and averaging them. During training, the number of windows is limited by GPU memory, so SPINT is trained with a small fixed number of windows per session. In the original submission, we used the same fixed number of windows at evaluation as well, which gave SPINT less temporal access than our NLEs.

We have now improved the SPINT baseline by matching its context length to our NLEs at evaluation, along with some hyperparameter tuning (number of training windows). For each target neuron, the ID encoder now averages over all non-overlapping windows in the full context available to all NLEs. This improves SPINT's performance on both datasets: on IBL (Table R1-3), and on Allen VBN from 0.086 to 0.115 R² on running speed and from 0.792 to 0.807 balanced accuracy on reward. While re-evaluating SPINT, we also found that the running-speed value in our submission (0.311) had a copying error, and it was actually 0.086. We have confirmed that no other reported numbers are affected, and we have updated Table 1 with the improved SPINT results. ZENITH's advantage persists on both datasets.

We also tried giving SPINT's ID encoder 20s windows, to match the window length used by NuCLR. This did not help overall. This shows that the gap between SPINT and ZENITH is not explained by the amount of target context, or by how that context is windowed.

Table R1-3: SPINT with matched neural calibration context on IBL. Performance reported as mean ± SEM over 3 seeds.

MethodWhisker ME R²Wheel Speed R²Reward bal. acc.
SPINT (previous evaluation)0.050 ± 0.0080.223 ± 0.0010.708 ± 0.007
SPINT (matched context, 1s windows)0.079 ± 0.0130.222 ± 0.0010.744 ± 0.008
SPINT (matched context, 20s windows)0.068 ± 0.0090.211 ± 0.0040.752 ± 0.005
ZENITH0.163 ± 0.0040.262 ± 0.0020.828 ± 0.006

In summary, all methods were already evaluated with neuron representations built from a disjoint pre-evaluation context. With our improved SPINT evaluation, all ZEN methods and SPINT are now also evaluated with equal temporal access to the target session.

Manuscript updates:

  • New paragraph in Appendix B.1 describing target-session NLE context details, with pseudocode.
  • Updated SPINT results in Table 1.
  • New paragraph in Appendix D.5 describing SPINT’s matched-context evaluation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion period

Dear Area Chair,

We thank you and the reviewers for the time and effort spent on our submission. We are glad that all three reviewers recognized the importance of the problem and the merits of our approach: “connecting self-supervised neuron-representation learning with neural decoders is novel” (sYzL), “a meaningful transfer problem and a reusable formulation” (c69y), and “a compelling approach to transferable neuron identity” (Rnsj). All three reviewers also highlighted the strong empirical results, both in the zero-shot setting and with few-shot finetuning. As the discussion period comes to an end, we would like to briefly summarize where the discussion stands, and the updates made to the manuscript.

Outcome of the discussion. All reviewers engaged with our responses, and two of them raised their rating. All three ratings are now in favor of acceptance (3, 3, and 4) :

  • Reviewer sYzL raised their rating from Weak Accept (3) to Accept (4), stating that our responses “fully addressed my concerns.” They also suggested that we add the new image-identity decoding experiment to the paper. We agree with this suggestion, and we will add this result along with a discussion to the final version of the manuscript.
  • Reviewer c69y raised their rating from Weak Reject (2) to Weak Accept (3), stating that “the new ablations, matched-context comparison, and per-animal analysis address my main concerns.”
  • Reviewer Rnsj (rating: Weak Accept, 3) raised two weaknesses in their initial review: limited evidence beyond acute mouse recordings, and that the sources of representation gains were not sufficiently characterized. We addressed both with new experiments: zero-shot across-session decoding on chronic recordings from two nonhuman primates (Appendix J), and ablations that separate the contributions of waveform, single-neuron spiking statistics, and population context (Appendix G). In their follow-up, the reviewer noted that the new results “broaden the evaluation to chronic primate recordings”, and asked that we make explicit that they evaluate cross-session and not cross-subject transfer, and that we clarify our code release plan. We have since revised Appendix J accordingly, and the reproducibility statement now says that the code and pretrained checkpoints will be publicly released upon acceptance.

Manuscript updates. In response to reviewer feedback, the following are included in the revised PDF:

  • Sources of the gains (c69y, Rnsj, sYzL; Appendix G): We have expanded our ablation analysis, and added ablations to characterize the performance gains from each component of our approach. We find that the pretrained neuron representations, population context, and the proposed Time-Patch decoder each contribute to the final performance, and the improvements vary across tasks. We have also qualified our claim about Time-Patch in Section 4.3, and now describe its advantage as reliability across representations and tasks rather than consistent superiority.
  • Fairness of the comparisons (c69y, sYzL; Appendix A.3, A.4, B.1 and D.5): We extended the documentation for how target neuron representations are constructed and which data each model is pretrained on. We also strengthened the SPINT baseline by giving it the same target-session context as our NLEs, and ran three control experiments that equalize access to neural data. We find that ZENITH’s advantage persists under all controls.
  • Reliability of comparisons across animals (c69y, sYzL; Appendix H): We added paired comparisons with 95% confidence intervals obtained by bootstrapping over held-out animals. ZENITH is significantly better than the strengthened SPINT baseline on all five tasks, and it is better on 49 of the 55 animal-task pairs.
  • Failure case analysis (sYzL; Appendix I): We analyzed all 10 of the 117 session-task pairs where zero-shot ZENITH performs below chance. We found that they fall under three failure modes, and reach interpretable conclusions for each of them.
  • Beyond acute mouse recordings (Rnsj; Appendix J): We added zero-shot cross-session decoding experiments on chronic recordings from two nonhuman primates, where ZENITH achieves the strongest performance on both animals.

We believe that the reviewers’ feedback has made the paper considerably stronger, more rigorous, and also more precise about where the approach currently reaches its limits. We are grateful for it, and we would be happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.