acceptodds
Under review as a conference paper at ICLR 2027

NeuroLens: Learning Latent Embeddings of Neural Semantics from Chronic Recordings

Abstract

Understanding how neural activity represents higher-order cognition and how these representations evolve over time has long been a central pursuit in neuroscience. However, current analytical tools cannot easily distinguish representational plasticity from recording instability in chronic neural recordings. Here, we introduce NeuroLens (Latent Embeddings of Neural Semantics), a self-supervised model based on the Joint-Embedding Predictive Architecture (JEPA) framework that learns denoised, semantically informative latents from chronic neural recordings. An adaptive encoder maps changing neural populations into a common latent space, while a temporal predictor learns structure that supports prediction of future latent states. By predicting in latent space, NeuroLens captures temporally predictable structure and reduces sensitivity to transient, recording-specific variability. Across chronic intracortical data in mice and humans, the learned representations improve decoding of decision-making and semantic task variables. Multi-day pretraining enables generalization to future sessions, rapid few-shot adaptation to unseen neural populations, and more stable decoding over time than state-of-the-art baselines. Together, these results establish NeuroLens as a new paradigm for studying how neural representations change during learning and over long timescales.

Primary area:
applications to neuroscience & cognitive science
Keywords:
chronic neural recordings, neural decoding, brain-computer interfaces, self-supervised learning, representation learning, foundation models, transformer, electrophysiology, neuropixels

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_apL3OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a causal transformer architecture with a SPINT-like neural tokenization scheme and a LeJEPA-based pretraining strategy. The authors show that the proposed architecture outperforms several baseline methods across decoding tasks and provides better stability across recording days.

Critical strengths
  • The paper is clearly written; it is easy to follow the motivation, methodology, and evaluation setup.
  • I appreciate that the authors included a decent number of baseline methods for performance comparisons and evaluated their model across multiple datasets.
Critical weaknesses
  • Even though the authors considered several baseline methods for comparison, I think models specifically developed for across-day stability could be considered beyond SPINT, for which no results could be shown for human data. For example, it would be interesting to see how ADAN or NoMAD perform in the authors' evaluation setup.
  • It is not clear which component in the formulation results in stable ranks across recording days and improved performance. The ablation studies show similar performance across variants and no direct comparison with baselines.
  • The results include no statistical tests, which is important since the performance margins seem pretty close.
  • Stability results across multi-unit vs. single-unit spikes can show significant performance differences. For example, NoMAD uses multi-unit spikes, which provides a more stable baseline for the aligner network compared to sorted spikes. I am curious how the proposed method would compare to NoMAD for the motor datasets used in the NoMAD paper across single-unit and multi-unit spikes. Even though the authors have two datasets in their manuscript, I think including NHP motor datasets and comparing with other methods such as NoMAD would significantly improve the manuscript.
Questions for authors

What is fundamentally different between day-specific FiLM (used for human recordings) and projecting FR, ACG, and PC into token space? Are the authors using the same FR, ACG, and PC statistics for FiLM conditioning? If so, is the main difference the inductive bias of creating a pathway for the model to leverage these statistics through modulation if needed, rather than directly encoding them?

4 comments

We thank the reviewer for recognizing our thorough baseline comparisons and our approach’s generalizability across multiple datasets. We also appreciate your suggestions to compare additional stable decoders, include chronic NHP motor datasets, and clarify day-specific FiLM, which we address below. Revisions are highlighted in blue in the updated paper.

Summary: Before answering your questions, we would like to highlight our novelty and contributions through our newly added analyses:

  1. Scientific: NeuroLens models neural dynamics, capturing learning-related representational changes (updated Figure 4 and Section 5.3).
  2. Performance: It also supports stable decoding, ranking 1st among non-ensemble methods across all few-shot adaptation settings on the FALCON benchmark’s M2 track [1] and more than doubling the strongest zero-shot baseline’s held-out decoding R2 (updated Table 1 and Section 5.2).

[1] Benchmark website: https://snel-repo.github.io/falcon/

M2 track: https://eval.ai/web/challenges/challenge-page/2319/leaderboard/5751


W1. Comparison with NoMAD, SPINT, and other baselines on the FALCON benchmark’s NHP motor dataset across zero-shot, few-shot unsupervised, and few-shot supervised adaptation settings

We thank the reviewer for this question. To compare NeuroLens with NoMAD, SPINT, and other well-known stable decoding methods on this task, we evaluated NeuroLens on the FALCON benchmark [1] M2 track (updated Section 5.2 and Table 1). We chose the M2 track because the task there is to predict 2-D finger velocity from Utah array multi-unit spiking activity recorded from a non-human private (NHP) monkey from motor brain areas.

At the time of submission, NeuroLens ranks 1st among non-ensemble methods across all few-shot adaptation settings: (1) zero-shot (ZS), which uses no held-out calibration data; (2) few-shot unsupervised (FSU), which uses only calibration neural activity; and (3) few-shot supervised (FSS), which uses both calibration neural activity and behavioral labels. NeuroLens flexibly supports all 3 adaptation settings using a unified model. Here, we summarize the results for NeuroLens, NoMAD, CycleGAN, SPINT and NDT2; see Sec 5.2 for more details.

The results are:

  • NeuroLens ranks 1st among non-ensemble methods across ZS, FSU, FSS settings.
  • NeuroLens achieves remarkable zero-shot performance (R2=0.16), substantially exceeding the strongest ZS baseline (R2=0.06) and approaching NoMAD's FSU performance (R2=0.2) without using any held-out calibration data.
  • With unlabeled calibration data, NeuroLens improves decoding R2 to 0.44 through gradient-free adaptation, substantially outperforming the strong FSU baseline SPINT (R2=0.26) and surpassing NDT2 Multi's FSS performance (R2=0.43).
  • With labeled calibration data, NeuroLens achieves an R2 of 0.53, approaching the oracle reference (R2=0.58) while using only 10% of each held-out session for calibration, compared with the oracle's use of 60% private labeled calibration data, showing greater data efficiency.

These results highlight NeuroLens's strong zero-shot generalization and effective adaptation with limited calibration data.

Table 1

MethodZero-shot (ZS)Few-shot unsupervised (FSU)Few-shot supervised (FSS)
Non-ensemble methods (10% calibration)
RNN-0.07 ± 0.23----
Wiener filter0.06 ± 0.04----
NoMAD + WF--0.20 ± 0.10--
CycleGAN + WF--0.22 ± 0.06--
SPINT (GF)--0.26 ± 0.13--
PRISM--0.27 ± 0.13--
OMEN----0.41 ± 0.08
S5 SSM----0.41 ± 0.11
NDT2 Multi----0.43 ± 0.08
NDT3----0.48 ± 0.06
TaskFuse----0.50 ± 0.11
PAR----0.53 ± 0.10
NeuroLens (ours)0.16 ± 0.160.44 ± 0.100.53 ± 0.08
Ensemble methods (10% calibration)
TCN + GRU ensemble----0.60 ± 0.07
Private-label oracle references (60% calibration)
NDT2 Multi----0.58 ± 0.04

W2. Clarification of day-specific FiLM

We thank the reviewer for the opportunity to clarify the motivation for day-specific FiLM. Fundamentally, both statistical neuron-ID embeddings and day-specific FiLM support adaptation to unseen neural populations, but address different sources of variability: statistical neuron IDs accommodate changes in the recorded neurons through matching their functional similarity, whereas FiLM infers and compensates for day-specific drift.

Mechanistically, statistical neuron ID embeddings leverage the firing properties of neurons (FR, ACG, and PC) to capture functional similarity that can transfer across sessions without knowing a neuron’s fixed identity. Once we embed these neurons’ firing signatures into the same latent space, similarly behaving neurons are mapped to similar locations in the token space.

We do not use this approach for channel-based Utah array recordings because (1) each channel records from multiple units, mixing their firing properties and making their signatures less reliable than those of single units; (2) it performs poorly empirically, as shown in the new ablation study summarized below (also in Appendix J and Table 7 of the updated paper); and (3) prior work on Utah array-based human BCI recordings commonly models per-day drift using day-specific affine transformations [2, 3].

Therefore, we propose per-day FiLM, a nonlinear extension of a per-day affine transformation, for multi-unit data. The MLP in per-day FiLM contains multiple linear layers, each with a hypernetwork trained to predict the parameters of each linear layer. When the model encounters drift on a new day, it can infer the parameters needed to compensate for the drift based on patterns learned during training.

Empirically, we compare statistical neuron IDs, per-day FiLM, and per-day linear transformations on human BCI data and summarize the ablation results below.

Table 2

Day adaptation methodPER (%) ↓Top-5 accuracy (%) ↑
Per-day FiLM (calibration statistics)38.8029.80
Per-day linear FiLM49.2022.10
Statistical neuron ID65.007.90

We have added this design rationale and the ablation results to the updated paper.

[2] Willett, Francis R., et al. "High-performance brain-to-text communication via handwriting." Nature 593.7858 (2021): 249-254.

[3] Willett, Francis R., et al. "A high-performance speech neuroprosthesis." Nature 620.7976 (2023): 1031-1036.


Other Questions:

  1. The results include no statistical tests, which is important since the performance margins seem pretty close.

For the quantitative results in the main paper, we provided boxplots summarizing each baseline’s performance distribution. To further address your concerns, we evaluated NeuroLens on the FALCON benchmark’s M2 track. We provide an anonymized codebase to reproduce these benchmark results here:

https://anonymous.4open.science/r/LATTE-4FA2/README.md

  1. Are the authors training the decoders for each held-out recording day, or are they using the same decoder fitted on the source day across GF, GB and full-finetune strategies? If not, I would like to see how the results look like when decoders are fixed, especially for the GB scenario which truly evaluates the zero-shot setting.

Thank you for the question. Because the IBL chronic mouse Neuropixels dataset is not yet publicly available for independent replication, we evaluated your proposed zero-shot setting on the public FALCON benchmark’s M2 track [1] and provided an anonymized codebase to reproduce our results. In this setting, we kept both the encoder and decoder fixed without calibration. As shown in Table 1, NeuroLens achieves strong zero-shot performance (R2=0.16), substantially exceeding the strongest ZS baseline (R2=0.06).

  1. It is not clear which component in the formulation results in stable ranks across recording days and improved performance. The ablation studies show similar performance across variants and no direct comparison with baselines.

We thank the reviewer for helping us clarify the contributions of individual model components. We expanded our ablations to compare statistical and learned neuron IDs (Appendix J), compare adaptation methods for channel-based neural recordings (Appendix K), and evaluate pretraining and calibration data size requirements (Appendix L).

I thank the authors for their rebuttal, I will adjust my score accordingly. That said, I would suggest avoiding words like "strong" and "remarkable" for 0.16 R2 metric for behavior decoding, even though I acknowledge it indicates a better performance compared to baseline methods, and the low-value is potentially dataset specific rather than model specific.

Thank you again for your constructive feedback and for raising your scores! We will revise the paper to improve the precision of our wording when describing the performance gains.

Reviewer_u7KEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NeuroLens, a JEPA-based self-supervised framework for learning neural representations from chronic recordings. It combines a day-adaptive encoder with a shared autoregressive Transformer that predicts future latent embeddings, using SIGReg to prevent representation collapse. The mouse encoder uses calibration-based neuron identification from firing-rate statistics, autocorrelograms, and population coupling, while human channel-level recordings use calibration-conditioned FiLM adaptation.

Experiments on chronic mouse Neuropixels and human intracortical recordings demonstrate improvements in behavioral and sentence-embedding decoding, as well as stable representations across recording days.

Critical strengths

1. An important problem with a reasonable methodological design.

The paper addresses recording nonstationarity in longitudinal neural recordings. Combining calibration-based neuron identification with latent-space prediction provides a potentially useful approach to adapting pretrained models to changing neural populations.

2. Evaluation on chronic neural recordings with decoding improvements.

The experiments cover both mouse and human intracortical recordings, with chronological held-out-day evaluations and comparisons against several established decoding methods.

NeuroLens achieves improvements in choice and movement decoding on future mouse recording days, reaching 0.83 balanced accuracy and 0.40 movement R², compared with 0.78 and 0.37 for the strongest reported baselines, respectively. It also improves sentence-embedding retrieval on human recordings.

Critical weaknesses

The main weaknesses are summarized below, with detailed questions in the Questions for Authors section.

1. Insufficient evidence for separating genuine representational changes from recording nonstationarity.

The paper explicitly emphasizes preserving biological changes in neural representations and concludes in Section 5.3 that NeuroLens can “separate task-related changes from recording nonstationarity”. However, the experiments primarily demonstrate improved decoding and greater cross-day representational stability, without directly testing whether genuine learning-related changes in neural geometry are preserved. This evidence appears insufficient to support the claimed separation.

Because JEPA emphasizes temporally predictable structure, it may also suppress qualitative changes in neural representations during learning. Furthermore, the Discussion acknowledges that the authors do not disentangle learning- and behavior-related changes from intrinsic neural changes, raising further questions about the claimed separation and its biological interpretation.

2. Insufficient evaluation of practical adaptation and computational efficiency.

For longitudinal BCI decoding, the amount of historical data and daily calibration required, as well as real-time inference efficiency, are important practical considerations.

However, NeuroLens is evaluated with a fixed 39-day pretraining history and 64 calibration trials, without systematically examining how performance depends on the amount of historical data or target-day supervision. Online inference efficiency is also not reported, leaving the practical data and computational requirements insufficiently characterized.

Questions for authors

Major Questions

Q1. Preservation of genuine learning-related representational changes.

Section 5.3 concludes that NeuroLens separates task-related changes from recording nonstationarity. However, Figure 2C shows changes in choice-related trajectories using models trained separately for each day, while Figure 4 primarily measures cross-day stability and consistency through representational similarity and effective rank. Neither establishes whether genuine learning-related changes are preserved rather than suppressed.

Could the authors evaluate the cross-day pretrained NeuroLens on real longitudinal data with independently characterized learning-related changes in neural geometry, such as the progressive orthogonalization of hippocampal task-state representations reported by Sun et al. (Nature, 2025)? Does NeuroLens preserve the qualitative changes and their progression across learning, rather than merely producing more consistent representations across days?

The Discussion also acknowledges that learning- and behavior-related changes are not disentangled from intrinsic neural changes. Could the authors clarify how this limitation relates to the separation claim in Section 5.3?

Q2. Pretraining requirements, calibration efficiency, and inference cost.

Given the intended application to longitudinal BCI decoding, where calibration burden and real-time inference efficiency are important, could the authors systematically evaluate:

  • Decoding performance as the number of historical pretraining sessions increases, rather than only comparing 39-day pretraining against training from scratch.
  • Performance as a function of labeled target-day adaptation trials and unlabeled calibration trials, comparing gradient-free and gradient-based adaptation with existing baselines.
  • Online inference time, particularly relative to real-time decoding methods such as POSSM, under comparable hardware conditions.

These evaluations would clarify the actual data requirements and computational advantages of NeuroLens for practical BCI deployment.

Q3. Contribution of statistical neuron-identity embeddings.

Section 4.1 states that Appendix H demonstrates comparable performance between statistical neuron identities and learned ID embeddings. However, Appendix H only compares models with and without neuron-identity embeddings. Could the authors provide the stated learned-ID comparison, particularly regarding adaptation to unseen neurons?

Additionally, could the authors separately ablate firing-rate statistics, autocorrelograms, and population coupling in statistical neuron identities to clarify their individual contributions?

Minor Question

1. NDT2 baseline.

NDT2 explicitly addresses multi-session pretraining and adaptation to new recording contexts, with architectural differences from the original NDT. Could the authors explain why NDT2 was not included as an observation-space self-supervised baseline, given its close relevance to the paper's objective?

2. Inconsistent ablation results.

For session 2023-05-24, Appendix F Table 1 reports choice balanced accuracy of 0.61 for pretrained NeuroLens, whereas Appendix H Table 2 reports 0.88, despite both reporting movement R² of 0.44. Please clarify whether the experimental settings differ or whether this is a reporting error.

3. Code availability and reproducibility.

Could the authors provide the anonymized code to facilitate reproducibility of the reported results?

The reviewer would update the score if reasonable answers are provided.

6 comments

We thank the reviewer for recognizing the importance of the problem and the strengths of our approach, and for the constructive feedback. We have addressed your concerns through additional experiments, as detailed below. Revisions are highlighted in blue in the updated paper.

Summary: We show the following contributions:(1) Scientific: NeuroLens preserves learning-related neural changes. (2) Performance: It ranks 1st among non-ensemble methods in FALCON benchmark M2 track across all few-shot adaptation settings, and more than doubles the strongest zero-shot baseline’s decoding performance. (3) Practical: We report pretraining and calibration requirements and show that statistical neuron-ID embeddings improve adaptation to unseen neurons.


Q1. Preservation of genuine learning-related representational changes

We thank the reviewer for the opportunity to show that NeuroLens latents preserve learning-related changes. Following the analytical approach of Sun et al. [1], we added longitudinal analyses of chronic Neuropixels recordings from an IBL mouse (updated Section 5.3 and Figure 4).

These analyses provide three complementary findings:

Learning-stage structure: We visualized cross-day neural activity similarity as a heatmap, which reveals 3 clusters corresponding to the behaviorally characterized debiasing, novice, and expert learning stages (Figure 4A). We then performed a coefficient of partial determination analysis of wheel kinematics [1], showing that these stages unfold in successive, overlapping waves (Figure 4B–D). Finally, we applied the same similarity analysis to NeuroLens latents and found that they preserve the stage-dependent neural structure (Figure 4A).

Progressive choice-related decorrelation: We fitted encoding models to isolate choice-related components in the neural activity, and then compared population patterns between left- and right-choice trials in neural activity and NeuroLens latents. Choice-related neural patterns become progressively decorrelated across the 3 learning stages, and NeuroLens latents preserve this progression toward orthogonalization (Figure 4F).

Task-axis reorganization: We further used NeuroLens latents to study how neural activity reorganizes during learning. We measured each day’s stimulus, choice, and feedback axis angles in neural latent space relative to the corresponding axis averaged across expert-stage days. Stimulus and choice axes reorganize most rapidly during the novice stage and stabilize during expert performance, whereas feedback-axis reorganization begins during debiasing (Figure 4E).

These findings support preservation of gradual, task-related representational changes in NeuroLens latents. However, they do not establish full disentanglement of learning from recording instability. We have therefore revised the separation claim in Section 5.3 and throughout the paper, and explicitly acknowledged this limitation in the Discussion. [1] Sun, Weinan, et al. "Learning produces an orthogonalized state machine in the hippocampus." Nature 640.8057 (2025): 165-175.



Q2. Pretraining requirements, calibration efficiency, and inference cost

We thank the reviewer for raising these practical considerations. We have conducted additional experiments and participated in the FALCON benchmark [2] to address these questions. We summarize our findings below. Full details are provided in (1) Appendix L and Figures 6–7 and (2) Section 5.2 and Table 1 in the revised paper.

Summary of Experiments:

Pretraining requirements (Appendix L.1): We evaluate decoding performance using 5, 10, 20, and 39 historical recording days to see how performance scales with pretraining data size.

Calibration efficiency (Appendix L.2 and Section 5.2): We measure decoding performance with varying amounts of calibration data to compare gradient-free and gradient-based adaptation. Also, we evaluate NeuroLens on the FALCON benchmark M2 track (Section 5.2 and Table 1) under zero-shot, few-shot unlabeled, and few-shot labeled settings to further assess calibration efficiency. See our response to Minor Question 1 for details.

Inference cost: We evaluate NeuroLens on the FALCON benchmark M2 track to assess its computational cost at inference time.

Summary of Findings:

Pretraining requirements (Fig. 6): Increasing the number of pretraining days improves decoding performance after adaptation to held-out days. We observe a ~10% improvement in decoding metrics when increasing the pretraining data from 5 to 39 days.

Calibration efficiency (Fig. 7 and Table 1): Decoding performance improves with increasing amounts of labeled calibration data for both gradient-free and gradient-based adaptation. Furthermore, the FALCON benchmark results show that NeuroLens, using only 10% of each held-out session for calibration, approaches the performance of the oracle reference decoder, which uses 60% private labeled calibration data, demonstrating greater data efficiency.

Inference cost: The FALCON benchmark results show that, although NeuroLens ranks 1st among non-ensemble models across all evaluation settings, its normalized latency is slightly higher than that of other baselines. This may be because NeuroLens requires (t-1) iterations of next-time-bin prediction for a window with (t) time bins. Nevertheless, its normalized latency remains well below 1, suggesting that it meets the benchmark's real-time decoding requirements.

[2] Benchmark website: https://snel-repo.github.io/falcon/

M2 track: https://eval.ai/web/challenges/challenge-page/2319/leaderboard/5751


Q3. Contribution of statistical neuron-identity embeddings

We added ablations comparing statistical and learned neuron-ID embeddings and assessing the contributions of individual statistical components, focusing on generalization to unseen neurons (updated Appendix I and Tables 5–6).

Specifically, we pretrained NeuroLens on 5 earlier recording days and evaluated them on 5 later held-out days. Using the same setup, we removed firing-rate statistics (FR), autocorrelograms (ACG), or population coupling (PC) one at a time from the statistical signatures. The table below summarizes 2 main findings:

  1. Statistical and learned neuron ID embeddings perform similarly for seen neurons, but statistical neuron ID embeddings generalize better to unseen neurons.
  2. All 3 statistical components contribute to movement decoding, while their effects on choice decoding vary. Overall, removing population coupling causes the largest mean performance drop in both tasks for unseen neurons, which shows its importance for adaptation.
Movement (seen neurons)Movement (unseen neurons)Choice (seen neurons)Choice (unseen neurons)
Identity ablation (R² / accuracy)
Statistical neuron ID (FR + ACG + PC)0.38 ± 0.080.38 ± 0.030.87 ± 0.060.66 ± 0.05
Learned neuron ID0.36 ± 0.080.33 ± 0.030.86 ± 0.020.62 ± 0.03
No neuron ID0.27 ± 0.060.29 ± 0.030.64 ± 0.020.55 ± 0.03
Component ablation (% changes)
FR excluded-1.51% ± 0.38%-1.18% ± 0.63%-5.22% ± 4.90%+2.74% ± 1.13%
ACG excluded-1.17% ± 0.38%-1.51% ± 0.41%-7.23% ± 8.15%+0.93% ± 4.11%
PC excluded-1.10% ± 0.48%-2.86% ± 0.51%-3.14% ± 3.65%-4.67% ± 4.38%
AuthorsOpenReview ↗

Minor Question


1. NDT2 baseline.

We thank the reviewer for this question. To compare NeuroLens with NDT2 on this task, we evaluated NeuroLens on the FALCON benchmark [2] M2 track (updated Section 5.2 and Table 1). At the time of submission, NeuroLens ranks first among non-ensemble methods across all 3 cross-day adaptation settings: (1) zero-shot (ZS), which uses no held-out calibration data; (2) few-shot unsupervised (FSU), which uses only calibration neural activity; and (3) few-shot supervised (FSS), which uses both calibration neural activity and behavioral labels. We summarize the results for NeuroLens and NDT2 below; the full comparison with other baselines is in Table 1 in the updated paper.

In FSS, NeuroLens outperforms NDT2 (R2=0.53 vs. 0.43) using the same amount of pretraining and calibration data. Even without behavioral labels, NeuroLens achieves a slightly higher decoding score in FSU (R2=0.44) than NDT2 does in FSS (R2=0.43). These results highlight NeuroLens’s ability to learn consistent neural latents for stable decoding.

MethodSettingHeld-out R2R^2
NDT2 MultiFSS0.43±0.080.43 \pm 0.08
NeuroLensFSU (gradient-free)0.44±0.100.44 \pm 0.10
NeuroLensFSS0.53±0.080.53 \pm 0.08

2. Inconsistent ablation results.

Thanks for the question. The two results come from different evaluation settings: the 0.61 accuracy was obtained using a logistic regression probe trained on frozen pretrained latents, while the 0.88 accuracy was obtained using a fine-tuned choice decoding head.


3. Code availability and reproducibility.

Thank you for this question. We did not provide code for the IBL chronic Neuropixels dataset because the data are not yet publicly available. We now provide anonymized code to reproduce our results on the FALCON benchmark’s M2 track, which uses publicly available data: https://anonymous.4open.science/r/LATTE-4FA2/README.md


Reviewer_1JMkOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method for modeling and predicting neural dynamics. The proposed method can handle neural features recorded from varying numbers of neurons and employs a GPT-style pretraining approach on representations aligned to a consistent shape. Experimental results demonstrate that the proposed method achieves good generalization performance.

Critical strengths

The paper is well written, and the figures are clear and intuitive, making the proposed method easy to understand. In addition, the appendix provides extensive technical details.

Critical weaknesses
  1. The most significant concern with this paper is its novelty and the insufficient discussion of related work. EEG2Rep (KDD 2024), Brain-JEPA (NeurIPS 2024), and S-JEPA (Graz BCI Conference 2024) have already explored self-supervised learning of neural representations through latent-space prediction. Furthermore, LUNA (NeurIPS 2025) also employs learnable queries and cross-attention to handle inputs with varying numbers of channels, which is one of the core contributions claimed by this paper. Despite these studies having been published in established conferences, the authors neither cite them nor discuss how the proposed method differs from these existing approaches.

  2. The paper does not provide performance comparisons on standard benchmarks or datasets under established evaluation protocols. Consequently, it is difficult for readers to assess whether the proposed method achieves state-of-the-art (SOTA) performance.

Questions for authors
  1. The paper does not report the number of training tokens. Based on the reported parameter count, the proposed model is relatively small. At this scale, the Transformer architecture may not be sufficiently trained to realize its full potential. Have the authors experimented with replacing the Transformer with an LSTM?

  2. Could the authors make the source code publicly available? Providing code that enables reproduction of the reported results would offer more valuable information for reproducibility than the extensive implementation details currently included in the appendix.

4 comments

Thank you for taking the time to review our paper. We appreciate your positive comments on its clarity and intuitive presentation. We address your concerns below, with revisions highlighted in blue in the updated manuscript.

W1. Novelty

We thank the reviewer for discussing these relevant studies.

  1. Although there are prior JEPA based methods for non-invasive data, our work addresses a fundamentally different scientific question. We propose a new method to study the learning process of an animal by extracting latents from chronic intracortical neural data, where neural activity changes over months and sometimes even years, a unique problem that does not exist in EEG and other non-invasive datasets, due to the distinct nature of the 2 different domains. We have added discussion of prior work in non-invasive data in the Related Work section.

  2. Our model design is motivated by the distinct properties of different neural recordings. We propose 2 adaptation mechanisms to address different sources of non-stationarity. Statistical neuron IDs accommodate changes in the recorded neurons through matching their functional similarity, whereas FiLM infers and compensates for day-specific drift; see our reply to Reviewer apL3. This design allows our model to support zero-shot, few-shot unsupervised, and few-shot supervised adaptation within a unified framework, achieving leading performance among non-ensemble methods on the FALCON benchmark (see W2 reply).

Also, the learnable queries in the LUNA method mentioned by the reviewer do not solve our unique problem and we actually use it as an ablation baseline to show the advantage of our proposed adaptation method. See our reply to Reviewer u7KE for this ablation.

  1. Beyond SOTA decoding, the learned latents allow us to track learning-related neural changes; no other prior methods have shown this capability due to the distinct nature of our chronic mouse Neuropixels recordings. In updated Section 5.3 and Figure 4, we show that NeuroLens captures gradual transitions between learning stages, and progressive cortical reorganization during such process. These results show that NeuroLens can preserve meaningful representational changes during learning, rather than simply maintaining stable latents across days.

W2. Benchmark Results

We participated in the FALCON benchmark’s M2 track [1], ranking 1st among non-ensemble methods across all few-shot adaptation settings, and more than doubling the strongest zero-shot baseline’s decoding performance.

FALCON is a public benchmark of stable cross-day decoding from intracortical recordings. Its M2 track requires predicting 2-D finger velocity from Utah array spiking activity recorded in a non-human primate’s motor cortex.

Because the IBL chronic Neuropixels dataset is not yet publicly available, we instead provide an anonymized codebase to reproduce our results on the public FALCON benchmark: https://anonymous.4open.science/r/LATTE-4FA2/README.md

FALCON uses 3 few-shot adaptation settings: (1) zero-shot (ZS), which uses no held-out calibration data; (2) few-shot unsupervised (FSU), which uses only calibration neural activity; and (3) few-shot supervised (FSS), which uses both calibration neural activity and behavioral labels.

The results show:

  • NeuroLens ranks 1st among non-ensemble methods across ZS, FSU, and FSS settings.
  • NeuroLens achieves remarkable zero-shot performance (R2=0.16), exceeding the strongest ZS baseline (R2=0.06).
  • With unlabeled calibration data, NeuroLens achieves an R2 of 0.44, substantially outperforming the strongest FSU baseline (R2=0.27).
  • With labeled calibration data, NeuroLens achieves an R2 of 0.53, approaching the oracle reference (R2=0.58) while using only 10% of held-out data for calibration, compared with the oracle’s 60% private calibration data, showing greater data efficiency.

These results show strong zero-shot generalization and effective adaptation with limited calibration data. The full comparison is in updated Section 5.2 and Table 1.

MethodZero-shot (ZS)Few-shot unsupervised (FSU)Few-shot supervised (FSS)
Non-ensemble methods (10% calibration)
RNN-0.07 ± 0.23----
Wiener filter0.06 ± 0.04----
NoMAD + WF--0.20 ± 0.10--
CycleGAN + WF--0.22 ± 0.06--
SPINT (GF)--0.26 ± 0.13--
PRISM--0.27 ± 0.13--
OMEN----0.41 ± 0.08
S5 SSM----0.41 ± 0.11
NDT2 Multi----0.43 ± 0.08
NDT3----0.48 ± 0.06
TaskFuse----0.50 ± 0.11
PAR----0.53 ± 0.10
NeuroLens (ours)0.16 ± 0.160.44 ± 0.100.53 ± 0.08
Ensemble methods (10% calibration)
TCN + GRU ensemble----0.60 ± 0.07
Private-label oracle references (60% calibration)
NDT2 Multi----0.58 ± 0.04

[1] Benchmark website: https://snel-repo.github.io/falcon/

M2 track: https://eval.ai/web/challenges/challenge-page/2319/leaderboard/5751


Questions:

  1. The paper does not report the number of training tokens. Based on the reported parameter count, the proposed model is relatively small. At this scale, the Transformer architecture may not be sufficiently trained to realize its full potential. Have the authors experimented with replacing the Transformer with an LSTM?

The model size and training data scale used in NeuroLens are comparable to those in prior studies using invasive neural datasets [2, 3] and are therefore representative of current practice in this field. We also compared NeuroLens against state-space model baselines (POSSM [4]).

  1. Could the authors make the source code publicly available? Providing code that enables reproduction of the reported results would offer more valuable information for reproducibility than the extensive implementation details currently included in the appendix.

Thank you for this question. We did not provide code for the IBL chronic Neuropixels dataset because the data are not yet publicly available. We now provide anonymized code to reproduce our results on the FALCON benchmark’s M2 track, which uses publicly available data: https://anonymous.4open.science/r/LATTE-4FA2/README.md

[2] Azabou, Mehdi, et al. "A unified, scalable framework for neural population decoding." Advances in Neural Information Processing Systems 36 (2023): 44937-44956.

[3] Feghhi, Ebrahim, et al. "Time-masked transformers with lightweight test-time adaptation for neural speech decoding." Advances in Neural Information Processing Systems 38 (2026): 93047-93072.

[4] Ryoo, Avery Hee-Woon, et al. "Generalizable, real-time neural decoding with hybrid state-space models." Advances in Neural Information Processing Systems 38 (2026): 51764-51791.

I thank the authors for their revisions, particularly for adding comparisons with existing methods and providing part of the source code. I have raised my score. However, the response regarding novelty has not fully addressed my concerns. I agree that innovation can lie in applications and scientific discoveries, rather than necessarily requiring the development of an entirely new algorithm.

Nevertheless, from an application perspective, previous studies have already investigated closely related scientific questions. For example, Wang et al., in their study entitled “Dynamics of mesoscale brain network during visual discrimination learning revealed by chronic, large-scale single-unit recording,” used chronic, large-scale single-unit recordings across multiple brain regions to investigate neural network reorganization during visual discrimination learning in mice. Notably, this study is also cited in the manuscript under review.

Thank you for raising your score and giving us the opportunity to clarify our contribution! Our dataset uniquely combines chronic recordings spanning over 3 months with single-neuron resolution, which allows us to study how neural representations evolve throughout learning.

Taking the Wang et al. paper you mentioned as an example, it’s true that we both studied a related scientific question, but there are 3 differences:

  1. Recording duration: Wang et al.’s recordings covered 2~3 weeks, while ours span over 3 months.
  2. Learning coverage: Our dataset captures the complete learning process, including the debiasing, novice, and expert stages (as shown in the updated paper), and further extends to a more challenging condition in which animals must relearn and adapt to changing task demands. In contrast, Wang et al. included only early and expert learning stages.
  3. Method focus: While Wang et al. analyze raw neural activity and inter-regional connectivity, our work tracks population-level latent representations during learning. This is particularly important over long timescales, as individual recorded neurons continuously drift in and out of recordings, making raw neural activity difficult to track consistently. Population-level latents provide a more stable substrate and target for long-term tracking, and we have shown that NeuroLens latents extract more task-related information from neural activity during this process.

Thank you again for your time!

Comments on the paper

2 comments

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at our response? We would greatly appreciate any further thoughts or questions.

Many thanks for your continued help!

AuthorsOpenReview ↗

Revision Summary

Revision Summary:

We sincerely thank all reviewers for their constructive feedback, positive assessments, and willingness to raise their scores. We appreciate your engagement throughout the review process. We also thank the Area Chair for their time and consideration.

NeuroLens makes 3 key contributions:

  1. A self-supervised model that learns stable latent representations from neural activity despite drift across days and maintains SOTA decoding performance over long timescales.
  2. Two adaptation mechanisms tailored to distinct sources of nonstationarity in chronic recordings.
  3. Scientific insights into how neural activity reorganizes over the full course of learning, allowed by tracking neural latents that preserve learning-related changes from chronic recordings.

In response to the reviewers’ feedback, we have substantially improved the paper with additional experiments and analyses:

  • Reviewers 1JMk and apL3: In response to concerns about benchmarking and practical applicability, we evaluated NeuroLens on the public FALCON decoding benchmark, where it ranked #1 among non-ensemble methods across all 3 adaptation settings: zero-shot, few-shot unsupervised, and few-shot supervised. We also publicly released our anonymized codebase for reproducibility.
  • Reviewer u7KE: We appreciate the reviewer’s recognition of our method design and scientific contributions. We have addressed the concerns through (1) new experiments on pretraining requirements, calibration efficiency, and inference cost, and (2) new analyses showing the preservation of learning-related changes in NeuroLens latents.

Thank you again for taking the time to read our paper and consider the changes we made during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.