acceptodds
Under review as a conference paper at ICLR 2027

A Controlled Comparison of Self-Supervised Pretraining Objectives and Corpora for Intracortical Neural Decoding

Abstract

Brain-computer interfaces (BCIs) have recently made impressive progress in restoring communication by decoding intended behavior directly from intracortical neural activity. Inspired by the success of foundation models in computer vision and natural language processing, recent work has begun to explore large-scale self-supervised pretraining for neural decoding. However, it is unclear whether these approaches already provide consistent advantages at the scale of data currently available, which is still limited and highly fragmented. In this work we present the first controlled comparison of three self-supervised objective families for intracortical sequence decoding under a unified experimental framework. Using a common transformer backbone, we compare masked autoencoding (MAE), Joint-Embedding Predictive Architectures (JEPA), autoregressive (AR) pretraining, and training from scratch across corpora spanning multiple subjects, behaviors and species: up to 345 hours of intracortical recordings, combining 208 hours from 6 human participants performing attempted and imagined speech, handwriting, typing and cursor control with 137 hours of motor tasks from 19 non-human primates. Our results show that the gains pretraining provides on downstream tasks strongly depend on the amount and diversity of available data, and the resulting performance is often comparable to that of carefully tuned task-specific training. These findings establish a controlled benchmark for future research on neural foundation models.

Primary area:
applications to neuroscience & cognitive science
Keywords:
self-supervised learning, intracortical recordings, neural decoding, brain-computer interfaces

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wBhpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a controlled comparison of MAE, JEPA, and autoregressive self-supervised pretraining against training from scratch for intracortical neural decoding. Using a shared BIT Transformer backbone, the study compares pretraining on human-only and combined human–nonhuman-primate corpora, comprising approximately 208 and 345 hours of data, respectively, and evaluates both subject-specific and joint fine-tuning. Downstream tasks include attempted and imagined speech, handwriting, and typing, evaluated using phoneme error rate (PER) or character error rate (CER) with greedy CTC decoding. The study uses a fixed hyperparameter search budget for fine-tuning and conducts five-seed stability analyses of selected configurations. The results show that the benefits of pretraining vary across objectives and datasets, adding nonhuman-primate data does not consistently improve performance, and carefully tuned models trained from scratch remain competitive under the evaluated settings.

Critical strengths
  1. A fair comparison with tuned scratch baselines. The study uses a shared BIT Transformer backbone and allocates the same fine-tuning hyperparameter search budget to pretrained and randomly initialized models. This makes it easier to assess whether pretraining provides gains beyond careful task-specific training, rather than gains that may simply reflect differences in tuning effort.

  2. Evaluation across tasks and pretraining data compositions. The study covers speech, handwriting, and typing, and compares different pretraining objectives and corpus compositions. The results show that pretraining does not consistently improve decoding performance, and adding nonhuman-primate recordings does not reliably benefit human decoding. These findings caution against assuming that pretraining, particularly with data from additional species, will necessarily improve neural decoding.

Critical weaknesses
  1. The stability analysis does not adequately establish the robustness of the main conclusion. The analysis uses only five fine-tuning seeds per selected configuration and removes high-error clusters based on the observed outcomes (Section 3; Figure 3). Although unsuccessful runs are retained in the main hyperparameter sweep, excluding high-error seeds from the stability analysis may bias the estimated differences between pretrained and scratch models. With only five seeds, it is also difficult to characterize training variability reliably. In addition, the absence of statistically significant improvements does not establish that pretraining offers no meaningful benefit. Reporting results with all seeds retained, along with failure rates and uncertainty estimates for effect sizes, would clarify how sensitive the conclusions are to the treatment of failed runs.

  2. The data separation protocol is insufficiently specified to verify the evaluation setting. Appendix A.3 describes the downstream data splits but does not clearly state whether recordings used for downstream validation and testing are excluded from self-supervised pretraining. Appendix A.2 also notes that normalization statistics are computed across all available sessions of the same subject where possible, leaving it unclear whether these statistics cross downstream split boundaries. Furthermore, some datasets come from the same biological participants, complicating the interpretation of evidence across datasets as independent. These omissions do not demonstrate data leakage, but they prevent a clear assessment of how strictly the pretraining and evaluation data are separated, weakening confidence in the reported benefits of pretraining.

Questions for authors
  1. Data separation — clarification: Were downstream validation and test recordings excluded from SSL pretraining and from the computation of normalization statistics? How was overlap between datasets from the same participant handled, and what unit—trial, session, or recording day—was used to define each split?

  2. Seed exclusion and training reliability — clarification and robustness analysis: How many fine-tuning seeds were excluded from each configuration, and according to what criteria? Do the conclusions hold when all five seeds from the stability analysis are retained? Please report the results with all seeds included, failure rates, and uncertainty estimates for the pretraining-versus-scratch differences. How many independent pretraining runs were performed for each objective–corpus combination? If multiple pretrained checkpoints are available, how much do downstream results vary across them? These details would help distinguish performance conditional on successful training from the reliability of the overall training procedure.

  3. Statistical evaluation — clarification: Were the statistical comparisons in Figure 3 based on validation or held-out test errors? Appendix F reports validation errors. Please specify the baseline used to calculate the relative error reductions in Figure 3 and the baseline used in the statistical tests, particularly for pretrained models under joint fine-tuning. Do the conclusions hold when pretrained models are compared with their corresponding scratch baselines on held-out test data?

  4. Independent participants and joint fine-tuning — clarification: Table 1 suggests that the downstream evaluation instances do not represent ten distinct biological participants, and both handwriting datasets include data from T5. Which results support the benefits of pooling data across different participants, rather than pooling datasets from the same participant? Can the current experiments distinguish these effects, and would this distinction change the interpretation of the joint fine-tuning results?

  5. Effect of adding NHP data — clarification and supporting analysis: How were human and NHP recordings sampled during pretraining, and did adding NHP data change the number of optimization steps or the exposure to human recordings? Table 5 reports that MAE reconstruction R2R^2 on the diagnostic pool decreases from 0.093 with human-only pretraining to 0.021 with human-plus-NHP pretraining. Do the available per-dataset diagnostics help distinguish negative transfer from optimization difficulties or sampling imbalance? Does the lower reconstruction R2R^2 correspond to consistent changes in downstream decoding performance across tasks, or do the diagnostic and downstream results suggest different effects of adding NHP data?

7 comments

W1) We thank the reviewer for this question. In the revised manuscript we have increased the stability analysis from 5 to 10 seeds per configuration (14 model–regime combinations × 10 subjects = 140 configurations, 1400 fine-tuning runs) and extended it to joint fine-tuning: each pre-trained checkpoint, fine-tuned jointly on all subjects of a task, is compared against joint fine-tuning from scratch (new Figure 3 in the main text and Figure 8 in Appendix G).

137 of 1400 runs (9.8%) were excluded. Failures are concentrated on typing and handwriting: 129 of 560 runs (23%), in 52 of 56 configurations. On speech only 8 of 840 runs failed (1%), in 6 of 84 configurations, 4 of which are Jude anarthria. The per-configuration counts and excluded values are reported in Appendix Table 10 of the revised manuscript and in the table below.

Failure rates per model and fine-tuning regime, pooled over the 10 subjects:

ModelSingle-subjectJoint
Scratch10%9%
MAE-H9%11%
MAE-HM6%7%
JEPA-H11%10%
JEPA-HM14%10%
AR-H9%15%
AR-HM8%8%

Convergence failures are therefore a property of the tasks (typing and handwriting, with small datasets), and pre-training neither causes nor prevents them.

For completeness, we report here the full analysis without excluding any seed. As requested, for each subject and model we report four quantities: the relative change of the median error with respect to scratch, with a percentile-bootstrap 95% confidence interval (CI; 10,000 resamples of seeds within each group); the probability of superiority, i.e. the probability that a seed of the pre-trained model reaches a lower error than a seed of the scratch model, with its bootstrap CI; the Mann–Whitney U test with Benjamini–Hochberg (BH) correction.

Retaining all seeds almost never changes the direction of an effect. The sign of the median change flips in only 3 of 60 single-subject comparisons, 1 of 60 joint comparisons and none of the 10 scratch single-vs-joint comparisons. Only two of these flips are substantial, both on Jude T17 with single-subject fine-tuning. There, at least half of the seeds failed, so the median falls among the failed runs. The other two flips are small effects (below 3%) close to zero. Significance decreases, because the retained failures widen the distributions. With failed seeds excluded, pre-training is significantly better than scratch in 8 and significantly worse in 38 of the 60 single-subject comparisons; with all seeds retained, these numbers become 4 and 34. In joint fine-tuning they go from 10 better and 23 worse to 7 better and 15 worse. For the comparison between scratch joint and scratch single training, they go from 5 better and 2 worse to 1 better and none worse. Most of the comparisons that lose significance (14 of 19) are on typing and handwriting, where failures are concentrated.

The bootstrap confidence intervals, computed with all seeds retained, lead to the same conclusions. In single-subject fine-tuning, the CI excludes zero in favour of pre-training in only 4 of 60 comparisons: AR-H on Kunz T12 and T18, AR-HM on T18, and MAE-HM on Willett handwriting. It excludes zero against pre-training in 30 comparisons, and includes zero in the remaining 26. JEPA accounts for 14 of the 30 unfavourable comparisons, being worse than scratch on every speech subject and on T18. The remaining 16 are all on speech subjects: 6 for MAE-H, 4 for AR-H, 4 for AR-HM and 2 for MAE-HM.

In joint fine-tuning, the CI excludes zero in favour of pre-training in 10 of 60 comparisons, all on speech subjects: AR-H on Wairagkar, Kunz T15, Willett T12 and Kunz T12; AR-HM on Wairagkar, Kunz T15 and Kunz T12; MAE-HM on Willett T12 and Kunz T12; and MAE-H on Kunz T12. It excludes zero against pre-training in 15 comparisons, all JEPA, and includes zero in the remaining 35. MAE-HM lowers the median error of the scratch joint model on all 10 subjects, both with and without seed exclusion. When scratch joint training is compared with scratch single-subject training, the CI excludes zero in favour of joint training on Card and Jude anarthria and against it on Kunz T15 and Kunz T12, and includes zero on the other six subjects.

In conclusion, retaining the failed seeds does not change our conclusion that pre-training offers no consistent benefit over training from scratch in single-subject fine-tuning, where differences are mostly within seed-to-seed variability or favour scratch, and that its advantages, when present, emerge in joint fine-tuning on speech, most consistently for MAE pre-trained on humans and monkeys.

Further details regarding the stability analysis are provided in the answer to Question 1 below.

W2) We thank the reviewer for raising this point; we have clarified the protocol in Appendix A.2–A.3. Self-supervised pre-training uses only the training split of each downstream dataset. The downstream validation and test recordings are never seen during pre-training. In addition to the training splits, pre-training uses the unlabelled official competition hold-out recordings of Card et al. and Willett et al. These are separate files from those from which our validation and test sets are drawn. The pre-training checkpoint is selected on the validation split of two subjects (Willett T12 and Card), and the fine-tuning configuration on the validation split of each subject. The test set is used only once, for the final evaluation. Normalisation uses only label-free per-channel statistics. Features are z-scored within each recording session. Regarding the datasets coming from the same biological participant: this happens for the following pairs: Card and Kunz T15, Willett and Kunz T12. These two pairs of subjects are always trained jointly (see also section 2.5 of the manuscript, paragraph “Single vs Joint Fine-Tuning”). While we agree that pooling together datasets coming from different subjects performing the same task is different compared to pooling together datasets coming from the same subject performing the same task, we believe that the following experiment disentangles the confusion. The joint-task experiment pools subjects performing the same task (speech, handwriting, typing), and its results can be compared to the pseudo-joint results of the pairs Card + Kunz T15 and Willett + Kunz T12. Please see in particular the answer to Q4 below.

AuthorsOpenReview ↗

Answer to Reviewer's Questions

Q1) Please see our answer to Weakness 1 above.

Q2) We thank the reviewer for this question. Please see also our answer to Weakness 1. A seed is considered failed if all of the following hold: the distribution of the 10 seeds of that configuration is bimodal (Silverman test, α = 0.10); the seed belongs to the upper cluster of a 2-means split; the ratio between the upper and lower clusters’ mean exceeds 1.25. Regarding the last question of pretraining variability: for each objective, model size and corpus we ran a hyperparameter sweep with a single seed per configuration. This amounts to 60 configurations for MAE and JEPA and 12 for AR, i.e. 180 MAE, 180 JEPA and 36 AR runs per corpus over three model sizes, about 19,000 GPU-hours in total only for pretraining. One checkpoint per objective-corpus combination was selected on the pre-training validation criterion and fine-tuned. We did not repeat pre-training with different seeds, and we did not fine-tune multiple checkpoints of the same combination, unfortunately, because of the extremely high computational cost.

Q3) Thanks for allowing us to clarify. The results in Figure 3 are reported on the validation set in all cases. The baseline of Figure 3 and subsequent analysis is the stability run of the model trained from scratch. Since then, we have repeated the analysis with n=10 stability seeds and added the comparison of the joint fine-tuning models with the joint from-scratch baseline (please see answer to Weakness 1 above). We repeated the full stability analysis evaluating on the held-out test set. The conclusions are unchanged. Exactly the same 137 seeds are identified as failed on validation and test, so failure rates are identical. Across the 130 pretrained-versus-scratch and joint-versus-single comparisons, the effect has the same sign in 121. No comparison is significantly better than scratch on one split and significantly worse on the other. Pre-training is significantly better than scratch in 6 of 60 single-subject comparisons on test (8 on validation) and in 9 of 60 joint comparisons (10 on validation). JEPA remains worse than scratch throughout. MAE-HM again lowers the median error of the scratch joint model on all 10 subjects. The only comparisons that lose significance on test are small effects or subjects with very small test sets (e.g. 25 sentences for Kunz T15), where test-set noise reduces statistical power.

Q4) We thank the reviewer for the question. Two experiments disentangle these effects. First, we compare the pseudo-joint fine-tuning (Card+KunzT15, Willett+KunzT12) to the joint-speech models. Adding data of different participants performing the same task is beneficial. Second, on typing: data comes from two separate participants (T17 and T18). Also in this case, training jointly on the available typing data from both participants is beneficial.

Q5) We thank the reviewer for raising this point. The training data was shuffled across human and non-human primates. The addition of NHP data does not change the exposure to human data, as we fix the total number of pre-training epochs to 400 regardless of the training corpus.

Lower reconstruction R² after adding NHP data does not necessarily indicate a worse model. With more and more heterogeneous data, a model of fixed capacity spreads its reconstruction ability over many more sources, so its fit to each individual dataset becomes more diluted. We report the MAE reconstruction R² on human and NHP datasets (final checkpoints; previous Table 5 values come from earlier runs):

DatasetMAE R² (H / HM)
Human
Speech (Wairagkar)0.36 / 0.31
Handwr. (Fan)0.02 / -0.42
Handwr. (Willett)0.02 / -0.75
Typing (T18)0.07 / -0.18
NHP (HM only)
O'Doherty– / -0.09
Churchland– / 0.03
Even-Chen– / 0.06
Ma– / -0.94
Temmar– / 0.01

The HM model's reconstruction R² is low on the NHP datasets as well, consistent with its capacity being spread across many sources rather than traded from human to NHP data. Yet after fine-tuning it decodes better than the H model on every human dataset except Kunz T15 under task-joint fine-tuning.

Reviewer_AxzNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper aims to compare different self-supervised objectives such as MAE, JEPA, and AR neural-only pretraining and their downstream performance relative to from-scratch supervised baselines for various human text-based tasks, while also comparing two pretraining corpora: human only and human+monkey. They adopt a controlled setup where the model architecture and various hyperparameters and regimes are fixed so as to isolate the differences arising from just the objective and the corpus. They find that these SSL pretrained models rarely outperform scratch baselines by a significant amount, arriving at performances just slightly better or comparable on average.

Critical strengths
  • The paper considers evaluation on several downstream human datasets which are important for BCI applications, and explores a few popular SSL objectives.
  • Controlling several axes of variability is important to some degree to isolate the effect of just the objective as done here.
  • The topic is very useful to understand what is useful for neural decoding and help us improve our architectures and objectives.
Critical weaknesses
  • The key issue for me is that there is no few-shot analysis in the paper. The benefits of pretraining are often best seen in few-shot transfer settings, in terms of speed to a certain level of performance and/or absolute final performance. So as such, the paper to me only evaluates on particular regime where with enough downstream data, a well-tuned scratch baseline might be able to match and sometimes exceed pretrained models (see results from NDT-3 by Ye et al., 2025 on this topic).
  • Relatedly, it would be important to see within-subject versus cross-subject evaluations as well to further understand the intricacies of the pretraining+finetuning results as opposed to just one evaluation scheme. In general, I appreciate controls on pretraining but appreciate diversity in the finetuning regimes, datasets, and analyses given the nature of claims here.
  • While the controlled comparisons are appreciated, the set of comparisons and experiments is lacking some breadth. Few-shot analysis is missing as suggested above, but the title says this is a comparative study of objective and corpora -- there are only two corpora variants (Human only and Human+Monkey), so this is quite limited compared to what the title may suggest. Furthermore, the only downstream tasks considered are human tasks with specific applications that are all similar: all of them use the CTC objective. Thus, none of the downstream tasks are representative of continuous motor control (e.g., cursor or cursor and click), or other decoding applications e.g., related to visual cortex (although more of these data are from mouse). To make a general claim about the utility of pretraining objectives, evaluation needs to be a lot more broad than explored here in terms of both the kinds of experiments and datasets/downstream evals. The authors have, for example, cited the FALCON benchmark (Karpowicz et al., 2024) but not considered it for evaluation.
  • More recent work in pretraining for neuroscience has explored joint supervised and self-supervised objectives and seen better benefits than broad self-supervised pretraining. For intracortical datasets where labels are often available, this could be a good strategy especially if the downstream tasks find representation in the pretraining corpus, because training readouts is an important part of identifying the brain-behaviour mapping and relevant latents for the decoding problem -- this may help better few-shot adaptation downstream. Work like Zhang et al., 2025 (NEDS -- joint encoding, decoding, and self-prediction), Ye et al., 2025 (NDT-3 -- autoregressive neural and behaviour), and more recently Mao et al., 2026 (MOJO -- joint masked auto-spike-count prediction and decoding) would be worth discussing in this particular context.
  • There is no equivalent comparison to a supervised pretrained baseline, which would be important in my opinion to see if the failures here are unique to SSL. Models such as POYO (Azabou et al. 2023 & 2025) and POSSM (Ryoo et al. 2025) have done well on downstream tasks with supervised pretraining. Especially since the data downstream are represented among the pretraining tasks, why not include a supervised pretrained baseline as either a ceiling or to see how much this is just SSL being poor here?
  • The hyperparameter sweep budget for pretraining itself is a little unclear -- doesn't seem like that was varied and I'm concerned how the results would change based on that aspect. I understand that it is expensive but the problem statement and claims would warrant a very careful and thorough study.
  • Minor as a weakness but an important note: the work is missing citations to important recent work in pretraining for neural decoding. For example, Ye et al. (2023) which introduced NDT-2, a self-supervised pretraining method for intracortical decoding, is absent from the references and discussion. POSSM by Ryoo et al. (2025) which most recently explored and showed benefits of monkey motor pretraining for human handwriting decoding is also missing (these papers also showed analyses like few-shot, several downstream tasks especially in POSSM like monkey and human motor as well as human speech) -- it would be important to add references to these and other recent work to properly contextualise the methods in the field, e.g., Mao et al. (2026) mentioned earlier.
  • Minor nit: BIT "BraIn to Text" is not the name that Feghhi et al., 2025 uses for their architecture, they refer to it as the time-masked transformer where the time-masking is an augmentation and not an SSL objective, to my knowledge. It is the Zhang et al., 2026 (BIT) paper that introduces the BIT acronym, so any differences between the two must be carefully highlighted and the reference to the papers and acronyms correctly mentioned.

References including suggested relevant work:

  • Ye et al. (2025). A Generalist Intracortical Motor Decoder. NeurIPS 2025. [NDT-3]
  • Karpowicz et al. (2024). Few-shot Algorithms for Consistent Neural Decoding (FALCON) Benchmark. NeurIPS 2024 D&B. [FALCON]
  • Zhang et al. (2025). Neural Encoding and Decoding at Scale. ICML 2025. [NEDS]
  • Mao et al. (2026). Leveraging unlabelled data for generalizable neural population decoding. arXiv 2026. [MOJO]
  • Azabou et al. (2023). A unified, scalable framework for neural population decoding. NeurIPS 2023. [POYO]
  • Azabou et al. (2025). Multi-session, multi-task neural decoding from distinct cell-types and brain regions. ICLR 2025. [POYO+]
  • Ryoo et al. (2025). Generalizable, real-time neural decoding with hybrid state-space models. NeurIPS 2025. [POSSM]
  • Ye et al. (2023). Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity. NeurIPS 2023. [NDT-2]
  • Feghhi et al. (2025). Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding. NeurIPS 2025.
  • Zhang et al. (2026). A cross-species neural foundation model for end-to-end speech decoding. ICLR 2026. [BIT]
Questions for authors
  • If there are no improvements for pretraining over the scratch baseline in Fig. 3, how were the best models in Fig. 2 selected and how are pretrained models there better, even if marginally, over the scratch baseline?
  • Have the authors considered a fully supervised baseline or a joint SSL + SL baseline?
  • Could the authors try out some experiments with few-shot analyses as well as quantify training speed gains downstream from pretraining (if any), regimes where backbones are frozen versus allowed to change, as well as other downstream tasks beyond just CTC-based ones? What about other finetuning regimes as mentioned in points 1 and 2 in the weaknesses?
  • Could the authors clarify the pretraining hyperparameter choices and whether they are justified?
11 comments

W1) We thank the reviewer for raising this point, which we are happy to address. To provide an answer, we ran labelled-data curves as follows. We fine-tuned the architecture on 11 fractions of the training set (1, 2, 3, 5, 7.5, 10, 15, 20, 30, 50, 100%) of one representative dataset per task: Wairagkar for attempted speech, Fan for handwriting, and participant T18 for typing. Each configuration was trained from scratch and from each of the 6 pretrained checkpoints, with 3 seeds, for 693 runs in total.

The training subsets are stratified by recording day and nested, so each fraction contains the smaller ones. For a given fraction and seed, all models are trained on exactly the same trials. All runs process the same number of training examples. The checkpoint is selected on the validation set, and we report test error. Each pretrained model is used at its best size and with its best fine-tuning configuration from our sweep.

Median test error (%) over 3 seeds: PER for speech, CER for handwriting and typing. Columns give the fraction of the training set and the corresponding hours of labelled recordings. Values of roughly 60% or more correspond to runs that stay on the CTC plateau.

Attempted speech (Wairagkar, PER)

1% (0.3 h)5% (1.4 h)10% (2.7 h)20% (5.2 h)50% (13 h)100% (26 h)
Scratch62.738.927.117.810.27.3
MAE-H53.834.626.218.610.97.6
MAE-HM57.636.927.018.410.07.5
JEPA-H76.339.329.419.511.78.7
JEPA-HM65.440.330.121.411.98.8
AR-H65.236.527.520.111.07.4
AR-HM64.037.528.020.011.07.6

Handwriting (Fan, CER)

1% (0.1 h)5% (0.5 h)10% (1.0 h)20% (2.0 h)50% (4.8 h)100% (9.7 h)
Scratch77.678.676.573.333.720.9
MAE-H73.173.769.648.629.522.1
MAE-HM74.672.572.459.626.220.4
JEPA-H79.680.078.773.937.026.0
JEPA-HM80.979.779.573.337.626.3
AR-H75.571.457.344.228.521.7
AR-HM76.574.574.748.728.621.7

Typing (T18, CER)

1% (0.1 h)5% (0.35 h)10% (0.7 h)20% (1.3 h)50% (3.1 h)100% (6.1 h)
Scratch75.633.022.717.512.410.2
MAE-H75.826.419.714.612.010.8
MAE-HM77.328.321.917.412.510.5
JEPA-H104.777.577.520.374.012.0
JEPA-HM90.971.824.819.372.012.1
AR-H50.425.822.315.711.810.1
AR-HM76.074.678.369.012.110.2

Attempted speech. In the low-data regime, pretrained models, especially MAE-H and MAE-HM, outperform the scratch baseline. With 0.3 h of labelled data, MAE-H reaches 53.8% PER against 62.7% from scratch: 8.9 points, or 14% relative. The gap is still 4.3 points at 1.4 h. The advantage decreases as the available data increases and fades at about 4 hours. We define the crossover as the point where the paired difference with scratch on the same trials reaches zero, interpolated between fractions. For MAE-H and scratch the crossover is at 4.0 h.. Beyond this point all models perform similarly: 7.3-8.8% PER with the full 26 h.

For handwriting and typing the picture is more complex, because of high training instability. Many runs stay on a CTC plateau (validation CER >= 60%) and do not converge to a useful solution.

On handwriting, no model escapes the plateau below about 1 h of data. Above that, pretrained models converge earlier and more reliably than scratch. At 2 h, AR-H and MAE-H reach 44.2% and 48.6% CER against 73.3% from scratch. With at least 10% of the data, scratch remains on the plateau in 12 of 18 runs, against 4/18 for MAE-H and 3/18 for AR-H. The advantage persists up to almost the full training set: the MAE-H crossover is at 8.9 h out of the 9.7 h available.

On typing, pretrained models are better between about 0.3 and 3 h. For example, at 0.35 h MAE-H and AR-H reach 26.4% and 25.8% CER against 33.0% from scratch. The MAE-H crossover is at 3.3 h. Here scratch is the most stable model: with at least 10% of the data it never stays on the plateau (0/18 runs), against 2-4/18 for MAE-H, MAE-HM and AR-H and 7-10/18 for AR-HM and the JEPA models. In the very-low-data regime (<= 0.2 h), the medians are therefore dominated by non-converged runs. When the best of the 3 seeds is selected on validation, MAE-H reaches 41.6% CER at 0.1 h, against 74.4% from scratch.

AuthorsOpenReview ↗

W1 - continued

W1 - continued) Attempted speech. In the low-data regime, pretrained models, especially MAE-H and MAE-HM, outperform the scratch baseline. With 0.3 h of labelled data, MAE-H reaches 53.8% PER against 62.7% from scratch: 8.9 points, or 14% relative. The gap is still 4.3 points at 1.4 h. The advantage decreases as the available data increases and fades at about 4 hours. We define the crossover as the point where the paired difference with scratch on the same trials reaches zero, interpolated between fractions. For MAE-H and scratch the crossover is at 4.0 h.. Beyond this point all models perform similarly: 7.3-8.8% PER with the full 26 h.

For handwriting and typing the picture is more complex, because of high training instability. Many runs stay on a CTC plateau (validation CER >= 60%) and do not converge to a useful solution.

On handwriting, no model escapes the plateau below about 1 h of data. Above that, pretrained models converge earlier and more reliably than scratch. At 2 h, AR-H and MAE-H reach 44.2% and 48.6% CER against 73.3% from scratch. With at least 10% of the data, scratch remains on the plateau in 12 of 18 runs, against 4/18 for MAE-H and 3/18 for AR-H. The advantage persists up to almost the full training set: the MAE-H crossover is at 8.9 h out of the 9.7 h available.

On typing, pretrained models are better between about 0.3 and 3 h. For example, at 0.35 h MAE-H and AR-H reach 26.4% and 25.8% CER against 33.0% from scratch. The MAE-H crossover is at 3.3 h. Here scratch is the most stable model: with at least 10% of the data it never stays on the plateau (0/18 runs), against 2-4/18 for MAE-H, MAE-HM and AR-H and 7-10/18 for AR-HM and the JEPA models. In the very-low-data regime (<= 0.2 h), the medians are therefore dominated by non-converged runs. When the best of the 3 seeds is selected on validation, MAE-H reaches 41.6% CER at 0.1 h, against 74.4% from scratch.

We do not observe a consistent advantage of the JEPA-pretrained models on any of the three tasks.

To verify that the advantage is not due to model size or to the per-model choice of fine-tuning hyper-parameters, we repeated the speech experiment with all encoders at the same size (384x7) and a single common fine-tuning configuration. The conclusions are unchanged. MAE-H remains the best model below about 4 h. The AR and JEPA encoders are within a few points of scratch. No pretrained model outperforms scratch beyond about 3-4 h of data.

This is a complementary result to what was found by Ye et al. (NDT-3) for a simpler motor task: the advantages of pretraining in their setting faded after 1.5 hours of labelled data became available.

W2) We thank the reviewer for this suggestion. A leave-one-subject-out evaluation, in which a participant is excluded from pre-training and the pre-trained model is then fine-tuned on that participant, would be the natural test of how well pre-training transfers to a new participant, and we agree it is an interesting direction. On the other hand, decoding a participant never seen during training (zero-shot) is not possible with our architecture by design: the input layer is subject-specific. However, the available intracortical datasets are few and small. We therefore chose to include all the available data in pre-training. Repeating the pre-training for each held-out participant was not feasible within the computational budget (the pre-training hyperparameter sweep alone required about 19,000 GPU-hours, see Weakness 6). We will discuss this limitation explicitly in the revised paper and keep leave-one-subject-out evaluation for future work. However, our evaluation already covers two fine-tuning regimes with increasing exposure to data from other recordings and participants: (i) subject-specific fine-tuning, where the model is fine-tuned and evaluated on the same participant and dataset; (ii) joint fine-tuning, where all datasets of the same task, from different participants, are combined, i.e. cross-subject fine-tuning. Our model has a subject-specific input layer (one per participant), while the rest of the network is shared, so in this regime knowledge is transferred across participants through the shared layers. Adding labelled data from other participants (joint vs subject-specific fine-tuning) improves results for most models, and more so for pre-trained models: over the 10 subjects it improves MAE-H on 9 (median -12% relative error), MAE-HM and AR-H on 8 (-9% and -11%) and AR-HM on 7 (-9%), against 7 subjects and -3% for the scratch baseline (see also our answer to Q2). This indicates that pre-trained representations make cross-subject labelled data more useful. The labelled-data curves (see our answer to W1) complement this analysis on the within-subject side, showing how much labelled data from the target participant is needed for pre-training to help. We add a further analysis in which we pretrain the network in a supervised manner over all subjects and tasks; please see A-W5.

W3) We thank the reviewer for these suggestions, and we agree that they are all interesting directions. Our work focuses on speech and communication BCIs: all the downstream tasks we consider (attempted speech, typing and handwriting) decode sequences of discrete symbols and are therefore all trained with the CTC objective. Within this scope, we compare the impact of three pre-training objectives and two corpora under controlled conditions across 10 datasets and three tasks. Continuous motor control, decoding from other areas such as visual cortex, and benchmarks such as FALCON (Karpowicz et al., 2024) fall outside this scope; they are a natural extension of this work, and we will mention them explicitly as future work. Regarding the few-shot analysis, we have added labelled-data curves (see our answer to Weakness 1). In the revised manuscript we will make the focus on communication explicit in the abstract and introduction, and further stress that our claims shall be read in the domain of BCIs, so that they are not read as general claims about the utility of pre-training objectives for all neural decoding applications. Should the work be accepted, we would be open to changing the title of the manuscript to “A Controlled Comparison of Self-Supervised Pretraining Objectives and Corpora for Intracortical Decoding of Speech, Handwriting and Typing”.

W4) We thank the reviewer for this suggestion. All objectives we compare in the main text are purely self-supervised, and we now state this scope explicitly in the Introduction and Methods. We have added a new analysis (please see AW-5 below) with a fully supervised pre-training, which is now in Appendix J. Furthermore, we have added NEDS, NDT3 and MOJO to the Introduction as examples of pretraining that combines self-supervised and supervised objectives, highlighting the differences between these approaches and ours: these works all use spike counts as input data, while we use spike band power and spike counts, when available; NEDS uses only mice Neuropixels recordings during decision-making tasks, NDT-3 uses spike counts from both human and non-human primates recorded during motor tasks, typically reaching and grasping, while the approach of MOJO spans monkey motor cortex during reaching tasks and multi-regional mouse recordings during vision and decision making tasks. In the Discussion we relate them to our finding that pooling labelled data across participants helps, even without pretraining, which is consistent with the reviewer’s argument that supervision on behaviour aids identification of the brain-behaviour mapping.

Reviewer_9RxLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work investigates impact of pre-training on neural recordings in terms of neural decoding performance. The authors compare approaches such as MAE, JEPA, and AR applied to BIT transformer model. Decoding performance is evaluated on a collections of tasks: speech, handwriting, typing. The work also tests whether adding primate recordings to the pre-training data is useful and impact of training across multiple subjects.

Main result appears to be that pre-training vs training from scratch does not result in consistent advantages. In gneral the experiments find that MAE performs better than the other pre-training approaches in several cases. Adding primate data generally doesn't help. Training jointly on labeled data from multiple subjects performing the same task appears more beneficial than pre-training.

Critical strengths

S1. The work assembles, aligns and standardize a large and diverse collection of intracortical recordings across subjects, tasks, and species. This is a significant effort and could be very useful for the community and further research.

S2. The paper includes a systematic and aligned comparison of leading pre-training approaches across datasets of neural recordings. The evaluation is extensive. Most of results are negative but potentially are useful for leading to methods to deal with limitations and challenges of pre-training on neural data.

Critical weaknesses

W1. The work relies entirely on BIT. There's limited exploration of architectural choices. It is possible that BIT arch is not particularly well suited for pre-training or transferring representations across subjects and tasks. Without testing other architectures or more extensive architectural ablations it is difficult to know whether the conclusions apply more broadly.

W2. Despite the effort to combine multiple datasets (which I really appreciate and think is a significant important contribution) the amount of pre-training data is still relatively small. Furthermore, the low-data regime is not really evaluated and this is where the pre-training can be impactful. In particular, it is unclear whether the limited gains reflect the pre-training methods themselves, insufficient pre-training data, or simply the availability of enough labeled data during finetuning.

W3. This is less critical. The comparison between pre-training methods may be affected by hyperparameters. The authors did perform sufficent tuning during finetuning but used fixed settings for pre-training. Without further experiments, it is difficult to determine whether the differences reflect the methods themselves or how they were trained.

Questions for authors

Q1. Why was BIT chosen as the only architecture? Is there evidence that it is well suited for pre-training and transferring representations across subjects and tasks?

Q2. How does performance scale with the amount of pre-training data?

9 comments

We thank the reviewer for his comments and suggestion. We reply below, in order, to all points.

W1) We thank the reviewer for raising this point, which we recognise as a limitation. The choice of BIT was guided by three main reasons. First, BIT is the current state-of-the-art architecture for intracortical speech decoding [Fogg et al., 2026, Feghhi et al., 2025]. It is also the backbone of the cross-species foundation model of Zhang et al., 2026, the study that reported benefits of pretraining, using a MAE strategy to pretrain the BIT. BIT is also, at its core, a standard transformer encoder, the architecture underlying foundation models in most other domains, so it is a natural choice if neural foundation models are to follow the same trajectory. Furthermore, holding the BIT backbone fixed is what allows us to attribute differences to the pretraining objective and corpus rather than to the architecture. A comparison that varied the architecture as well would need a much larger design to remain interpretable. Finally, we fully agree that whether our conclusions extend to other architectures is an important question. Answering it was beyond our computational budget. Each additional architecture requires repeating the full pretraining + finetuning + stability pipeline, which comes down to > 19,000 GPU hours per architecture. We will state this limitation explicitly in the limitations of the revised manuscript: “We use a single transformer backbone, which isolates the effect of the pretraining objective and corpus but limits the generality of our conclusions: other architectures may interact differently with these objectives, and whether our findings extend to them remains to be tested.” We have also better motivated the choice of the BIT in Section 2.3 of the manuscript.

Fogg et al., 2026: A generalizable speech neuroprosthesis, Feghhi et al., 2025: Time-Masked Transformers with Lightweight Test-Time Adaptation for Neural Speech Decoding

We thank the reviewer for recognizing the contribution of combining and standardizing several datasets. To the best of our knowledge, this study gathered all the publicly available intracortical recordings of humans from Utah arrays.

“ it is unclear whether the limited gains reflect the pre-training methods themselves, insufficient pre-training data, or simply the availability of enough labeled data during finetuning.”

We agree with the reviewer that this is an important question. To address the last hypothesis, we ran labelled-data curves. We fine-tuned the architecture on 11 fractions of the training set (1, 2, 3, 5, 7.5, 10, 15, 20, 30, 50, 100%) of one representative dataset per task: Wairagkar for attempted speech, Fan for handwriting, and participant T18 for typing. Each configuration was trained from scratch and from each of the 6 pretrained checkpoints, with 3 seeds, for 693 runs in total.

The training subsets are stratified by recording day and nested, so each fraction contains the smaller ones. For a given fraction and seed, all models are trained on exactly the same trials. All runs process the same number of training examples. The checkpoint is selected on the validation set, and we report test error. Each pretrained model is used at its best size and with its best fine-tuning configuration from our sweep.

Median test error (%) over 3 seeds: PER for speech, CER for handwriting and typing. Columns give the fraction of the training set and the corresponding hours of labelled recordings. Values of roughly 60% or more correspond to runs that stay on the CTC plateau.

Attempted speech (Wairagkar, PER)

1% (0.3 h)5% (1.4 h)10% (2.7 h)20% (5.2 h)50% (13 h)100% (26 h)
Scratch62.738.927.117.810.27.3
MAE-H53.834.626.218.610.97.6
MAE-HM57.636.927.018.410.07.5
JEPA-H76.339.329.419.511.78.7
JEPA-HM65.440.330.121.411.98.8
AR-H65.236.527.520.111.07.4
AR-HM64.037.528.020.011.07.6

Handwriting (Fan, CER)

1% (0.1 h)5% (0.5 h)10% (1.0 h)20% (2.0 h)50% (4.8 h)100% (9.7 h)
Scratch77.678.676.573.333.720.9
MAE-H73.173.769.648.629.522.1
MAE-HM74.672.572.459.626.220.4
JEPA-H79.680.078.773.937.026.0
JEPA-HM80.979.779.573.337.626.3
AR-H75.571.457.344.228.521.7
AR-HM76.574.574.748.728.621.7

Typing (T18, CER)

1% (0.1 h)5% (0.35 h)10% (0.7 h)20% (1.3 h)50% (3.1 h)100% (6.1 h)
Scratch75.633.022.717.512.410.2
MAE-H75.826.419.714.612.010.8
MAE-HM77.328.321.917.412.510.5
JEPA-H104.777.577.520.374.012.0
JEPA-HM90.971.824.819.372.012.1
AR-H50.425.822.315.711.810.1
AR-HM76.074.678.369.012.110.2

Attempted speech:. In the low-data regime, pretrained models, especially MAE-H and MAE-HM, outperform the scratch baseline. With 0.3 h of labelled data, MAE-H reaches 53.8% PER against 62.7% from scratch: 8.9 points, or 14% relative. The gap is still 4.3 points at 1.4 h. The advantage decreases as the available data increases and fades at about 4 hours. We define the crossover as the point where the paired difference with scratch on the same trials reaches zero, interpolated between fractions. For MAE-H and scratch the crossover is at 4.0 h.. Beyond this point all models perform similarly: 7.3-8.8% PER with the full 26 h.

For handwriting and typing the picture is more complex, because of high training instability. Many runs stay on a CTC plateau (validation CER >= 60%) and do not converge to a useful solution.

On handwriting, no model escapes the plateau below about 1 h of data. Above that, pretrained models converge earlier and more reliably than scratch. At 2 h, AR-H and MAE-H reach 44.2% and 48.6% CER against 73.3% from scratch. With at least 10% of the data, scratch remains on the plateau in 12 of 18 runs, against 4/18 for MAE-H and 3/18 for AR-H. The advantage persists up to almost the full training set: the MAE-H crossover is at 8.9 h out of the 9.7 h available.

AuthorsOpenReview ↗

W2 - continued

W2 - continued) On typing, pretrained models are better between about 0.3 and 3 h. For example, at 0.35 h MAE-H and AR-H reach 26.4% and 25.8% CER against 33.0% from scratch. The MAE-H crossover is at 3.3 h. Here scratch is the most stable model: with at least 10% of the data it never stays on the plateau (0/18 runs), against 2-4/18 for MAE-H, MAE-HM and AR-H and 7-10/18 for AR-HM and the JEPA models. In the very-low-data regime (<= 0.2 h), the medians are therefore dominated by non-converged runs. When the best of the 3 seeds is selected on validation, MAE-H reaches 41.6% CER at 0.1 h, against 74.4% from scratch.

We do not observe a consistent advantage of the JEPA-pretrained models on any of the three tasks.

To verify that the advantage is not due to model size or to the per-model choice of fine-tuning hyper-parameters, we repeated the speech experiment with all encoders at the same size (384x7) and a single common fine-tuning configuration. The conclusions are unchanged. MAE-H remains the best model below about 4 h. The AR and JEPA encoders are within a few points of scratch. No pretrained model outperforms scratch beyond about 3-4 h of data.

This is a complementary result to what was found by Ye et al. (NDT-3) for a simpler motor task: the advantages of pretraining in their setting faded after 1.5 hours of labelled data became available.

About the pre-training methods: to improve the broadness of our results, we have run a comprehensive optimization of the hyperparameters during pretraining for all the considered strategies, which are the most common techniques in literature. The Reviewer will find all details in the answer to W3 below.

In conclusion, regarding the Reviewer’s question: “is it simply that we have enough labelled data for fine tuning, or do we have limited data for pre-training?”. The availability of enough labeled data during fine-tuning is confirmed by the labelled data curves, which show that the advantages of pretraining are present only in the low-data regime and fade as the whole fine-tuning dataset is used. This result, however, is of course conditioned by the amount of pre-training data that is available today. A different conclusion may be reached if the pre-training data were to be scaled significantly. The field is still far from the data moles that are available in other domains, where the real emergent capabilities of foundation models began to show at tens to hundreds of thousands of hours of audio, millions of images or trillions of text tokens. To the best of our knowledge, the amount of Utah array recordings used for pre-training here is close to the ceiling of the field at its current state.

Te et al., 2023: Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Update, approaching the end of the discussion phase: we sincerely thank the Reviewers for their engagement. Two Reviewers (AxzN and wBhp21) have acknowledged our revisions and decided to raise their scores. The summary below reflects the state of the revised manuscript.

We provide a systematic evaluation of self-supervised pretraining for intracortical neural decoding. Using a shared BIT Transformer backbone, we compare three widely used pretraining objectives (MAE, JEPA, and autoregressive modeling), two pretraining corpus compositions (human-only and human-plus-non-human-primate data), and training from scratch across speech, handwriting, and typing tasks. As also acknowledged by the reviewers, key strengths of the work include the effort to assemble, align, and standardize a large and diverse collection of intracortical recordings; the broad evaluation across relevant downstream BCI tasks; and the controlled comparison of different pretraining strategies under consistent experimental conditions. Importantly, the study also provides useful evidence about the current limitations of pretraining for neural decoding, including findings that challenge the assumption that more pretraining data or additional species necessarily translate into better downstream performance at the current data scale.

As several comments converged on some points, we summarize them below and explain how these have been addressed.

  1. Both Reviewer 9RxL and AxzN identified the lack of a pretraining hyperparameter sweep as a weakness. We agree with the reviewers that tuning hyperparameters also during the pretraining phase may improve the broadness of our results. To address this, we have performed a thorough sweep during pretraining, including varying the transformer dimension, an experiment which amounted to approximately 19,000 GPU-hours. We repeated the fine-tunings and subsequent analysis with the new, improved version of the pre-trained models. We report all obtained results in the answers to the two reviewers, identifying all the changes with respect to the previous version of the manuscript, as well as in the revised version of the article now available.

  2. All Reviewers identified the lack of labelled-data curves as a weakness. We agree that this analysis would improve the quality of our discussion and the relevance of our results. We have performed the analysis on three representative downstream datasets, one per task: Wairagkar for attempted speech, Fan for handwriting, and subject T18 for typing. The main takeaway message here is that pretraining proves useful in the low-data regime, with benefits with respect to the trained-from-scratch model fading as the percentage of each considered training set increases. Further details are given below, in the answers to the Reviewers, and in the updated manuscript.

  3. Reviewer wBhp21 raised concerns on the limited number of seeds (5) of the stability analysis. We agree that n=5 was a limited pool; we have increased the number of seeds to 10 to address this concern. Further details are in the answers below and in the revised manuscript.

In addition to what stated above, to address the further concerns of Reviewer AxzN we have added the following analyses: a fully supervised pretrained baseline in two separate variants, evaluated on all downstream tasks; an additional linear probing experiment; an additional training speed analysis comparing fine-tuning speeds for pretrained models to the from-scratch baseline. We have done our best effort to address all the Reviewer’s concerns and hope to have solved most of them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.